Stream: implementers
Topic: About _include:iterate
nicola (RIO/SS) (Nov 26 2019 at 12:28):
Do we really need to use :iterate modifier for chained includes? It just introduces confusion between chaining and the recursive case. From the implementer's point of view, this is two completely different algorithms - chained and recursive one.
nicola (RIO/SS) (Nov 26 2019 at 12:30):
What to do if iterate missed ? For example: MedicationDispense?_include=MedicationDispense:prescription&_include=MedicationRequest:performer should we ignore include or show error message?
nicola (RIO/SS) (Nov 26 2019 at 12:31):
How to distinguish between a recursive case and just self-reference without hierarchy?
nicola (RIO/SS) (Nov 26 2019 at 12:32):
I would suggest using something like :recursive instead of ':iterate' or use ':iterate' only for recursive case (not chained).
Grahame Grieve (Nov 26 2019 at 14:57):
Isn’t this too late to change?
nicola (RIO/SS) (Nov 26 2019 at 14:59):
Did anybody implement this without any problems? @James Agnew @Christiaan Knaap ?
nicola (RIO/SS) (Nov 26 2019 at 15:00):
We missed this correction and just added in aidbox alias for iterate=recursive :(
James Agnew (Nov 26 2019 at 15:18):
I do tend to agree the way this works is a bit confusing..
I know HAPI's current behaviour for @nicola (RIO/SS) 's example query above wouldn't actually return any performer, since there is no :iterate on the second include so it doesn't apply to subsequent rounds of include loading. You can work around this by specifying _include:iterate=MedicationRequest:performer, which is easy to do but I find it counter-intuitive since you're not really iterating on that include at all. You're just asking for it to be applied to the results of the first include. Re-reading the spec, I think it's safe to say that this is the way it's supposed to work at least.
TBH I'm not sure how this could be improved though. Adding other modifiers seems like it's just layering more confusing stuff on (not to mention the fact that the http page is normative). We could add text perhaps that specifies that _include on a resource type other than the one being searched for will apply to any resources returned by the first round of include loading? I feel like that would be a nightmare to describe, but it would certainly make the example above a bit more intuitive.
nicola (RIO/SS) (Nov 26 2019 at 16:08):
@James Agnew did you implement recursive version of iterate?
nicola (RIO/SS) (Nov 26 2019 at 16:22):
how do you distinguish is it recursive or not?
nicola (RIO/SS) (Nov 26 2019 at 16:23):
or user has to write :iterate for each iteration?
James Agnew (Nov 26 2019 at 16:34):
@nicola (RIO/SS) i'm not sure i understand your question. We did implement it, and i am, well, reeeeeasonably sure it's implemented to spec.
You have to add :iterate (or :recurse - we also just treat them as synonyms) to each include parameter. I.e. any include without iterate is only applied to the actual search results, and any include with iterate is applied to search results but also to new results pulled in by previous includes. Is that what you mean?
nicola (RIO/SS) (Nov 26 2019 at 18:41):
i mean Initial idea of recursive was for request like Organization?_include:recursive=Organization:part-of make as much iterations as needed to get all chlidren and children of children,.... Did you implement this logic? Or hapi makes only one iteration for part-of?
nicola (RIO/SS) (Nov 26 2019 at 18:44):
This recursive iterations logic is expensive and makes great diff with just chained includes. So for performance reason it is important to distinguish this two.
Grahame Grieve (Nov 26 2019 at 19:01):
right, important to differentiate, which is why we added iterate. The server doesn't have to keep iterating for ever
nicola (RIO/SS) (Nov 26 2019 at 21:27):
do you mean query should be like this: ?_inc:iterate=partof&_inc:iterate=partof to get grandparents?
Patrick Werner (Nov 27 2019 at 11:19):
i always assumed that Organization?_include:iterate=Organization:part-of iterates into all children and children of children.
i understood the iterate here MedicationDispense?_include=MedicationDispense:prescription&_include:iterate=MedicationRequest:performer as a for-each
Christiaan Knaap (Nov 27 2019 at 12:07):
Did anybody implement this without any problems? James Agnew Christiaan Knaap ?
We haven't implemented :iterate yet. And apparently nobody has missed it either.
nicola (RIO/SS) (Nov 27 2019 at 12:23):
We have some clients, who use it :(
Just to demonstrate a real cases :
GET /RequestGroup
?_id=reqgroup1
&_include=patient,author
&_include:iterate=RequestGroup:target:MedicationRequest
&_include:iterate=MedicationRequest:medication
&_include:iterate=MedicationRequest:requester:PractitionerRole
&_include:iterate=MedicationRequest:intended-performer:Organization
&_include:iterate=PractitionerRole:practitioner
&_include:iterate=Patient:organization
&_include:iterate=PractitionerRole:location
&_revinclude:iterate=AllergyIntolerance:patient
:)
nicola (RIO/SS) (Nov 27 2019 at 13:52):
I drafted a tiny proposal for _(rev)include alternative - _with parameter - https://gist.github.com/niquola/48d106a88dbf93eb17783c0d5c5beb36 :)
Patrick Werner (Nov 27 2019 at 14:27):
i mean Initial idea of recursive was for request like Organization?_include:recursive=Organization:part-of make as much iterations as needed to get all chlidren and children of children,.... Did you implement this logic? Or hapi makes only one iteration for part-of?
hapi doesn't stop after one Iteration (which i think is the right approach):
https://fhir.molit.eu/r4/Observation?_id=559&_include:iterate=Observation:has-member
nicola (RIO/SS) (Nov 27 2019 at 14:36):
Try nested case: Patient?_id=???&_revinclude=Observation:subject&_include:iterate=Observation:has-member
Patrick Werner (Nov 27 2019 at 15:05):
works and gets parent, child, grandchild:
https://fhir.molit.eu/r4/Patient?_id=564&_revinclude=Observation:subject&_include:iterate=Observation:has-member
nicola (RIO/SS) (Nov 27 2019 at 16:11):
Cool! Now how to interpret the following request:
is it recursive or not?
Patrick Werner (Nov 28 2019 at 11:09):
i changed the second include to the correct revinclude, works.....:
nicola (RIO/SS) (Nov 29 2019 at 08:13):
@Grahame Grieve Are we ok with this ambiguity of iterate? I would suggest to get back :recursvie for recursive cases.
nicola (RIO/SS) (Nov 29 2019 at 08:15):
i changed the second include to the correct revinclude, works.....:
the problem is that it can be both recursive and not! And now it is impossible to say this implicitly :(
Grahame Grieve (Nov 29 2019 at 12:04):
I haven't followed where this is ambiguous?
nicola (RIO/SS) (Nov 29 2019 at 15:38):
@Grahame Grieve here is the simplest explanation!
How can I include only the first layer of parents Observation:derived-from (i.e. with no recursion) for this example - Patient?_id=603&_revinclude=Observation:subject&_revinclude:iterate=Observation:derived-from ?
Problem with mixed things - is to unmix them :)
Grahame Grieve (Nov 29 2019 at 17:42):
you can't. It's all or nothing. In the spec
nicola (RIO/SS) (Nov 30 2019 at 11:23):
isn't it unaesthetic - with :recursive user will be able to do this and implementation will be simpler ;)
Grahame Grieve (Nov 30 2019 at 18:06):
umm? :recurse and :iterate are synonyms?
nicola (RIO/SS) (Dec 02 2019 at 08:47):
Or do not use :iterate for chained includes (it's redundantly)
nicola (RIO/SS) (Dec 02 2019 at 16:49):
@Grahame Grieve should I stop? or create an issue?
Grahame Grieve (Dec 02 2019 at 17:29):
I'm still not sure that I follow the issue. If you want to be able to specify a 'only -once' or a count of iterations to do, I follow that. Otherwise, I'm not sure what the issue is - the committee renamed :recurse to :iterate, and that seems to have caused confusion?
nicola (RIO/SS) (Dec 03 2019 at 21:17):
Committee has extended meaning of :iterate to chained incudes - i.e. mixed up two different things ;(
Grahame Grieve (Dec 04 2019 at 00:05):
I'm still not seeing the different things
nicola (RIO/SS) (Dec 04 2019 at 04:54):
With chained include you just do one "iteration", for recursive you do as many as possible - isn't it serious difference?
nicola (RIO/SS) (Dec 04 2019 at 05:01):
@James Agnew what do you think? Is it a problem or not?
Paul Church (Dec 04 2019 at 14:44):
I see the distinction but it only makes a difference in cases of type X referring to type X - for most references there is no recursion possible. It looks like we are only discussing the case where the client wants to follow exactly one level of Observation:derived-from or Patient:link or similar, and is not happy if the server continues into more than one level. But the server already has the discretion to decide how deep to follow the recursion, so does the client really have that level of control? If you could limit to no more than 1 level, why not also limit to no more than 2 levels?
nicola (RIO/SS) (Dec 04 2019 at 16:52):
@Paul Church there are many references, which can refer to the same resource type and/or other - for example, https://www.hl7.org/fhir/observation-definitions.html#Observation.hasMember - should for example Observation?_include=has-member be recursive for self-references or not?
nicola (RIO/SS) (Dec 04 2019 at 16:54):
Ability to explicitly specify a limit for recursion is a good idea (we are thinking to introduce it as an extension)
René Spronk (Dec 05 2019 at 07:40):
The use case seems exotic - so an extension would probably be a good idea should someone need it. A lot of things in the query syntax don't meet the "80%" criterium anyway (ok, that doesn't officially apply to query parameters, but it might as well), so I'd rather see stuff removed (as if that's likely to happen..) than added.
nicola (RIO/SS) (Dec 05 2019 at 08:56):
@René Spronk just to show you "real-world" query:
/RequestGroup?
_include=target&
_include=patient:Patient
_include=Patient:organization,
_revinclude:iterate=AllergyIntolerance:patient:Patient&
_revinclude=Condition:subject:Patient&
_include=author:PractitionerRole&
_include=PractitionerRole:practitioner:Pracitioner&
_include=PractitionerRole:location&
_revinclude=Contract:subject:PractitionerRole&
_include=RequestGroup:target:MedicationRequest&
_include=MedicationRequest:medication&
_include=MedicationRequest:requester:PractitionerRole&
_include=MedicationRequest:intended-performer:Organization&
_include=MedicationRequest:intended-performer:Organization
René Spronk (Dec 05 2019 at 09:03):
That's fine - but having to distinguish between 1 or 2 or unlimited depth in an iterative _include ? That sounds exotic to me.
nicola (RIO/SS) (Dec 05 2019 at 09:04):
That's mostly for performance reason - handling such huge includes efficiently is a challenge.
nicola (RIO/SS) (Dec 09 2019 at 10:52):
Looks like nobody interested in this inconsistency right now - ok let's wait for more implementers and users will report this problem :)
nicola (RIO/SS) (Dec 09 2019 at 13:17):
@Patrik Sundberg have you implemented (rev)include:iterate in google health API?
Paul Church (Dec 11 2019 at 15:40):
The Google API supports :iterate but limited to a max depth of 2 at the moment. Our interpretation is that all iterates are recursive for self-references. We have performance and pagination concerns about increasing the max depth. I think for your complex include example we would prefer to implement GraphDefinition operations or GraphQL rather than try to make that query performant in the form of a search.
Paul Church (Dec 11 2019 at 15:50):
Related question about _include - if a Patient?_include=Patient:organization search result contains two Patients that both refer to the same Organization, does that Organization resource appear once or twice in the searchset bundle? If once, how does this interact with pagination?
Paul Church (Dec 12 2019 at 17:27):
Bumping this question - how do other implementers handle resources from _include that could appear more than once in the results? Do they appear multiple times? Once per page? Only once overall?
Michele Mottini (Dec 12 2019 at 17:58):
Once in each page where it is referenced or referencing - the process is compute the page of results, look for all include related to the resources in the page and add them to the page,
nicola (RIO/SS) (Dec 13 2019 at 07:27):
We decided to iterprer :iterate as recursive only if source and target specified and equal. As well we support old :recurse for explicit recursion. To be honest, you don't need :iterate modifier for chained includes - in aidbox they are optional. https://docs.aidbox.app/basic-concepts/search-1/_include-and-_revinclude
nicola (RIO/SS) (Dec 13 2019 at 07:29):
@Paul Church if we make recursion explicit, i do not see performance complications to limit depth of includes: each include is just one query.
nicola (RIO/SS) (Dec 13 2019 at 07:35):
@grahame i worry how this breaking change was introduced without real feedback from major implementers. Can we organize kinda subscriptions/notifications for fhir server implementers about such changes?
nicola (RIO/SS) (Dec 13 2019 at 07:42):
just little bit (technically open) open and formal process right after such changes is appear and before bulk balloting? Something like email with google form and short description for those who don't participate in wg meetings, but still interested and affected in/by such changes
Grahame Grieve (Dec 13 2019 at 09:31):
well, we had several rounds of balloting and calls for comments. The process couldn't be more open than it is
Grahame Grieve (Dec 13 2019 at 09:32):
it's true, though, that I don't find anything in Zulip about this
Grahame Grieve (Dec 13 2019 at 09:34):
no, I did: https://chat.fhir.org/#narrow/stream/179166-implementers/topic/_include
Last updated: Apr 12 2022 at 19:14 UTC
