Stream: implementers
Topic: slice ordering
Jose Costa Teixeira (Feb 16 2022 at 21:20):
Can we enforce a specific order of slices? Something like: medications in a list shall be ordered by date of start ascending.
Jose Costa Teixeira (Feb 16 2022 at 21:21):
or medication ingredients shall be sorted by role
Grahame Grieve (Feb 16 2022 at 21:21):
is that actually slicing? Or just an invariant?
Shamil Nizamov (Feb 16 2022 at 21:21):
If a final resource instance is represented in JSON then ordering is not guaranteed at all.
Elliot Silver (Feb 16 2022 at 21:23):
If ordering by date is important to the client, why don't they sort the medications by date? If role is important, then shouldn't there be a type or category of ingredient you can group by?
Chris Moesel (Feb 16 2022 at 21:42):
Shamil Nizamov said:
If a final resource instance is represented in JSON then ordering is not guaranteed at all.
Ordering of keys is not guaranteed, but I think @Jose Costa Teixeira is talking about ordering of items in arrays, which is definitely supported in JSON.
Jose Costa Teixeira (Feb 16 2022 at 21:44):
right, ordering of items in an array.
Jose Costa Teixeira (Feb 16 2022 at 21:47):
Challenge at hand: I have data coming in and I need to ensure a given order of the repeatable elements. While the functional analysts are seeing what order is that, once they do, I want to see if there is any mechanism in FHIR.
Jose Costa Teixeira (Feb 16 2022 at 21:47):
These are repeatable elements and I don't have so far the need for slicing, unless slicing is the way to enforce a given sequence (which I don't see how it would be)
Jose Costa Teixeira (Feb 16 2022 at 21:48):
taking medication.ingredient. I want to sort by role
Grahame Grieve (Feb 16 2022 at 22:27):
this is pretty difficult on two parts. The first is expressing an order predicate in FHIRPath. I suspect it's possible by playing around with the aggregate() function, but it's not immediately obvious. if it turns out that it's not possible, we could propose a more direct way (on #fhirpath)
The second and probably less tractable issue is the actual correct order for 'role' - typically the alphabetical order of codes is irrelevant and there's some deeper meaning.
Lloyd McKenzie (Feb 16 2022 at 22:47):
Note that enforcing order on elements where order doesn't have meaning is a very bad idea and will generally interfere with interoperability. (And if order does have a defined meaning, then the sort is already defined, and you don't have to do it.)
Lloyd McKenzie (Feb 16 2022 at 22:47):
Only the core resource definition can define meaning to order - it can't be introduced by IGs or profiles.
Eric Haas (Feb 16 2022 at 23:21):
I know that ordering of arrays is supposed to be maintained but if you don't trust that is going to happen. you could add an ordering extension. (as we did in our Questionnaire IG).
Grahame Grieve (Feb 16 2022 at 23:56):
@Eric Haas I don't know why what would make any difference
Eric Haas (Feb 17 2022 at 00:10):
if the resource get decomposed and the recomposed between system that are FHIR facades it adds a redundant layer of cross checking. At the time it seemed important to the folks on the call.
Grahame Grieve (Feb 17 2022 at 00:10):
well, ok. redundant is redundant...
Lloyd McKenzie (Feb 17 2022 at 01:51):
If you don't trust the system to maintain the order, why do you trust it to keep the extension?
Daniel Venton (Feb 17 2022 at 14:43):
In general, you can never ever trust that the order of anything will be maintained forever exactly as you saw it/stored it unless somebody applies that ordering. If you aren't absolutely sure that the provider of the data ordered the list for you, then you'll have to do so. There are so many layers happening where somebody might not have cared about maintaining sequence it's just better for the processor to do a quick sort based on whatever attribute the processor wants to process them in. There are some lists in FHIR that have a sequence number because in those cases sequence is important and nobody can trust that physical sequence will be maintained forever and ever by everybody that touches it.
Lloyd McKenzie (Feb 17 2022 at 15:28):
If a FHIR system is conformant, it's required to retain the order of array elements - end stop. If it doesn't, it will - at minimum - break any embedded signatures. At worst, it's going to mis-order elements where order is significant.
Obviously systems can do non-conformant things, but that doesn't mean that adding extra layers is going to make things better.
Daniel Venton (Feb 17 2022 at 15:42):
That's a fine requirement on a FHIR server. Does every JSON/XML serializer/deserializer have the same requirement? Does every other tool that might be used in the eco-system?
Chris Moesel (Feb 17 2022 at 15:51):
Lloyd McKenzie said:
If a FHIR system is conformant, it's required to retain the order of array elements - end stop.
@Lloyd McKenzie -- I don't think that's true for every array element (or even most). The spec actually says ordering does not need to be retained if the meaning of the order has not been defined (here):
For elements that have cardinality > 1, the order in which they appear may have meaning. Unless the element definition (either in this specification or the extension) defines a meaning to the order explicitly (using ElementDefinition.orderMeaning), the meaning of the order is not defined, and implementations are allowed to reorder the elements. Note that it is not possible to define a meaning for the order of the elements in a profile using a StructureDefinition.
Chris Moesel (Feb 17 2022 at 15:56):
Expecting an implementation to maintain the order of every array makes a lot of assumptions about how that implementation stores data. For example, the Encounter.reasonReference array could be represented on a source system via a join table, a relationship to problems/procedures/etc on encounter, or a relationship to encounters on problems/procedures/etc. Not every system is going to maintain an order on those relationships -- especially if there is no business (or clinical) reason to do so.
Lloyd McKenzie (Feb 17 2022 at 16:09):
If we allow implementations to re-order elements, then we can kiss any hope for digital signatures good-bye... Wow, that wording has been there since STU2. @Grahame Grieve - how do we rationalize this statement and the need for signatures?
Daniel Venton (Feb 17 2022 at 16:20):
Would a digital signature be considered part of the resource (and stored) or would a digital signature be added JIT as the resource is being serialized for transmission, validated at the consumer and then dropped.
David Pyke (Feb 17 2022 at 16:50):
Where a signature is part of the resource (like Bundle) it should be stored within the resource.
Daniel Venton (Feb 17 2022 at 16:59):
If you are storing the serialized copy of the resource, then you can store the signature as well. But if you are decomposing the resources and attempting to "recreate" the bundle then it's unlikely that you'll ever be able to get the bytes lined up exactly correct in order to match a signature.
John Moehrke (Feb 17 2022 at 17:29):
very dependent on the kind of digital signature. Fundamentally there are digital signatures that are (a) encapsulating, (b) encapsulated, and (c) external.
Grahame Grieve (Feb 17 2022 at 20:10):
well, obviously, a system that doesn't preserve order can't make a digital signature survive whatever it does that loses the digital signature. That's not new or surprising, but a system that's taking a resource apart and putting it in a database, and then it's not going to be concerned with that.
Lloyd McKenzie (Feb 17 2022 at 20:12):
Which essentially means that digital signatures can generally only survive passing through systems designed to handle them. A digital signature that passes through an arbitrary set if intermediaries is likely to get trashed.
Lloyd McKenzie (Feb 17 2022 at 20:14):
I didn't realize we'd agreed to this as reality. (It certainly makes FHIR conformance easier for systems that don't care about signatures, but it makes the use of signatures really tenuous. And it sort of argues for a canonicalization algorithm that automatically sorts data where sort order is meaningless.
Lloyd McKenzie (Feb 17 2022 at 20:14):
Should we have a change request to a) call attention to this issue; and/or b) define such a canonicalization mechanism?
Grahame Grieve (Feb 17 2022 at 21:29):
A digital signature that passes through an arbitrary set if intermediaries is likely to get trashed
yup. That's a statement of truth. Which is why we pretty much don't see digital signatures used with RESTful APIs anywhere.
Grahame Grieve (Feb 17 2022 at 21:29):
not just with FHIR
Grahame Grieve (Feb 17 2022 at 21:29):
I thought it was pretty clear, actually
Jose Costa Teixeira (Feb 17 2022 at 21:59):
Lloyd McKenzie said:
Note that enforcing order on elements where order doesn't have meaning is a very bad idea and will generally interfere with interoperability. (And if order does have a defined meaning, then the sort is already defined, and you don't have to do it.)
I will need to pass my entries through an algorithm and I want to do it in a specific sequence. Is there a way to see in FHIR whether that sequence is in place?
Jose Costa Teixeira (Feb 17 2022 at 22:00):
For example, if order is alphabetical, i'd like to see that entry[1] is alphabetically after entry[0] and so on
Lloyd McKenzie (Feb 17 2022 at 23:37):
@Grahame Grieve - where do you think it's clearly stated now?
Grahame Grieve (Feb 17 2022 at 23:56):
http://build.fhir.org/updates.html and and both format pages have an implementation note in the canonical section about the use of signatures outside document bundles, but that could be clearer. And the page about updates could talk about signatures
Lloyd McKenzie (Feb 18 2022 at 00:11):
I would think the signature page itself should indicate that there's no guarantee the signatures will be valid across persistence. And also define canonicalization algorithms that force a sort order (to allow signatures).
Grahame Grieve (Feb 18 2022 at 01:22):
we have a signature page?
Lloyd McKenzie (Feb 18 2022 at 02:10):
We have a signature data type with a whole bunch of information about signatures.
Grahame Grieve (Feb 18 2022 at 02:37):
it has the same comment
Lloyd McKenzie (Feb 18 2022 at 14:51):
I don't see any comment that talks about the fact that persisting order isn't reliable and that therefore, regardless of reidentification, signatures can't be counted on.
John Moehrke (Feb 18 2022 at 20:52):
there are canonical for signatures.. but for multi value elements I don't think we have a rule.
John Moehrke (Feb 18 2022 at 20:54):
There is a bunch of signature pages
- General discussion of Digital Signatures -- https://hl7.org/fhir/signatures.html
- Signature Datatype -- https://hl7.org/fhir/datatypes.html#signature
- XML vs JSON signatures
- FAQ on signatures -- https://hl7.org/fhir/secpriv-module.html#signature
John Moehrke (Feb 18 2022 at 20:55):
The Signature datatype does address the fact that the signature will be specific to a FHIR mime-type; thus the signer and the validator do need to agree to one of the FHIR mime-types... (Or both).
John Moehrke (Feb 18 2022 at 20:56):
but restating ... I don't think we have a solution for multi value elements... hmm
John Moehrke (Feb 18 2022 at 21:03):
All of these are reasons why I have cautioned EVERYONE against thinking signatures are easy. Signatures are only practical across a Binary. That Binary may have a Provenance linkage to a FHIR Document Bundle, or to the accessible Resources within; where that Provenance declares that it is proof that the Binary was received and that the signature on the Binary was checked. This can be done for anything, but my first point is that signatures of Binary are really the only practical use-case.
Last updated: Apr 12 2022 at 19:14 UTC
