Stream: conformance
Topic: preserve order of repeatable elements
Allan Bro Hansen (Aug 18 2016 at 11:47):
Hi
I disagree with this statement about ordering of xml-element (or a least with a possible interpretation of it):
"When an element is allowed to repeat, the elements are ordered, and implementations must preserve order (note: the meaning of the order may not be known)"
(from https://www.hl7.org/fhir/xml.html)
This could be read as: Whenever a FHIR-ressource is created/updated: the order of ..* related elements must either be persisted or validated against some rules to ensure that the same order is returned.
I strongly believe that it should be a case-by-case decision whether the order must be maintained/matters.
Some examples of what I consider strange behaviour if we follow the statement:
Example1
If a client reads a fhir-ressource with 2 business identifier:
FHIR-ressource
- Businessidentifer:
-> SomeBusinessIdentifier
-> AnotherBusinessIdentifier
And this clients now adds a third business identifer and PUT's this to the server:
FHIR-ressource
- Businessidentifer:
-> SomeBusinessIdentifier
-> SomeNewBusinessIdentifier
-> AnotherBusinessIdentifier
Now the server should be now obligued to maintain this order?
Example2
If a client reads a fhir-appointment with 3 participations:
FHIR-appointment
Participants
-> PatientParticipation (it is a patient because of the actor)
-> LocationParticipation (it is a location because of the actor)
-> PractitionerParticipation (it is a practitioner because of the actor)
And this clients now updates the appointment in some way but changes the order of the participants.
Now the server should be now obligued to maintain this order of the participants - or just reject the update because of the order.
Some other thoughts:
- if FHIR did use xsd we would have xs:all to define a unordered behaviour
- HL7 version 3 did have ActRrelationship.sequenceNumber to define a fixed sequence
Any comments on this? Do you agree or disagree with me? (or have I missed some FHIR-defined way to specify an unordered list)
Grahame Grieve (Aug 18 2016 at 11:50):
You have interpeted 'implementations must preserve order' more expansively than we intended.
Grahame Grieve (Aug 18 2016 at 11:51):
http://hl7-fhir.github.io/updates.html
Grahame Grieve (Aug 18 2016 at 11:52):
What that sentence intends to say is that when you are reading and writing elements, the order does matter, and the reading and writing code needs to able to preserve order.
Grahame Grieve (Aug 18 2016 at 11:53):
and reordering the elements may change the meaning
Grahame Grieve (Aug 18 2016 at 11:53):
The requirement to preserve order doesn't automatically apply if you are editing the resource, but you should understand the meaning of the order, if there is one, when re-ordering
Grahame Grieve (Aug 18 2016 at 11:54):
we could be clearer about the scope of our statement
Josh Mandel (Aug 18 2016 at 11:54):
In other words: this rule applies to serialization libraries and persistence layers -- not to clients editing a resource. One reason the current language may be unclear is that the word "implementations" can mean so many things (and to different people).
Grahame Grieve (Aug 18 2016 at 12:17):
yes that's what's unclear
Lloyd McKenzie (Aug 18 2016 at 23:00):
@Allan Bro Hansen Would you be willing to submit a change request for us to clarify the spec?
Allan Bro Hansen (Aug 19 2016 at 08:17):
Thank you for the clarification. And yes, of course "the code" should maintain the order until it knows whether the order matters (almost goes without saying). @Lloyd McKenzie : I'll be looking at a change request later today or monday.
Jens Villadsen (Aug 19 2016 at 08:39):
Could this be exemplified even more? AFAIK, order in xml is not part of the xml spec but can of course be mandated with schemas. On the other hand, in JSON, an array of objects is an ordered sequence. Since, FHIR is supporting both formats, I need some clarification on this matter. It must somehow mean that a 'collection' (an unordered list/set) when serialized in one format (xml) has one meaning but some other meaning when serialized to another format (JSON)?
Allan Bro Hansen (Aug 19 2016 at 08:51):
A possible "better" statement:
"
- When an element is allowed to repeat, the elements are ordered, and implementations must be able to preserve order if the order matters
(see also https://www.hl7.org/fhir/conformance-rules.html#cardinality for a further description of elements with cardinality > 1)
"
To replace
- When an element is allowed to repeat, the elements are ordered, and implementations must preserve order (note: the meaning of the order may not be known)
Grahame Grieve (Aug 19 2016 at 10:52):
@Jens Villadsen meaning is the same in both formats. Both formats are explicitly serialised in a particular order. What we are saying is that the serialization order is significant, and the technical infrastructure needs to be able to access the items in the right orde
Grahame Grieve (Aug 19 2016 at 10:53):
of course, the order might be be meaningful. or it might be meaningful to the writer but not the reader. That's tricky, and we can't make tha untricky
Grahame Grieve (Aug 19 2016 at 10:54):
Allan, I'd like to go further than your proposal and replace implementations with 'the technical infrastructure needs to be able to access the items in the right order'
Jens Villadsen (Aug 19 2016 at 11:15):
OK, just to understand it from https://www.hl7.org/fhir/conformance-rules.html#cardinality:
...
"For elements that have cardinality > 1, the order in which they appear may have meaning. Unless the element definition (either in this specification or the extension) defines a meaning to the order explicitly, the meaning of the order is not defined, and implementations are allowed to reorder the elements. Note that it is not possible to define a meaning for the order of the elements in a profile using a StructureDefinition. When there is no definition of the meaning of the order, implementations that need to choose a single element from a list of elements for some use SHALL do so based on the semantics of the content of the elements that repeats. Profiles and Implementation guides may often make rules about this selection process."
The order cannot be defined using StructureDefinitions. - So if there is a defined order, it is not expressed in terms of computable syntax or computable semantics by the StructureDefinition. Does that mean that it is OK to define an order expressed in human computable language and that is currently the only way do define such an order?
Grahame Grieve (Aug 19 2016 at 11:37):
yes that's what it means. But I don't know why we said you couldn't define the order in a profile. I think we were concerned that people would change the order from the base. but if the base doesn't assign an order....
Jens Villadsen (Aug 19 2016 at 12:17):
OK - its just, when you read that order can be defined (with a technical mindset - FHIR is, in many aspects a technical standard), I would expect that order to be computable by a machine. It is however not the case here
Grahame Grieve (Aug 19 2016 at 12:24):
what would 'computable by machine' mean?
Jens Villadsen (Aug 19 2016 at 12:25):
eg. order expressed in a schema or having an index attribue on the elements
Grahame Grieve (Aug 19 2016 at 12:27):
index attribute is a different question. that would mean that the actual order wasn't the order. It has nothing to do with meaning
Grahame Grieve (Aug 19 2016 at 12:27):
what would the schema express?
Jens Villadsen (Aug 19 2016 at 12:45):
you're right ... it cannot be done
Jens Villadsen (Aug 19 2016 at 12:59):
at least I dont know how
Allan Bro Hansen (Aug 19 2016 at 13:16):
@Grahame Grieve : In respect to replacing "implementations": I think the important part is to include the "if the order matters". "Technical infrastructure" sounds very low-level (hardware/network) and I don't think FHIR should be concerned with that.
Lloyd McKenzie (Aug 19 2016 at 16:53):
When you persist an instance, you must retain the order of repeating elements whether the order has meaning or not - otherwise you'll break digital signatures
Lloyd McKenzie (Aug 19 2016 at 16:54):
Obviously if you're updating the instance, you're going to break the signature anyhow, so changing the order of things doesn't matter. Changing the order should only be done in an update with an awareness of what meaning (if any) is associated to the ordering of a collection.
Allan Bro Hansen (Aug 22 2016 at 11:30):
Lloyd, could you please elaborate on "otherwise you'll break digital signatures" - is it the hashCode of e.g. a Java-object?
My point is that the order is only mandatory to persist in the our system/database if the order matters. Likewise the order should not be guaranteed between multiple GETs.
So e.g. in my first initial example given that the order of business identifers does not matter: it is complety legit to - as a response to a GET or a PUT - return the business identifers in whatever order the server likes - and the order may even be different on the next GET.
Lloyd McKenzie (Aug 22 2016 at 21:39):
If you store an instance and then respond to a query, if the order of the elements on the way out isn't identical to what the order was when the data was signed, the signature will be broken. The FHIR canonical algorithm for signatures treats the order of all repeating elements as significant (because some of them are significant and differentiating would create more complexity than it's worth).
Allan Bro Hansen (Aug 22 2016 at 22:53):
Okay, now I understand that you are in fact addressing the XML signature, and the signature value of the XML document.
Still, I am talking about a POST or PUT operation in which the client has requested to get the created/updated ressource returned (Prefer-header with value return=representation). The returned resource will most likely be different than the one specified by the client: a logical id is added for a POST (if the resource is presisted), a version or lastUpdated meta-data will be updated (if included), other derived values may have changed. So changing the order of repeatable elements is just one of many possible differences between the post and returned ressource (unless logical id and meta-data are not included in the FHIR canonical algorithm for signatures?).
So to me it appears that changing the order of repeatable elements for a PUT/POST response is allowed altough not preferable.
The signature issue is also addressed here: https://www.hl7.org/fhir/updates.html#2.10.6.
Lloyd McKenzie (Aug 22 2016 at 23:02):
When we sign an instance, it's a canonical signature that can be used for XML or JSON. If you want to retain signature validity, the id would need to be pre-assigned (using PUT or as part of a Bundle or something). And certainly there are circumstances where a server might not retain everything as received. But the base expectation in FHIR is that order is retained. Changing the order is equivalent in impact to throwing away a couple of elements. It's within the rights of servers to do so, but doing so degrades the data received.
Allan Bro Hansen (Aug 22 2016 at 23:14):
ok, thank you for the clarification. We do not let the clients define the logical id of our ressource - so for a POST the signature will always be broken with the returned ressource (if a ressource is returned)
Lloyd McKenzie (Aug 23 2016 at 00:41):
We should probably talk about whether id should actually be part of the canonicalization algorithm because that's probably true of most servers - and it's not clear it should be part of the signed content.
Grahame Grieve (Aug 23 2016 at 00:49):
if id is not part of the signature, then links are not signed. There's almost no point then.
John Moehrke (Aug 23 2016 at 00:53):
I am not sure we have proven the point of signing FHIR resource instances... An assembled and serialized document, yes. Not much actual work and lots of questions http://wiki.hl7.org/index.php?title=FHIR_Digital_Signature_Working_Page
Lloyd McKenzie (Aug 23 2016 at 03:38):
You're signing the link, not the linked-to content. If you care about the linked-to content, then you'd need an extension on the reference containing a hash of the thing you're pointing to. Signing the ids inside a bundle makes sense, just not the root id of the instance.
Last updated: Apr 12 2022 at 19:14 UTC
