FHIR Chat · Snapshot Generation · conformance

Stream: conformance

Topic: Snapshot Generation

Grahame Grieve (Oct 19 2016 at 12:10):

@Chris Grenz @Michel Rutten I have committed all the changes from Thursday night discussion at Baltimore (svn revision 10025). Please check that there are no remaining issues

Michel Rutten (Oct 19 2016 at 12:46):

Will do!

Michel Rutten (Oct 19 2016 at 16:19):

I've downloaded the new definitions.zip from hl7-fhir.github.io. The contained xml files have timestamp 2016-10-18 10:45 - Is that the most recent version?

Lloyd McKenzie (Oct 19 2016 at 16:21):

Github hasn't been updating - see the committer's chat

Michel Rutten (Oct 19 2016 at 16:21):

OK, thanks Lloyd!

Michel Rutten (Oct 19 2016 at 16:22):

I'll try again tomorrow.

Chris Grenz (Oct 20 2016 at 20:01):

@Grahame Grieve Seeing the following differences to my implementation when comparing the Patient profile. Not sure all of these are agreed upon:

• Patient.id: base.max:"*" - appears often, should be max:"1"
• ele-1 constraint (and condition) not inherited, also ext-1. Is included on BackboneElement?
• No base element where element introduced, e.g. Patient.identifier - not sure if this is intentional? Thought base was to be included always to simplify implementations?
• Not including mappings from types (see Patient.identifier - Identifier type mappings not included)
• Patient.contact.id - comments not included

Grahame Grieve (Oct 20 2016 at 21:30):

found and fixed the base.max problem - was getting cardinalities from the parent not the child

Grahame Grieve (Oct 20 2016 at 21:30):

don'tf follow the ele-1 issue - it's appearing everywhere I think it should

Grahame Grieve (Oct 20 2016 at 21:33):

same for ext-1

Grahame Grieve (Oct 20 2016 at 21:34):

base: 'is provided when the element definition is not the original definition of an element'

Grahame Grieve (Oct 20 2016 at 21:34):

I don't think think constraints and mappings should be cloned out of the type - they apply, so they shouldn't be denormalised

Grahame Grieve (Oct 20 2016 at 21:35):

and I don't follow the issue about comments on Patient.contact.id

Chris Grenz (Oct 21 2016 at 14:38):

I think ele-1/ext-1 and comments are both how we're handling inheritance from the types - I grab everything from the types the same way as from base (type upstream from base). I think this was the consensus from Thursday as well as between @Michel Rutten and I. It's in line with the thinking that all things needed to validate an element should be included in the snapshot (if the element is present, all relevant constraints are included).

Chris Grenz (Oct 21 2016 at 14:40):

definition of base is unfortunate...makes a parser look in 2 places. We planned to use base.max in our JSON/XML converter to determine whether an element is an array - now have to look in 2 places and hope that a snapshot provided doesn't elect to omit base (it's optional?). Would rather see it mandatory.

Grahame Grieve (Oct 21 2016 at 20:11):

base is not optional. It must be provided if the definition is not the original definition

Grahame Grieve (Oct 21 2016 at 20:12):

I don't recall agreeing that we would inline everything from the types into the snapshot. In fact, what I remember is that I was passionately opposed to that, but we agreed that there are some cases in which it would make sense to inline everything into the snapshot, and so a snapshot generator could choose to do that. but I don't have time to do that as an option between now and my milestone deadline

Grahame Grieve (Oct 21 2016 at 20:13):

as it happens, I fixed a bug in my code around the dual location for max yesterday, so I can see the point of just making base mandatory

Grahame Grieve (Oct 21 2016 at 20:13):

do you want to propose a task?

Michel Rutten (Oct 24 2016 at 10:05):

• Concerning the merging of type profiles, I seem to remember from the Baltimore discussion that we agreed that this is the correct and desired approach. I specifically asked for confirmation that all elements should inherit the "ele-1" constraint from Element root, and @Gramahe I seem to recall that you agreed on this. IMHO, this makes the snapshot much more complete and useful. (Sorry Grahame...)

• Concerning Patient.contact.id - comment should be merged from the string datatype profile
  => "RFC 4122"

• Same for Patient.photo - comment should be merged from Attachment datatype profile
  => "When providing a summary view (for example with Observation.value[x]) Attachment should be represented with a brief display text such as "Attachment"."

• Concerning ElementDefinition.Base - IIRC we agreed that we only generate these for elements that are actually derived from a base, and not for newly introduced elements. (Sorry Chris...)

Michel Rutten (Oct 24 2016 at 10:10):

Note that if a profile specifies a constraint on a child element of a type profile, than the snapshot need to merge these onto the original child element definitions from the type profile. So we need to implement logic to merge type profiles anway, right?

Grahame Grieve (Oct 24 2016 at 10:22):

hey Michel. We agree that ele-1 would be on all types, including all anonymous types. I've made it so

Grahame Grieve (Oct 24 2016 at 10:23):

Chris wants more - he wants to inline content from the types to the containing models

Grahame Grieve (Oct 24 2016 at 10:23):

we agree that this could be a useful option for generating snapshots, but would not be necessary

Grahame Grieve (Oct 24 2016 at 10:24):

but your next 2 comments make it sound as though you think some information should be inlined. I don't agree

Grahame Grieve (Oct 24 2016 at 10:24):

I think I do type profiles correctly now

Michel Rutten (Oct 24 2016 at 11:27):

@Grahame Grieve I've noticed that fully merging type profiles has huge benefits in Forge & Simplifier. It makes it completely explicit to the end user which constraints are being inherited. Also this allows Forge to completely verify that your differential constraints are valid and don't conflict with any of the inherited constraints. For example, a profiler may not define a custom mapping on any element with id "ele-1". It also works great for extensions, by merging in the underlying extension definition. Without type profile merging, all of this becomes implicit and is no longer verifiable based on the snapshot. IMHO this conflicts with the ultimate goal of making everything fully computable.

Frankly, I don't really see a valid reason not to merge type profiles into the snapshot, except maybe for the additional burden on the library implementers. However many users will rely on the existing open source FHIR API libraries. And if not, then they can inspect the existing .NET/Java library source code as a guideline for their own custom implementation. I've found that the hardest challenge is solving the recursion problem. Eventually I've realized that you can avoid recursion by doing a breadth-first expansion. The .NET library currently implements a simplified version of this, just smart enough to handle Element.id and Extension.extension.

Considering that the FHIR .NET library is already capable of merging type profiles, and it provides such useful benefits in Forge & Simplifier, we'd really like to maintain and leverage our current implementation. We could easily make this behavior configurable, but imho providing various snapshot options will only confuse the community so I'm not a strong proponent of that.

I fully understand that you have serious time constraints and won't be able to implement this anytime soon. However imho this shouldn't hold us back from correcting/clarifying the spec. Maybe we can assist by generating official snapshots from the current .NET library, until the build can handle this?

Michel Rutten (Oct 24 2016 at 11:27):

Patient.structuredefinition.xml

Michel Rutten (Oct 24 2016 at 11:28):

Attached an example snapshot of the default DSTU-2 Patient resource by the current .NET API.

Michel Rutten (Oct 24 2016 at 11:40):

Taking a step back, if FHIR defines the snapshot component in such a form that by default, it does not provide sufficient information for profiling tools such as Forge & Simplifier, then what is the use of generating and including this? In that case, profiling tools would always need to implement their own differential completion logic anyway. Which seems to defeat the purpose of serializing the snapshot component. Right?

Michel Rutten (Oct 24 2016 at 13:22):

"Chris wants more - he wants to inline content from the types to the containing models"

What Chris and I are proposing is to always merge the constraints of the root element of the datatype. However datatype child nodes are only inlined if the user profile specifies any child constraints. So for example, Patient.identifier is merged with constraints from Identifier datatype root element definition. However e.g. Identifier.system is not inlined, unless the user profile specifies any constraints on children of Patient.identifier. In that case, the snapshot generator inlines all the child elements of the Identifier datatype and merges any user profile child constraints on top of them.

For example, primitive datatypes are merged with the base profile Element:
- Merge primitive root element on top of base profile root = Element (=> datatype root inherits "ele-1")
- Inline children of base profile = Element.id, Element.extension

So eventually, Element.id recursively picks up the "ele-1" constraint from Element root:
- string root inherits "ele-1" from Element snapshot root
- identifier root inherits "ele-1" from string snapshot root
- Element.id inherits "ele-1" from identifier snapshot root

Chris Grenz (Oct 24 2016 at 13:51):

@Michel Rutten Agree. If an element is included, it should include everything needed to validate. Note that this makes no statement about which elements to include.

Michel Rutten (Oct 24 2016 at 13:53):

@Chris Grenz Correct, that is what I meant to explain.

Chris Grenz (Oct 24 2016 at 14:28):

GF#12179 supports this viewpoint...moved by Grahame?

Michel Rutten (Oct 24 2016 at 16:48):

So my memory isn't failing me... thx Chris ;p

Michel Rutten (Oct 24 2016 at 16:53):

Anyway @Grahame Grieve considering your huge workload, I think that for now it would be sufficient to clearly define the desired behavior in the spec. I don't think it's that big of a problem if it takes a while for the official build to support this functionality. Forge & Simplifier will be able to generate their own (complete) snapshots anyway, even for core datatypes and profiles. Also the FHIR .NET API library can provide a customized version of definitions.xml.zip that contains complete snapshots. So delayed implementation in the build wouldn't keep us from progressing.

Michel Rutten (Oct 24 2016 at 17:00):

BTW I'd rather keep ElementDefinition.Base empty for newly introduced elements. IMHO this better conveys the intended meaning than a self-referential Base declaration, which to me seems a bit of a leaky abstraction. @Chris Grenz if it is more convenient for your implementation to always have a base, then couldn't you add a simple pre-processing step to fill in missing Base elements? This way, self-referential base elements remain an internal implementation detail of your logic and don't need to be serialized to the snapshot.

Chris Grenz (Oct 24 2016 at 17:01):

it's certainly possible either way - I don't find it a "leaky abstraction" - the base of a new element is itself.

Chris Grenz (Oct 24 2016 at 17:02):

The intent is to convey the original cardinality, not convey whether the element in hand is a new specialization or not.

Chris Grenz (Oct 24 2016 at 17:11):

And if it were, having base in all cases explicitly indicates whether a new element *is* the introduction/specialization - no further inspection needed.

Michel Rutten (Oct 24 2016 at 17:12):

@Chris Grenz I see your point, your suggestion does improve symmetry/uniformity. I don't have a very strong opinion on this. Just seems a bit noisy, but I guess I can live with it. Impact on our implementation is minimal.

Chris Grenz (Oct 24 2016 at 17:14):

Yeah, I wasn't crazy about base being introduced at all since it's redundant. But if it's there, might as well ensure it's consistent (which current technical constraints don't/cannot).

Lloyd McKenzie (Oct 24 2016 at 17:33):

I agree we should inline types when we can - the challenge is that type profiles might not all be available at the time the snapshot is generated. I don't think we want to make snapshot generation dependent on the availability of all profiles. That in turn means that snapshots may not generate the same each time. Is that something we're ok with?

Chris Grenz (Oct 24 2016 at 17:58):

I think we define a "standard" form and then describe the ways in which it may degrade (or expound) based on need. It may degrade primarily due to lack of access to dependent profiles. It could also degrade due to app specific needs (such as omitting descriptive elements) within the limits of the constraints.

Grahame Grieve (Oct 24 2016 at 18:05):

so I think you're using confusing language here:

Grahame Grieve (Oct 24 2016 at 18:07):

why would we inline some things from a type, and why would we not?

Chris Grenz (Oct 24 2016 at 18:08):

We would unless the type profile wasn't available for some reason

Grahame Grieve (Oct 24 2016 at 18:08):

firstly, are you claiming that if you don't duplicate the constraints out of a type, to the element, then they don't apply?

Chris Grenz (Oct 24 2016 at 18:08):

Not at all. The inverse - since they apply, they should be inlined.

Grahame Grieve (Oct 24 2016 at 18:09):

and if you claim that the do apply after all, then why inline *some* things from a type - which things? - and confuse matters by putting them in 2 places?

Chris Grenz (Oct 24 2016 at 18:11):

The proposal was that the element lineage passes up the resource lineage (e.g. Patient, DomainResource, Resource) and then into the type lineage (Patient.meta.tag -> Resource.meta.tag -> Meta.tag -> Coding -> Element)

Grahame Grieve (Oct 24 2016 at 18:11):

I don't understand how that is an answer to my questions

Chris Grenz (Oct 24 2016 at 18:12):

A "standard" snapshot should accumulate/inherit from all points in this lineage

Grahame Grieve (Oct 24 2016 at 18:14):

what we agreed was that (a) types should inherit properly - and I think i do that now - and (b) you could inline things from elements to types if you wanted

Chris Grenz (Oct 24 2016 at 18:15):

"if you wanted" meaning a snapshot form other than the "standard" ("canonical"? not sure what the best word is here) one.

Grahame Grieve (Oct 24 2016 at 18:16):

my memory of the discussion is that we agreed that you could inline into the snapshot if you wanted. Though I have no memory of talking about inlining *some* things from a type and not others

Chris Grenz (Oct 24 2016 at 18:16):

So in @Lloyd McKenzie 's case, if the type profiles weren't available, a legal snapshot could omit them, but they wouldn't match to the "standard" profile. The build was intended to implement this "standard" snapshot (which would for instance included ele-1 for all elements).

Chris Grenz (Oct 24 2016 at 18:17):

I don't remember that either - *all* "things" from a type should be inlined

Chris Grenz (Oct 24 2016 at 18:17):

which would include ele-1

Grahame Grieve (Oct 24 2016 at 18:18):

Lloyd's case beats me. It's degenerate. I can't generate or process things correctly when things are missing

Grahame Grieve (Oct 24 2016 at 18:18):

but you're saying that children elements are not inlined. so you're saying some things

Chris Grenz (Oct 24 2016 at 18:19):

the selection of *which* elements is a separate topic. For the ones that are included, they should include the complete elementdefinition (inherited from the entire lineage).

Chris Grenz (Oct 24 2016 at 18:21):

GF#12179 is explicit on the inheriting from types even including the ele-1 example...

Grahame Grieve (Oct 24 2016 at 18:22):

no. I have *no idea* what you and Michel are on about then. I inherit everything correctly, so far as I can tell, and you're talking about something else.

Chris Grenz (Oct 24 2016 at 18:27):

In the current build for Patient, the snapshot for the Patient.telecom element has no constraints. The lineage for Patient.telecom is (Patient.telecom -> ContactPoint -> Element). Therefore, the snapshot should include 2 constraints, cpt-2 and ele-1.

Grahame Grieve (Oct 24 2016 at 18:28):

this is *not* a matter of inheritance at all

Grahame Grieve (Oct 24 2016 at 18:29):

you are saying that information from the type should inlined in the element so that you can process it correctly without access to the type. But only some information, so you can only process it partially correctly

Grahame Grieve (Oct 24 2016 at 18:29):

sorry, but I'm not making sense of that

Chris Grenz (Oct 24 2016 at 18:30):

Your point is that since I don't include Patient.telecom.system, I'm not including *all* the type information?

Grahame Grieve (Oct 24 2016 at 18:31):

yes

Chris Grenz (Oct 24 2016 at 18:33):

I think our point was that selection of which elements to include was an independent decision from what is the correct form of a single element.

Chris Grenz (Oct 24 2016 at 18:33):

And that the contents of a single element in "standard form" should be character-wise comparable (possibly with some ordering concessions) with any generated snapshot.

Grahame Grieve (Oct 24 2016 at 18:34):

I disagree with that and what I remember is that the inclusion of any duplicative information was a choice for the generator, and that a snapshot comparision tool had to allow for that

Chris Grenz (Oct 24 2016 at 18:35):

What were you supporting in GF#12179 ?

Grahame Grieve (Oct 24 2016 at 18:35):

doing the inheritence correctly, which I had not been

Chris Grenz (Oct 24 2016 at 18:40):

If I created a Patient profile (my-patient) and constrained Patient.telecom.system min=1, would Patient.telecom *then* include cpt-1 and ele-1 (since all children of telecom would be included)?

Grahame Grieve (Oct 24 2016 at 18:42):

I don't follow your 'since'

Chris Grenz (Oct 24 2016 at 18:42):

once a child of telecom is included, all children would be included.

Chris Grenz (Oct 24 2016 at 18:42):

telecom.system, .value, .use, .rank, etc.

Grahame Grieve (Oct 24 2016 at 18:44):

oh yes. I mis read that. Yes, you would inline all of it in the snapshot.

Chris Grenz (Oct 24 2016 at 18:44):

so then cpt-1 and ele-1 would appear in Patient.telecom

Grahame Grieve (Oct 24 2016 at 18:52):

then they would, yes, because you inlining the data tyep

Grahame Grieve (Oct 24 2016 at 18:52):

and I did check that this works properly now too

Chris Grenz (Oct 24 2016 at 18:54):

you're adding things profile-by-profile...I'm adding them element-by-element. Our world-views are different....

Grahame Grieve (Oct 24 2016 at 18:56):

sufficiently different that I don't even know what you mean

Chris Grenz (Oct 24 2016 at 18:57):

I have all the profiles loaded into my tool and build all the lineages as I describe. I then generate snapshots by deciding which (fully populated) elements to include. I'm not loading or not loading additional profiles to add to the snapshot.

Chris Grenz (Oct 24 2016 at 18:58):

So my point of view was to agree upon what a *complete* element looks like.

Michel Rutten (Oct 25 2016 at 09:04):

"the challenge is that type profiles might not all be available at the time the snapshot is generated. I don't think we want to make snapshot generation dependent on the availability of all profiles. That in turn means that snapshots may not generate the same each time."

To quote Grahame, seems like we're moving the deckchairs around.

If we want to validate conformance against a profile that has references to external profiles, then we always need access to the information in those external profiles. If we allow the snapshot component to omit information from external profiles because they may be missing, then we put the burden to resolve the external profiles on the consuming application. If the application can't resolve external profiles either, then all bets are off and any computations against the profile are unreliable, esp. validation.

Therefore, I don't see the benefits of allowing a "partial" snapshot by defining type profile merging as optional. If an application cannot rely on the snapshot to provide all required structural and type information, then the whole effort of serializing the snapshot seems moot. Because in that case, profile-based applications will have to dynamically resolve and generate all the required information necessary to compute reliable results anyway. So then why serialize a useless partial snapshot?

Concrete, I propose:
- if the differential contains an element constraint with a single type, then for the snapshot we merge those diff constraints on top of the snapshot root element from the target type profile
- if the differential also specifies constraints on child elements, then we inline all the datatype child elements and merge the diff constraints on top of them

Note: the snapshot generator in the .NET API library tries to handle errors (such as unresolved external references) gracefully by trying to continue generating the snapshot, finally returning a partial, incomplete result and a set of operation outcome issues. This way, a client application can detect and interpret errors and decide how to deal with the returned partial and incomplete snapshot. If the generator reports any issues, then the generated snapshot is not reliable.

Grahame Grieve (Oct 25 2016 at 09:14):

what we said in baltimore was:
- generated types need to get inheritance right. I've done that now
- inlining the types would be optional. I'm not doing that right now, but I may get to it later (to make it optional)
- if you don't inline the types, any consuming software will have to resolve the types
- if you do inline the types, you could have consistency issues because editors won't have authority to update dependent profiles

I don't think that a minimal snapshot is useless. The hard part of snapshot is doing the actual snapshot; following a link and applying a new set of rules is really quite trivial by comparison.

Michel Rutten (Oct 25 2016 at 09:14):

According to general FHIR principles, I'd rather move the burden to the server implementers and resolve external type profiles at authoring time, when tools can provide useful feedback to the profile author.

Grahame Grieve (Oct 25 2016 at 09:15):

well, you can. But I object strenuously to making it always required

Michel Rutten (Oct 25 2016 at 09:16):

Off course in practice there will be situations where an external profile cannot be resolved. That's why the FHIR .NET API tries to handle this gracefully. But the resulting snapshot will be incomplete and unreliable.

Michel Rutten (Oct 25 2016 at 09:16):

"if you do inline the types, you could have consistency issues because editors won't have authority to update dependent profiles" @Grahame Grieve can you explain this?

Grahame Grieve (Oct 25 2016 at 09:19):

so Author A writes type profile Ta
Author B write resource profile Rb
Rb uses Ta, so it inlines Ta
Now Author A updates profile Ta to version 2.
Rb is now wrong

Michel Rutten (Oct 25 2016 at 09:39):

Yes, but IIRC FHIR states that if a new version of a profile introduces any breaking changes, then the new version should have a new, unique canonical url. Right?

- Author A creates Ta*, compatible with Ta
- Profile author B regenerates Tb* by inlining Ta*.
Then profile Tb* is compatible with profile Tb.
After all, the total set of resource instances conforming to Tb* is the same as the set that conforms to Tb.
So for profile validation, Tb and Tb* are interchangeable.
So author B is not strictly required to update the snapshot.
However if he chooses to, then the author is allowed to republish Tb* as a new version of Tb with the same canonical url.
- Author A creates Ta2, incompatible with Ta and Ta*
- Profile author B regenerates Tb2 by inlining Ta2.
Then profile Tb2 is incompatible with profile Tb and Tb*.
So profile Tb2 must be published with a new canonical url.

Grahame Grieve (Oct 25 2016 at 09:40):

actually, your interpretation of breaking changes is not correct

Michel Rutten (Oct 25 2016 at 09:40):

Can you please enlighten me?

Grahame Grieve (Oct 25 2016 at 09:41):

you've said that compatible means they are interchangeable for validation, but what we've said is that as long as instances conformant to Ta are also conformant to Ta*, then this is not a breaking change

Grahame Grieve (Oct 25 2016 at 09:41):

so they are not interchangable

Michel Rutten (Oct 25 2016 at 09:42):

Ah so Ta* is allowed to relax constraints of Ta, so a bigger set of resources would now be conformant?

Grahame Grieve (Oct 25 2016 at 09:43):

yes

Martijn Harthoorn (Oct 25 2016 at 09:52):

I think the definition of "breaking changes" is problematic. Since there will be definitions of both the old version and the new version - both with the same canonical uri living in the world. So from the author perspective, there is backwards compatibility. But from a collaboration perspective, this order is not guaranteed.

Ewout Kramer (Oct 25 2016 at 09:53):

Chris wants more - he wants to inline content from the types to the containing models

Yes, if a differential moves into a type, that type should be included in the snapshot. At least, that's the assumption I've been working with in my validator and the previous versions of the snapshot generator.

Grahame Grieve (Oct 25 2016 at 09:55):

no he wants something different to that. If you reference a type he wants to duplicate some information from the type into the profile - when you don't walk into it

Grahame Grieve (Oct 25 2016 at 09:56):

@Martijn yes the definition is problematic, but the converse - Michel's definition - is much more problematic, because it means that there's no forwards migration path that doesn't mean a rewrite for trivial changes. In CDA space, this has proven a major issue

Michel Rutten (Oct 25 2016 at 10:11):

- Author A publishes Ta* which relaxes Ta constraints
- profile Tb*, which inlines Ta*, describes a new version of the implemented system

- Assume that Sb represents the set of profiles that validate against Tb
Assume that Sb* represents the set of resources that validate against Tb*
So Sb* contains Sb and some additional resources

- I don't think that there is a reasonable expectation that a system that has implemented Tb will automatically support Tb*
Since it is not guaranteed that the system is able to handle the additional resources in set Sb*
So profile Tb* is not guaranteed to describe the actual capabilities of the implemented system
- So profile Tb still describes the guaranteed capabilities of the implemented system
So I wouldn't say Tb is now wrong.

Grahame Grieve (Oct 25 2016 at 10:11):

there is a reaonable assumption unless the reference was version specific

Ewout Kramer (Oct 25 2016 at 10:14):

so Author A writes type profile Ta
Author B write resource profile Rb
Rb uses Ta, so it inlines Ta
Now Author A updates profile Ta to version 2.
Rb is now wrong

This issue exists when the profile "walks into" Ta as well. Author B -when he wrote his profile- looked at type profile Ta when he produced his profile. and added applicable child constraints. It's debatable whether author B would like his profile to "automatically" accepts a broader set of instances if author A decides to change Ta by easing a constraint. Though I do agree that is our current methodology. It's even more debatable for developers, who probably want a "snapshotted" version at compile time of their software. Except maybe for some generic servers, I expect most developers don't want their software to automagically adapt if a type profile they depend on changes.

Michel Rutten (Oct 25 2016 at 10:14):

How can there be an expectation on a system to support additional resources that it wasn't originally designed for?

Ewout Kramer (Oct 25 2016 at 10:15):

Well, Spark could do that for example. But systems built on top of an existing database - no you can't expect them to do that.

Michel Rutten (Oct 25 2016 at 10:16):

Yes, I was thinking about e.g. a (non-dynamic) user interface such as an input form.

Grahame Grieve (Oct 25 2016 at 10:17):

is that how you design your software? clone type definitions so that when the source type definition is changed, you also have to manually update all the things that use it to tell them to use the new version?

Ewout Kramer (Oct 25 2016 at 10:45):

Yes, I fully expect Epic to work this way for example. They will implement Argonaut, but they will take a snapshot at the moment they implement and test the software. Any changes to dependent types will be done - but manually when the Epic team decides to update the dependent definitions. And this is the moment snapshots are re-generated.

But Spark - no - it will adapt immediately

Simplifier - yet another situation: we'd like to generate Snapshots once, not every single time someone requests the StructureDefinition. This requires dependency tracking.

Michel Rutten (Oct 25 2016 at 11:33):

@Grahame Grieve software design & implementation usually depends on the requirements and budget of the client. For specific use cases, I expect that systems implement a final, vetted version of the IG that both client and vendor/integrator agreed upon (and may even be set into law). In such cases, I don't think there is an expectation for the implemented system to support unforeseen future requirements. Otherwise, vendors would have to rethink their business model... ;p

Also, the term "snapshot" suggests that the component represents the state of some information at a specific moment in time (specified by StructureDefinition.date).

Chris Grenz (Oct 25 2016 at 14:29):

While I'm certainly interested in how we'll handle dependencies between profiles, determining the "standard" element format is a separate issue. ALL snapshots will have dependency management issues - we've been dealing with that for years at this point.
My question is: "Should an element included in a snapshot contain all known parts (mappings, constraints, aliases, etc.) including those parts inherited from base *and* type profiles?"

Lloyd McKenzie (Oct 25 2016 at 16:25):

Changing the canonical URLs of all impacted profiles as soon as there's a breaking change to one of them is a huge pain.

Grahame Grieve (Oct 25 2016 at 17:07):

@Chris Grenz the answer to your question is no. When you say that an element has a type, then you are saying that the value domain of the type applies to the element. It is not necessary to copy any of the value domain into the element that is assigned the type in order for those things to apply.

Grahame Grieve (Oct 25 2016 at 17:09):

you can choose to duplicate some information as a matter of publishing convenience but it should never be required to do so, and replicating some, but not all, is problematic for any consumer since they are required to find the source in order to know what you have and haven't duplicated

Chris Grenz (Oct 25 2016 at 17:15):

While I understand your position (I think), to the casual observer you are advocating just that - replicating some but not all of what's needed to evaluate/describe an element (not including anything "brought in" from the type). If we can be clear enough about that, I might be amenable.

I don't much care for though the contents of a single element changing based on (what looks like) the addition of child elements, e.g. the content of the element for Patient.maritalStatus changes if Patient.maritalStatus.system is included (since now everything from CodeableConcept is inlined). I understand *why* that happens, but it's subtle to an observer.

Grahame Grieve (Oct 25 2016 at 17:16):

I don't understand in which respect I am advocating some but not all

Chris Grenz (Oct 25 2016 at 17:17):

Again, you're "building up" the snapshot by applying whole profiles incrementally. I'm assembling the whole thing and then determining which elements to include.

Chris Grenz (Oct 25 2016 at 17:20):

If something forces the application of the type profile (including a child), then Patient.maritalStatus will include the ele-1 constriant. If not, it won't. What's included in an element depends on outside factors (even though the constraint is effective either way).

Chris Grenz (Oct 25 2016 at 17:20):

Running use cases in my head...

Grahame Grieve (Oct 25 2016 at 17:22):

I'm not sure what you mean 'application of the type profile'

Grahame Grieve (Oct 25 2016 at 17:22):

if the profile uses a type profile, and the adds additional constraints on it by walking into it, then the snapshot has to include all the type it references

Chris Grenz (Oct 25 2016 at 17:43):

Different mental model here - when you "walk into" the type, you apply the type profile at that point. I hold everything in memory/model and link together a lineage - Patient.maritalStatus -> CodeableConcept -> Element. I then build each element by aggregating the contents of the lineage. That why my element contents don't change when I opt to include more/less elements in the snapshot.

Grahame Grieve (Oct 25 2016 at 17:44):

so what's the difference between 'there are no children' and 'I didn't include the children'?

Chris Grenz (Oct 25 2016 at 17:44):

in the snapshot, nothing. By either of our methods.

Chris Grenz (Oct 25 2016 at 17:52):

In my model, not include Patient.maritalStatus.system is just suppression of extra detail. In yours, it's an effect of not considering ("walking into") the type at all. I think both are understandable (after extensive discussion!) - which would the average IT person assume?

Chris Grenz (Oct 25 2016 at 17:54):

maybe dependent on whether you're a modeler (my viewpoint) or an implementer (yours).

Chris Grenz (Oct 25 2016 at 17:54):

@Michel Rutten 's coming at it from a modeling standpoint as well...

Grahame Grieve (Oct 25 2016 at 17:56):

I think he's primarily looking at from a publishing view point - it's about dependencies. you've got an element focus, not a type focus. What I was establishing is that your interest is different to Michel's focus

Chris Grenz (Oct 25 2016 at 17:56):

True...

Grahame Grieve (Oct 25 2016 at 17:56):

I still think that we should stick to what we agreed in Baltimore: It's not necessary to inline profiles, but you can to the degree you find it useful. You and Michel have differing degrees to each (and to me)

Chris Grenz (Oct 25 2016 at 17:58):

Interesting how we can come out of meetings nodding together with different ideas in our heads!

Chris Grenz (Oct 25 2016 at 17:58):

I can get on board with a type focus....

Chris Grenz (Oct 25 2016 at 17:59):

breaks my tooling

Chris Grenz (Oct 25 2016 at 17:59):

badly

Chris Grenz (Oct 25 2016 at 17:59):

now I have to decide how deeply to navigate "up" the tree and stop (maybe) before I jump into the types.

Chris Grenz (Oct 25 2016 at 18:06):

@Michel Rutten - how does Forge work - do you decide which elements to include independently of calculating the contents of an element? Or do the elements "appear" as profiles are added (Patient.maritalStatus.system appears when CodeableConcept is added/applied)?

Michel Rutten (Oct 26 2016 at 11:21):

Forge goes further than a regular snapshot; when loading/creating a profile, Forge expands the children of all complex types, regardless of wether the profile specifies differential child constraints - so the author can select the children and constrain them. The fully expanded structure is kept in memory. A separate layer generates the differential by including only nodes with actual constraints.
On order to determine which elements to expand, Forge inspects the element type. If an element has a single complex type, then the children are being expanded (recursively).
Recently I've integrated the new and improved snapshot generation logic from the FHIR .NET API into Forge. I'm now reimplementing change tracking. When change tracking is finished, I want to leverage the new API snapshot generator logic to dynamically expand elements on the fly. So for example, if you constrain a choice type to single complex type such as CodeableConcept, then I'd like Forge to inline the child elements on the fly. Same for extensions. Also I want to implement a command to allow authors to manually expand elements with name references.

Michel Rutten (Oct 26 2016 at 14:59):

@Grahame Grieve I agree that merging a (partial) type profile should not (and does not) change the meaning of the element definition, as the merged constraints are implied by the type profile constraint. However it does affect the completeness/usefulness of the snapshot component.

@Chris Grenz:

 "I don't much care for though the contents of a single element changing based on (what looks like) the addition of child elements"

Agreed. Let's consider Grahame's approach for a profile with and without child constraints on Patient.identifier.system:

(1) Patient profile with constraint on identifier, but without any child constraints:

Patient.identifier => not merged

(2) Patient profile with constraint on identifier and additional child constraints:

Patient.identifier => merged onto Identifier
Patient.identifier.use => merged onto Identifier.use

So snapshot representation of the element "Patient.identifier" depends on the existance of nested child constraints...? So for example, element mappings and constraints inherited from the type profile root element are not merged into the snapshot, unless there are child constraints. This seems strange and could become a source of confusion.

I agree that for convenience, we should be able to generate a (partial) snapshot even if some external profiles are unavailable. However it should be clear to the author that the resulting snapshot will be incomplete.

Grahame Grieve (Oct 26 2016 at 20:22):

so the basic problem is that a user of the snapshot does not know when they need to consult with the type or not? I've been deciding that based on whether there is children or not - and automatically doing it for primitive types

Michel Rutten (Oct 27 2016 at 14:16):

Correct. I think the snapshot is generally more useful if individual snapshot ElementDefinitions always include relevant type constraints. Regardless of the presence or content of associated child constraints in the differential.
Maximizing completeness improves the usefulness of the snapshot, as well as minimizing variance. The latter also facilitates comparison.

Ewout Kramer (Oct 31 2016 at 10:05):

I think there are excellent patterns already in place for package management in Java and .NET (and others) - should we have something similar in the FHIR SD model?

Yes, so this is a different issue than the "merging" and "path walking" issues above. I don't want to confuse both here, but it's really also true I realized that our dependencies are comparable to what happens in package management for libraries - and that means that our version management can learn from that. We know one of the popular strategies recently is actually doing what I described: give every solution a private copy of the dependencies (and its dependencies). Though it's fair to say there are fierce discussions about what's the right solution there too.

Last updated: Apr 12 2022 at 19:14 UTC