Stream: conformance
Topic: Slicer vs Slice
Grahame Grieve (Mar 28 2017 at 21:07):
An issue has come up for Lloyd and I from doing test cases for snapshot generation that I'd like to get wider consensus about.
Grahame Grieve (Mar 28 2017 at 21:09):
it appears to be implicit - but not obvious - that if you define constraints on the slicing element (and sub-elements) - which are constraints on every slice - that any slices will be proper constraints on the slicer as well as the base structure definition
Grahame Grieve (Mar 28 2017 at 21:09):
is that something everyone agrees to? Should we make it explicit?
Grahame Grieve (Mar 28 2017 at 21:10):
the corrollary to this is the question of what happens in snapshot generation: should the snapshot generator copy constraints from the slicer into the slices in the snapshot?
Lloyd McKenzie (Mar 28 2017 at 21:10):
I don't see any alternative interpretation. We already agreed on a special name for a "default" slice, which is the only other interpretation I can think of for constraints declared on the slicing element and children. (And that would have been a bit odd.)
Grahame Grieve (Mar 28 2017 at 21:11):
thinking about how to do this - 3 way merge - makes my had hurt. And I wonder how anyone (well, any normal person, not including Lloyd) could ever understand what is going on here
Lloyd McKenzie (Mar 28 2017 at 21:11):
In the snapshot, I would expect that when you look at a particular element, you see all constraints that hold for that element - that's sort of the point. So I think we have to incorporate constraints from the slicer into the slices for it to work.
Lloyd McKenzie (Mar 28 2017 at 21:12):
(And thanks for implicitly including me in the set of 'normal person' ;))
Oliver Egger (Mar 28 2017 at 21:51):
i don't knwo if i categorize as a normal person, but i'm trying to understand whats going on here:-) i think implicit was my assumption to, how would I have it do to make it explicit?
Michel Rutten (Mar 29 2017 at 09:43):
Hi Grahame, interesting question!
We had an extensive discussion about this topic within our team about two weeks ago and decided on the following approach.
Differential element constraints are merged onto (override) the associated snapshot element definition in the base profile.
This also applies to slice entries and named slices.
Constraints on the slice entry apply to all the individual slices.
So a validator should also verify individual slices against the slice entry constraints.
However IMO this aspect does not effect the behavior of the snapshot generator.
Differential constraints on named slices are only merged with the associated element in the base profile - NOT with the slice entry constraints.
So there is no three-way merging.
It is up to the validator to verify named slices against the associated slice definition as well as the slice entry constraints.
Example 1
- the base element has an example binding to a valueset.
- The slice entry constrains the binding to a specific required valueset.
- Then in the snapshot, the named slices do NOT explicitly repeat the binding from the slice entry.
However they can (and often will) fix the element to a specific value.
Example 2
- base element has cardinality 0...1
- slice entry has cardinality 1...1 (mandatory)
- individual slices have cardinality 0...1 (optional)
Note that the individual slices should NOT inherit minimum cardinality 1 from the slice entry...!
Otherwise a profile can no longer introduce options.
So a named slice always has a base minimum cardinality of 0, regardless of the minimum cardinality of the associated base element.
Michel Rutten (Mar 29 2017 at 10:05):
Michel Rutten (Mar 29 2017 at 10:06):
Michel Rutten (Mar 29 2017 at 10:06):
Hope this helps.
Lloyd McKenzie (Mar 29 2017 at 14:04):
@Michel Rutten What's the point of declaring constraints on the slicing element if those constraints aren't enforced on all slices? I don't understand why a profile would declare constraints that would affect anything other than derived profiles but would have no impact on validating instances against that profile. (Keep in mind that we have a separate way of declaring constraints on elements that don't match any of the named slices.)
Michel Rutten (Mar 29 2017 at 14:23):
But the constraints on the slicing entry SHOULD be enforced! This is the responsibility of the validator. However in the snapshot representation, slicing entry constraints are not merged onto named slices. These are different aspects and one does not affect the other. This way, the author has complete freedom to author constraints on slicing entry level and/or named slice level. It is also up to the validator to verify that slicing entry constraints don't conflict with constraints on named slices. However, a profile with conflicting constraints is not automatically invalid; it's just that no resources will match the profile, so it is probably not what the author intended. A validator could e.g. flag this with a warning.
Lloyd McKenzie (Mar 29 2017 at 14:26):
That's kind of evil if you ask me. The whole point of the snapshot is to say "these are all the rules that apply here". Needing to look in multiple places makes things harder. That said, the cardinalities have to be able to be different, so there are certainly some propagation rules that aren't the same here as they are when propagating from base.
Michel Rutten (Mar 29 2017 at 14:27):
Suppose that the snapshot generator would merge the slicing entry constraints on the named slices. This implies that named slices now inherit constraints from an element in the same profile (instead of an element in the base profile like everywhere else). That's like trying to hit a moving target...
Also, suppose the slicing entry constrains cardinality from 0...* to 1...*. Now if we would merge these onto named slices, then the profile can no longer introduce an optional named slice with cardinality 0...1. This would prevent profiles to model common use cases.
Lloyd McKenzie (Mar 29 2017 at 14:30):
Merging the cardinality in is the one thing you can't do. But I don't see how it's hitting a moving target. The snapshot reflects whatever the differential says. As the differential changes, the snapshot will change. Merging the slicing and slice rules together doesn't change that aspect
Michel Rutten (Mar 29 2017 at 14:31):
I realize that this is quite subtle. Initially, we tried to implement the proposed behavior (merge slice entry constraint onto named slices). But we received feedback from our customers who found the behavior unpredictable and counterproductive. So we had a team discussion about this topic and analyzed a number of differerent use cases. This resulted in the current behavior, as I described above.
Michel Rutten (Mar 29 2017 at 14:34):
Yes, snapshot generation is relatively straightforward. But e.g. in Forge, we need to dynamically maintain references to the correct base element in order to determine what information to include in the differential. It turns out that these in-memory base element references are not 1-1 equal to snapshot element definitions, as Forge also needs to merge external type profiles and extension definitions. This becomes much harder if the base element references point to another element in the referencing profile itself.
Michel Rutten (Mar 29 2017 at 14:36):
Also note that Forge needs to expand *all* elements, even without child constraints, in order to generate a full element tree. This involves an additional processing step after snapshot generation.
Michel Rutten (Mar 29 2017 at 14:38):
So although the snapshot is definitely useful, it does not provide *ALL* information. In general, this is not even possible due to recursive relations (i.e. Identifier.assigner / ResourceReference.identifier).
Michel Rutten (Mar 29 2017 at 14:38):
Ewout's validator e.g. dynamically pulls in external type profiles on demand.
Michel Rutten (Mar 29 2017 at 14:38):
So there is always a need for additional processing.
Lloyd McKenzie (Mar 29 2017 at 14:53):
Does Forge ensure that constraints in the slices don't conflict with constraints in the slicing element? What about descriptive information? Do slices inherit the generic descriptions of the parent instead of the more specific descriptions of the slicing element?
Michel Rutten (Mar 29 2017 at 14:59):
Both the API snapshot generator as well as Forge do NOT actively merge slice entry constraints onto named slices. However Forge already provides a number of validation warnings for insensical/conflicting constraints, and we will improve this continously in future builds.
Michel Rutten (Mar 29 2017 at 15:04):
Slice entries and named slices inherit constraints from the associated element in the base resource. Note that this only affects what is included in/excluded from the differential. Named slices in the same profile inherit constraints (including e.g. description, but specifically excluding min cardinality) from the associated element in the base profile. This can be an unsliced element, a slice entry or a named slice. But elements never inherit constraints from other elements *in the same profile*, and a named slice also never inherits from the slice entry *in the same profile*.
Michel Rutten (Mar 29 2017 at 15:05):
Again, all this does NOT imply that constraints on a slice entry don't apply to named slices - this still holds. It only affects the contents of differential and snapshot representations.
Lloyd McKenzie (Mar 29 2017 at 15:07):
If I override a definition or usage notes in the "slicing" section, it's going to be somewhat annoying to have to do the same override on every single slice (or to end up inheriting the documentation of the base)
Michel Rutten (Mar 29 2017 at 15:08):
If you want to explicitly include inherited slice entry constraints in the profile, then you can create a base profile with a slice entry (but no slices) and a derived profile that actually introduces named slices. Then the named slices in the snapshot of the derived profile WILL inherit constraints from the slice entry in the base profile.
Lloyd McKenzie (Mar 29 2017 at 15:09):
Creating extra profiles is rather kludgy
Michel Rutten (Mar 29 2017 at 15:09):
I see your point. It definitely helps to get a clear picture of all the different use cases. That's how we finally realized that minimum cardinality of named slices should not follow normal inheritance rules.
Lloyd McKenzie (Mar 29 2017 at 15:10):
Sure. I get the cardinality difference. But I don't see any other negative impacts. What other impacts did your users report?
Michel Rutten (Mar 29 2017 at 15:17):
Well they found the (old style) snapshot merging to be confusing.
Michel Rutten (Mar 29 2017 at 15:18):
Specifically, the NHS has been a tremendous help in testing all the different slicing scenario's and use cases.
Michel Rutten (Mar 29 2017 at 15:20):
They have built a custom profile rendering tool that depends on the snapshot. Their experiences with publishing (complex) profiles led to a number of change requests for Forge and the API.
Lloyd McKenzie (Mar 29 2017 at 16:40):
Given that the merging *does* occur from a validation perspective, I'm curious what aspects they found confusing
Chris Grenz (Mar 29 2017 at 19:13):
Late to the game...I agree with @Lloyd McKenzie (see here: https://github.com/chrisgrenz/FHIR-Primer/wiki/Snapshots-Determining-Refines#new-specializations) - a named slice (e.g. MRN) refines it's slice root (e.g. Patient.identifier) and accumulates any constraints there, in this case a combination of Element, Identifier, Patient.identifier, and slice MRN. A reslice (MRN/MyLegacyEHR) refines it's root (MRN) and would accumulate from there. How to accumulate is already pretty well defined I think? Not sure where @Grahame Grieve is seeing a 3-way merge? It's straight accumulation up the path.
Lloyd McKenzie (Mar 29 2017 at 19:32):
3-way merge is base + slicing root from differential + slice from differential
Chris Grenz (Mar 29 2017 at 19:40):
That's not 3-way, that's 2-way twice (slice + slice root, combination + base). We already do this *lots*.
Lloyd McKenzie (Mar 29 2017 at 19:44):
Well, 2 2-way merges is still a 3-way. It's just a question of how you architect it. It actually gets a tad more fun if you're constraining existing slices. In that case, it's a 4-way merge - New slice snapshot needs to include the intersection of the original slice differential + the application of new slice base on old slice base + new slice constraints. It's not overly pretty to do.
Lloyd McKenzie (Mar 29 2017 at 19:45):
But it's even less pretty for downstream systems to have to integrate the slice base constraints and slice constraints every time they need to use the snapshot
Chris Grenz (Mar 29 2017 at 19:46):
It's more than that. You also have to include constraints from the types. And yes, the snapshot tooling should do all that up front - downstream systems shouldn't be dealing with *any* of this complexity
Lloyd McKenzie (Mar 29 2017 at 19:54):
Right. It's an evil thing to do, so do it all in one place. I'd really like more information about where the NHS had concerns with doing this. Any insights @Richard Kavanagh ?
David Hay (Mar 30 2017 at 01:16):
Can I just confirm (in my simple clinFHIR way) Lloyds comment that a snapshot represents "all the rules that apply here". The details of this discussion are a bit deep for me, but I just want to be sure that when I get StructureDefinition resource, the snapshot is all I need to build the UI...
Michel Rutten (Mar 30 2017 at 08:21):
@David Hay The Forge UI needs to expand all elements with a complex datatype - so the user can define new constraints on child elements. The snapshot does not contain sufficient information, as complex elements are only expanded when the profile introduces differential constraints on child elements. So Forge needs to perform post-processing of the snapshot anyway, this is unavoidable.
Michel Rutten (Mar 30 2017 at 08:25):
And consider nested content references. Such a profile can be expanded to an infinitely deep nesting level. Those elements always need to be expanded on demand.
Michel Rutten (Mar 30 2017 at 08:35):
@Chris Grenz, @Lloyd McKenzie concerning the merging of sliced elements in the snapshot, I strongly recommend that we always resolve the merge base from the (external!) base profile. Resolving base elements from the same differential introduces a can of worms, as now the base is no longer stable. Please keep in mind that not all logic works as a pipeline. A profile editor such as Forge needs to dynamically process changes throughout a profle in any order. Specifically, a user could create a named slice and subsequently go back and edit the constraints on the associated slice entry. According to the proposed merging scheme, all the existing named slices would then need to be re-merged with the updated slice entry, while handling all kinds of possible conflicts. I actually tried this and it proved to be extremely complex (as in: unfeasible) to implement correctly and confusing for the end user.
Michel Rutten (Mar 30 2017 at 08:40):
Ewout suggested that Forge could assist the modeller by automatically initializing newly created named slices by duplicating the constraints on the associated slice entry. All constraints on named slices would then be included in the differential and in the snapshot, as they are not actually inherited from the base.
Michel Rutten (Mar 30 2017 at 08:42):
This approach would provide a convenient workflow to the modeller, while still keeping the complexity of snapshot generation to a feasible/reasonable level...
Michel Rutten (Mar 30 2017 at 08:46):
Again, it is important to keep in mind that not all systems can be implemented as a pipeline. If we keep Introducing additional complexity in the snapshot generation rules, then the implementation of dynamic systems (such as editors) becomes unfeasible.
Michel Rutten (Mar 30 2017 at 08:50):
Therefore I'd strongly recommend for base elements to *always* be stable, i.e. resolved from external profiles.
Michel Rutten (Mar 30 2017 at 08:53):
Of course, a modeller can create a derived profile and then update the underlying base profile. Forge can detect this, re-load the derived profile and re-generate the snapshot. So this will kick of the pipeline, so to say.
Richard Kavanagh (Mar 30 2017 at 12:03):
@Lloyd McKenzie We dont have any specific issues regarding re-slicing at the moment but that's probably because we dont use that capability yet. My understanding is that this is only an issue if you "profile a profile" something which we do not yet do. We definitely have a requirement for this though concerns around tooling have meant we have deferred this for the moment.
Lloyd McKenzie (Mar 30 2017 at 17:20):
@Michel Rutten It doesn't matter what you do with your in-memory representation at runtime. What we're talking about here is what's done when you persist. And when you persist, all of the merging of the different pieces that come into play should have been performed. Downstream applications who receive a snapshot shouldn't have to do any merging at all. (Because merging is complicated and ugly and we want to limit the number of pieces of software that have to deal with that as much as possible.)
Lloyd McKenzie (Mar 30 2017 at 17:21):
The resource is, by definition, a pipeline because it represents how data is exchanged. It's not a representation of what you do dynamically in memory.
Lloyd McKenzie (Mar 30 2017 at 17:21):
@Richard Kavanagh What I'm actually interested in is not re-slicing, but rather why you would not want the snapshot view of a slice to include the rules that you've set for all slices.
David Hay (Mar 30 2017 at 20:40):
Yes, that what I was checking - that the snapshot in the saved resource was complete...
Richard Kavanagh (Mar 30 2017 at 22:11):
@Lloyd McKenzie not sure I have a view on this. It's really not an area of FHIR I am dealing with.
Lloyd McKenzie (Mar 30 2017 at 22:30):
@Michel Rutten Who from the NHS was objecting to the integration of the slicing base into the slices in snapshot? And was that driven by runtime behavior (which doesn't really matter for this conversation) or for persistence?
Grahame Grieve (Apr 03 2017 at 03:34):
@Michel Rutten reading this thread makes it clear that these kind of issues should be discussed more widely than just in your team. Will save you (and me) from having to re-work...
Grahame Grieve (Apr 03 2017 at 03:38):
Michel, I'm confused by your description of the problem in Forge. What Lloyd proposed is that the *snapshot* merge all this stuff. Are you implying that the user edits the snapshot?
Michel Rutten (Apr 03 2017 at 10:56):
I've given this a lot of thought. Tomorrow I'm going to discuss the issue with Ewout. Then we'll get back on this.
Grahame Grieve (Apr 03 2017 at 10:59):
ok
Lloyd McKenzie (Apr 06 2017 at 03:46):
@Michel Rutten @Ewout Kramer How did the discussion go?
Michel Rutten (Apr 06 2017 at 08:08):
Hi @Lloyd McKenzie we've had a fruitful discussion about this issue. I've written down our findings in a presentation that I'd like to share. As all this is quite involved, Ewout & I suggest that we schedule a Skype call to discuss this with you, Grahame, Chris and anyone else interested in this topic.
Lloyd McKenzie (Apr 06 2017 at 15:02):
Sounds good to me. Perhaps send a doodle poll out to the FHIR list? (and cc it on this thread)
Michel Rutten (Apr 06 2017 at 15:12):
Sure, we could do a community poll - for the happy few that understand the topic...
Brian Postlethwaite (Apr 07 2017 at 22:22):
I'll be interested to join too.
David Hay (Apr 08 2017 at 00:35):
Me too ...
Eric Haas (Apr 08 2017 at 01:46):
+1
Oliver Egger (Apr 08 2017 at 16:57):
+1
Michel Rutten (Apr 10 2017 at 08:45):
Maybe we can discuss this topic in today's infrastructure call?
Michel Rutten (Apr 10 2017 at 08:45):
Mon Apr 10, 2017 03:00 PM (US Eastern Time, GMT -4 DST)
Please consult http://www.timeanddate.com/worldclock/fixedtime.html?iso=20170410T1500&p1=784 for your local times.
Michel Rutten (Apr 11 2017 at 08:15):
FYI: the discussion about slicing base has been scheduled for the FHIR-I call on Monday April 24th. Feel free to join!
Last updated: Apr 12 2022 at 19:14 UTC
