Stream: implementers
Topic: Are contained resources evil?
Lloyd McKenzie (Feb 24 2021 at 15:29):
James Agnew said:
Not to change the subject here but that example just says to me that the MedicationRequest resource is fundamentally broken, if we have a common scenario like this one that can only be achieved using contained resources. IMO even extensions on the MedicationRequest would be vastly preferable to contained elements, which satisfy the "modelling purity" crowd but are horribly hostile to implementers.
I don't understand this. Why are contained resources problematic for implementers? It reflects existing design. If what you've got in your database is a recipe stored with your prescription, you're going to have to ship it around 1..1 with a prescription. While we could define a complex extension that duplicates all of the information in the Medication resource, I don't understand how that's somehow better than just leveraging the existing data model and saying "do this same thing, but treat it as contained". Can you expand @James Agnew?
Kevin Mayfield (Feb 24 2021 at 15:53):
Do you have context behind this point. However I have viewed using contained as difficult for developers, especially on resources being used as events and in workflow.
Most contained examples I've seen have been trying to follow restful paradigm when really message interactions should have been used. For UK events we can do away with contained by using reference identifiers instead.
James Agnew (Feb 24 2021 at 15:55):
Heh, fair question. My main issues are these:
-
They suck for client developers. The fact that instead of a clean and consistent pattern (thing A here refers to thing B over there) you have the possibility of a second pattern (thing A refers to a different part of thing A where they are tied together via a non-globally-unique ID so you need to iterate over the list of those things until you find one that has a matching ID) is just unpleasant code to actually write.
-
They suck SO BAD for human eyeballing of resources. The second contained resources come into play, a resource goes from being easy to grok at a glance to being almost impossible to follow unless you have a giant monitor and great eyes. All of the advantages of FHIR's clean and shallow structure are instantly out the window.
-
Indexing them is really hard to get right as a server developer. It has to be telling that absolutely no servers other than Grahame's have implemented support for indexing them (although HAPI and many of the others are now being forced to because the IGs went ahead and mandated support despite a complete lack of capability from the available tooling)
-
They are easily misused. I know others have had different experiences, but my own experience is that "in the wild" I would say that 90% of the contained resources I've seen do not pass the actual criteria of "only use this if you can't assign an independent identity". People quite commonly use contained resources as a way of turning any resource into a Bundle resource.
Lloyd McKenzie (Feb 24 2021 at 16:05):
Sorry, thought my quote included a link to the original stream: https://chat.fhir.org/#narrow/stream/179166-implementers/topic/Profiles.20requiring.20meta.2Eprofile.20with.20version
Jean Duteau (Feb 24 2021 at 16:32):
So I'm currently writing a guide where we made a conscious decision to use contained resources. We have a system that is registering Organizations and those Organizations can have sub-Organizations and a set of HealthcareServices. Normally, this would be a set of Organization, OrganizationAffiliation, and HealthcareService instances in a Bundle and that was how we had it modelled at first. But we switched to contained because:
1) The HealthcareService and sub-Organization instances didn't have a life of their own. Even though they represented separate Organizations, the architecture of the system was that if I deleted the parent Organization, I had to delete the sub-Organizations, OrganizationAffiliations, and HealthcareServices. As opposed to just leaving the sub-Organizations alone and just deleting the sub-Organizations, OrganizationAffiliations and HealthcareServices.
2) Again, the architecture of the system meant that if two different parent Organizations had a relationship to the same sub-Organization, they didn't search for and re-use that Organization, they actually just sent new information for the sub-Organization. And that new information could actually be new. Even if two different organizations had the same business identifier, it was possible for contact information, name, address, telephones, to actually be different if the organization was widespread. The business identifiers weren't down at a sub-department level but the rest of the Organization information was.
Given those two points, we decided that we needed to send the resources as a set of contained resource on the parent organization. If we were designing the system from scratch, we wouldn't do this and we would actually impose some rules around the re-use of the child organizations. But since we are architecting a system to replace and co-exist with an existing system, we are unable to do that.
So although we could force an independent existence to our contained resources, it didn't make sense for our system to do so - it in fact would cause more work for the clients and system if we did. Am I saying that every IGuide author puts this level of thought into the decision? No, but let's make sure that we don't paint too broad a brush over authors who do. :)
Rik Smithies (Feb 24 2021 at 16:46):
I agree with Jean and can see good uses for contained. And when deciding if they are evil we need to consider if the alternatives are worse.
René Spronk (Feb 24 2021 at 17:02):
Contained is a necessary evil - interoperability is full of those..
Lloyd McKenzie (Feb 24 2021 at 17:10):
@James Agnew
-
There are 3 possibilities - data is always referenced, data is always contained, or data could be either. Most "real" implementations will fall into one of the first two buckets. The pain really only manifests in the last. (Indexing the contained resources by id is a super simple thing to do, so I don't understand the pain of iteration?) If you have a situation where sometimes you've got a 'partial' non-independent resource and sometimes you've got a proper reference to an independent object, then you've got real complexity, either because your business space is hard, or because you're trying to be a generic system that supports a variety of architectures. The pain isn't coming from 'contained', it's coming from the architectural reality of varying data
-
There are four things that interfere with readability - contained resources, fat narrative, base64-encoded data and Bundles where you've got a super-large set of content. Yes, those suck to read raw. But that doesn't mean we take those features away because they suck for humans to scan. If you've got a decent JSON or XML viewer, it's easy enough to collapse those sections (they're all nicely isolated), so I don't know I see this as a huge issue. Also, on the balance side, with non-contained, if there's a reference, you've got to go look it up. With contained, that's an easier process. (Not that that's a reason to contain or not contain, just a consideration on 'easier to read'.)
-
Again, I think this is an issue with "generic" struggling with different architectures. If I've got a database that stores prescriber license number as part of the prescription, my indexing is easy. If I've got a database that stores prescriber as a foreign key to a separate table that is indexed by license number, my indexing is easy. In the real world, there are systems with both designs. If I need to accommodate both types of data, then yes, my life gets hard. But that's not because of 'contained' but because of the need to support the mix of contained and non-contained
-
Yup. That's a problem. However, all sorts of things are subject to abuse too. Custom extensions being used when there's a standard extension or core element or Observation that should be used instead; people doing horrible things with custom codes (changing meaning, not defining namespaces, etc.), abusing meta.profile. It's hard to stop abuse other than education and documentation. But the fact something can be abused doesn't mean that use-case that drives its existence is invalid. Maybe we should have a "suspect elements" report that gets generated on IGs we publish that highlights elements commonly abused (meta.profile, all extensions, all contained resources) so that we can more efficiently review and scan for improper designs?
Gino Canessa (Feb 24 2021 at 18:03):
Without knowing more about the Jean's system, I'm not sure if it is relevant - but how do I look for a local sub-organization?
Many servers do not support search on contained*, so I'm stuck pulling down all of the organizations and searching through them myself. What's worse, without an indexing structure I have to pull all of them and search through every sub-organization to discover this.
*(1)as James mentioned
*(2)having glanced at this recently, I'm not sure this is defined well enough that it will be implemented consistently (when it is implemented)
Since I took long enough to type this, I can factor in @Lloyd McKenzie's most recent message too =). I feel like you are vastly understating the issue of "generic". Nobody wants to build in custom support for every IG based on its containment rules. Contained takes a resource and adds "sub content", which can be of any resource type, which IGs then add things like SearchParameters and Operations to. It essentially creates an additional relation map rooted in every resource. I think this is even more complicated if you consider facade servers, since they need to map their internal structures to FHIR as an additional step as well.
Jean Duteau (Feb 24 2021 at 18:10):
Gino Canessa said:
Without knowing more about the Jean's system, I'm not sure if it is relevant - but how do I look for a local sub-organization?
Many servers do not support search on contained*, so I'm stuck pulling down all of the organizations and searching through them myself. What's worse, without an indexing structure I have to pull all of them and search through every sub-organization to discover this.
For our system capabilities, you don't. That was also one of the factors that went into our architecture decision. The HealthcareService, OrganizationAffiliation, sub-Organizations are really just additional information about the parent Organization. We did try and take into account what FHIR servers can do with contained resources when we made our decision.
Gino Canessa (Feb 24 2021 at 18:17):
Good to know! If every IG took this much care, we probably wouldn't be having this discussion =)
Shamil Nizamov (Feb 24 2021 at 20:50):
James Agnew said:
- They suck SO BAD for human eyeballing of resources. The second contained resources come into play, a resource goes from being easy to grok at a glance to being almost impossible to follow unless you have a giant monitor and great eyes. All of the advantages of FHIR's clean and shallow structure are instantly out the window.
I wonder if SAFIR (Quebec) has ever contacted you before designing their ProcedureRequest based CRDS solution with tons of contained resources!
Gino Canessa (Feb 24 2021 at 21:04):
As a follow-up (@Jean Duteau), does the IG surface any of that information via 'other' methods (e.g., Search Parameters, Operations, etc.)?
Grahame Grieve (Feb 24 2021 at 22:02):
@James Agnew for all the reasons you've said, we detest contained resources, no question about that. But the problem that forced them to exist is going to create all those problems no matter what we do
James Agnew (Feb 24 2021 at 22:26):
Sure - To be clear I'm not arguing they should go away. Mostly that we should be questioning their use more..
The classic use case I use when teaching FHIR is "i have a bunch of lab tests but I only know the patient's name and not an identifier for them, so a contained resource is a good way to keep the name with the observation". That use case makes perfect sense, and I can't think of any less messy way to do it. Truthfully I've never seen this happen in practice, but I assume it must happen occasionally..
For the "i have a medication order and the exact blend of ingredients is unique to the order", I still feel like that sounds like a massive shortcoming of the medicationrequest resource if we need a contained resource for a routine scenario. We should address that shortcoming rather than accepting contained resources as the "right way".
Jean's scenario I don't even know what to make of.. In that case contained resources aren't even following Lloyd's stated hard rule about when it's appropriate to use a contained resource, they're being used to imply a workflow about when parents and children should be automatically deleted. I have no doubt the use case is valid - it's crazy to me though that contained resources are the right solution.
Matt Ping (Feb 24 2021 at 22:33):
At least what I've found is that more and more there are a number of FHIR-related specifications out there. For instance DaVinci handles a lot of Provider-Payer interactions. In those cases a Bundle or Parameters object encapsulates the same business functionality that you're going for by that kind of contained usage. DaVinci is just one example and not every use case would be solved. However, I've noticed a fair number of things have started popping up for specific "business transactions"
Grahame Grieve (Feb 24 2021 at 22:36):
Mostly that we should be questioning their use more..
Amen. @Lloyd McKenzie we should make this part of a standard FMG review of an IG. I should add 'use of contained resources' to the IG QA report. What's an example of an IG that uses contained resources?
Jean Duteau (Feb 24 2021 at 22:43):
http://build.fhir.org/ig/HL7/fhir-spl/branches/main/index.html
Jean Duteau (Feb 24 2021 at 22:52):
James Agnew said:
Sure - To be clear I'm not arguing they should go away. Mostly that we should be questioning their use more..
Jean's scenario I don't even know what to make of.. In that case contained resources aren't even following Lloyd's stated hard rule about when it's appropriate to use a contained resource, they're being used to imply a workflow about when parents and children should be automatically deleted. I have no doubt the use case is valid - it's crazy to me though that contained resources are the right solution.
I agree with you. As I said, we started with a Bundle of resources. But then we realized that the existing operations that we were trying to replicate didn't even acknowledge the presence of those child resources. Clients don't just update the contact info on a child organization. They update the parent organization and include the child organization details.
To be even mind-blowing, we actually have one operation where you do send us a Bundle of Organizations and OrganizationAffiliations where the relationship is managed independently. And within that Bundle, the "participatingOrganizations" (to use OrgAffiliation speak) have contained resources.
Like I said, we did put a lot of thought into making the decision to use contained resources and realized that it is the right tool for our problem.
Lloyd McKenzie (Feb 25 2021 at 00:16):
In general, IGs should not be forcing content to be contained unless it's for something esoteric like "anonymized submissions" where containment is part of the process. They might require support for containment where the source systems can be expected to not have independent identity for the resource in question.
Lloyd McKenzie (Feb 25 2021 at 00:18):
@Grahame Grieve Can we flag all three trouble spots I listed? contained resources, examples with profile declarations (or profiles that force values for meta.profile), and a list of extensions defined?
Jose Costa Teixeira (Feb 25 2021 at 04:18):
We can make those IG rules, but what about the core spec? We have in the core spec some resources that require/stimulate contained resources even in normal usage.
Grahame Grieve (Feb 25 2021 at 04:24):
how so?
Jose Costa Teixeira (Feb 25 2021 at 05:30):
https://chat.fhir.org/#narrow/stream/179308-BRR--.20Pharmacy.20model/topic/Ingredient
Grahame Grieve (Feb 25 2021 at 06:01):
oh. well, a design trade-off, rather than any specific things we say about contained resources
Kevin Mayfield (Feb 25 2021 at 09:04):
@James Agnew We are working on those two scenarios (unsolicited-observation and prescription-order) at the moment.
We are classing both as FHIR Events. It's not clear whether FHIR http or Message is the preferred style.
As FHIR Messaging, the preference appears to be FHIR Message with a single focus resource and Patient is either referenced (via identifier) or is included in the Bundle.
If majority want Patient included with the event, it's likely we will settle on FHIR Message for events.
Kevin Mayfield (Feb 25 2021 at 09:08):
It's patient identifiers causing the move towards Messaging (and is the primary reason for contained).
Would it not be easier to change the cardinality of FHIR Reference.identifier from 1..1 to 1..* ?
James Agnew (Feb 25 2021 at 10:40):
Certainly my point here is that we shouldn't just accept contained resources as a "design tradeoff".
For all the points I listed, I personally consider them to be completely hostile to implementors. Sure they are a last resort in some cases and we can't do away with them, but if our core models require their use in order to model a common scenario, that on its own should be a signal to the author of the core models that they need to go back to the drawing board.
Grahame Grieve (Feb 25 2021 at 12:17):
though "require" is not the right word for @Jose Costa Teixeira 's case. The general tradeoffs make contained resources easier in this particular case. But not at all necessary. I think.
John Moehrke (Feb 25 2021 at 13:24):
The use-case in IHE for XDS/MHD where contained resources are required are where mostly where the contained resource data is desired to be kept at a the values that were known at the time the DocumentReference was created. (e.g. SourcePatientInfo), where as the subject is intended to point to a maintained identity. The second has a similar need for "at the time" characteristic, but also the use-cases tend to not have full details, usually just an identifier (which could have been reference.identiifer) but might have more (so contained).
I support IG warning flags, that I can override with purpose.
John Moehrke (Feb 25 2021 at 13:26):
and we have had enough pushback that there is a named Option in MHD for "UnContained", for use in a community where these data are managed and it is desired to be managed.
Daniel Venton (Feb 25 2021 at 15:29):
If the desire is to know a state in time, that's what vread and versioned resources are for. You reference the version of the resource that was in play at the time, NOT copy that data into a .contained resource. Harder yes, but that's what it's for.
Last updated: Apr 12 2022 at 19:14 UTC
