Stream: implementers
Topic: Period Datatype Invariant
Kathy Pickering (Oct 19 2020 at 17:17):
Hi @Grahame Grieve @Jean Duteau
We are requesting that the http://hl7.org/fhir/datatypes-definitions.html#Period invariant text is changed to match the expression like "If present, start SHALL have a lower or equal value than end". We have real world same day encounters with a date only where period.start = period.end.
We have logged FHIR-29268. We are hoping to hear that you agree and will approve our change request.
Grahame Grieve (Oct 20 2020 at 04:03):
yes that sounds like a very good idea
Cooper Thompson (Dec 03 2021 at 19:31):
Resurrecting this, but I just submitted FHIR#34417. We have Encounters that have periods like this:
{
"start": "2023-06-21",
"end": "2023-06-21T06:20:00Z"
}
This is invalid per the per-1 invariant, given the FHIRPath rules for comparing dates with differing precision. When FHIR#29268 was approved, was that intended to allow periods like the example I gave?
Cooper Thompson (Dec 03 2021 at 19:32):
Related chat.fhir topic on FHIRPath datatime comparison.
Lloyd McKenzie (Dec 03 2021 at 19:42):
@Grahame Grieve
Brian Postlethwaite (Dec 06 2021 at 04:09):
If a comparison has different levels of precision, and the result is an empty set, then as an invariant that is considered a pass, not a false.
Cooper Thompson (Dec 06 2021 at 16:28):
Hmm... I wonder if this is an Inferno or HAPI validator issue then. The Inferno validator is failing this resource:
{
"resourceType": "Encounter",
"id": "e1ROhluOalPIMvQ6iOXOINQ40Te8Rfz.3DCjEkUpKtpc3",
"identifier": [
{
"use": "usual",
"system": "urn:oid:1.2.840.114350.1.13.5325.1.7.3.698084.8",
"value": "10001466261"
}
],
"status": "unknown",
"class": {
"system": "urn:oid:1.2.840.114350.1.72.1.7.7.10.696784.13260",
"code": "1",
"display": "Surgery Case"
},
"subject": {
"reference": "Patient/eNO3wqOfAltfnWMfWBQ1WmQ3",
"display": "Test, Emily FHIR"
},
"period": {
"start": "2023-06-21",
"end": "2023-06-21T06:20:00Z"
},
"location": [
{
"location": {
"reference": "Location/eZq-E51GdFzpYxmfoRNU8pRvm2xL.sTjR8Beh3hPnIBE3",
"display": "OR LOCATION"
}
}
]
}
Cooper Thompson (Dec 06 2021 at 16:28):
@Yunwei Wang - does Inferno use the HAPI validator for BDGV-18 and https://inferno.healthit.gov/validator/?
Daniel Venton (Dec 06 2021 at 16:46):
Comparing dates where one one of them has a time (w/offset) to one that doesn't is always going to be problematic. What time do you associate with "start" in this case? Midnight, 1 min before midnight tomorrow, noon? What offset? Zulu, local, same offset as the other value (end in this case)? How do you know the exact minute the encounter ended but don't know the hour/minute of when it started? Maybe you should ignore the time on end if you don't have time on start? There are no good answerers here.
Yunwei Wang (Dec 06 2021 at 18:23):
@Cooper Thompson Yes. Inferno validator (https://inferno.healthit.gov/validator/) uses Inferno Validator Service v2.0.0 which depends on HL7 FHIR validator v5.3.14.
Inferno Program uses Inferno Validator Service v1.2.0 which depends on HL7 FHIR validator v5.2.10
Yunwei Wang (Dec 06 2021 at 18:42):
Based on the FHIR date search (http://build.fhir.org/search.html#date), 2023-06-21 is a range [2023-06-21T00:00:00, 2023-06-22T00:00:00). This range is not less than (or below) the range [2023-06-21T06:20:00, 2023-06-21T06:20:01). So I think the validation result is correct.
Cooper Thompson (Dec 06 2021 at 19:09):
I think FHIRPath has different date comparison semantics than the range comparison from the date searching. My understanding is that FHIRPath comparison will return empty set if dates have different precisions.
Cooper Thompson (Dec 06 2021 at 19:20):
So for per-1:
start.hasValue().not() or end.hasValue().not() or (start <= end)
This should evaluate to: false or false or empty set. The FHIRPath Tester does evaluate (start <=end) to True, so Brian's claim that it should pass checks out, at least in terms of the FHIRPath Tester implementation (I'm still a noob on the FHIRPath spec, and haven't found the rules define there for handling empty set conversion to Boolean values for expressions).
Yunwei Wang (Dec 06 2021 at 19:32):
I got the same error from validator_cli.jar v5.6.4
Cooper Thompson (Dec 06 2021 at 19:37):
So is the validator not using the FHIRPath semantics for invariant validation? I.e. this is a bug in the validator instead of the spec?
Yunwei Wang (Dec 06 2021 at 19:52):
I tried using -version 4.0.1. I cannot run validator_cli against R5, I got "unable to resolve package id null#4.6.0" or "Unknown FHIRVersion code 4.6.0" depending on what parameter I provided.
Vassil Peytchev (Dec 06 2021 at 22:50):
I don't know if it would make any difference, but the latest validator is 5.6.7
Grahame Grieve (Dec 07 2021 at 00:20):
I think that the FHIRPath tester is wrong here. The FHIRPath specification says:
For partial Date, DateTime, and Time values, the comparison is performed by comparing the values at each precision, beginning with years, and proceeding to the finest precision specified in either input, and respecting timezone offsets. If one value is specified to a different level of precision than the other, the result is empty ({ }) to indicate that the result of the comparison is unknown.
The validator rule is that this SHALL be true, and it's not known to be true, so all versions of the validator will fail this encounter
Grahame Grieve (Dec 07 2021 at 00:21):
and to me, this makes sense. A start period of sometime on the day is not known to be less than a particular time during that day. If you mean, 00:00:00, then say so.
Brian Postlethwaite (Dec 07 2021 at 02:04):
then an invariant on patient for birthDate < today() would fail in the event that the patient doesn't have a date.
Brian Postlethwaite (Dec 07 2021 at 02:06):
@Ewout Kramer can you look into this one too - it's pointing at a difference between the dotnet validator too (or at least my understanding of it)
Grahame Grieve (Dec 07 2021 at 02:11):
yes it would fail. If you wanted it not to fail then you'd say birthDate.exists().not() or (birthDate < today())
Cooper Thompson (Dec 08 2021 at 15:57):
Grahame Grieve said:
yes it would fail. If you wanted it not to fail then you'd say
birthDate.exists().not() or (birthDate < today())
Wouldn't you have to handle all the different possible date precisions separately for this evaluate intuitively? I.e. you'd need to independently handle a birthDate that is null vs. YYYY vs. YYYYMM vs. YYYYMMDD? YYYY < today() would resolve to empty set (false), same with YYYYMM < today() (false).
Cooper Thompson (Dec 08 2021 at 16:00):
For example, if today() is 20211208, and you have a birthDate with a precision of YYYY like 2020, then birthDate < today() would be empty set which would eval to false.
Grahame Grieve (Dec 08 2021 at 18:51):
@Bryn Rhodes this is a very good point, and I bet no one writing CQL is catering for this. We need a function/operator for difference between dates (can't do date - date -> quantity right now) and also some way to do greater than / less than for dates that's precision robust - right now the FHIRPath specification is clear that comparison of dates with different precision is {}
@Cooper Thompson what would you do for birthdate of YYYY-MM where they match the today's y and m?
Cooper Thompson (Dec 08 2021 at 20:07):
If you are asking what I think would be intuitive, I'd think YYYYMM < today() would be false, but YYYYMM =< today() would be true. So as long as the precision you have matches, then the equality check would pass. Though that implies that YYYYMM = today(), which sometimes feels right, but I'm not sure it always does. Maybe we need another comparison operator, where = will be true as long as the precision you have matches, and == returns true only if you have matching precision?
Cooper Thompson (Dec 08 2021 at 20:08):
I think making a new operator be the full precision check and changing the semantics of the existing = operator would mean that the existing invariants that I know of would work "as expected". But that has all sorts of implications I'm sure.
Grahame Grieve (Dec 08 2021 at 20:09):
well, we do have the ~ operator
Cooper Thompson (Dec 08 2021 at 21:28):
Ah yeah. I should probably read the FHIRPath spec top to bottom instead of skimming and Ctrl+F'ing.
~ seems like the opposite of what we want though right? We want an operator that returns true when the dates have different precision, but the precision that does exist is equal.
Chris Moesel (Dec 08 2021 at 21:55):
Note that there is a difference between the empty set {} and false. In the context of date comparison, empty set ({}) basically means maybe (i.e., not enough information to return a concrete answer), vs. false (which means definitively false).
So, YYYYMM < today() is true if YYYYMM is before today's year and month, false if YYYYMM is after today's year and month, and empty {} if YYYYMM matches today's year and month. This gives you flexibility because you can decide if you want empty {} treated as true or false for your use case. E.g.:
- If you want YYYYMM < today() to be true when the year and month match today's year and month, then write it as
(birthDate < today()).allTrue()(sinceallTrue()returns true for empty sets). - If you want YYYMM < today() to be false when the year and month match today's year and month, then write it as
birthDate < today()(since an empty set will be treated as false in that case)(birthDate < today()).anyTrue()sinceanyTruereturns false for empty sets).
Note that while I used YYYYMM as an example, the logic works the same way if the birthdate way only specified to years (YYYY). I don't think there is a way in FHIRPath to easily switch logic based on the exact precision of the input.
Ewout Kramer (Dec 09 2021 at 11:56):
@Ewout Kramer can you look into this one too - it's pointing at a difference between the dotnet validator too (or at least my understanding of it)
Yes, there is a difference. The validator is running invariants as predicates, which means that the empty result evaluates to true, not false. It's been like that for years, and probably never led to problems because of the fully quantified invariants using ... exists().not() or ......
Ewout Kramer (Dec 09 2021 at 12:00):
Sounds like a big change....
Chris Moesel (Dec 09 2021 at 14:45):
Actually, that's a good point, @Ewout Kramer. Somewhere I got the impression that {} defaults to false when a true boolean is expected, but I actually don't see anywhere in the spec that indicates this. Since {} generally means "I don't know" in the context of boolean logic, it may make sense that you don't treat that as a violation of an invariant.
Chris Moesel (Dec 09 2021 at 14:48):
I've updated my suggestion above to explicitly return true/false rather than potentially return empty set.
Cooper Thompson (Dec 09 2021 at 15:02):
It seems odd folks haven't run into this before. While we've been using birthDate as the example, my actual issue is with the per-1 invariant, which is on the Period data type. And the Period data type is used all over.
Cooper Thompson (Dec 09 2021 at 15:03):
FHIR#34417 is the tracker I'm interested in. And if we take @Chris Moesel's suggestion, then we'd be updating the invariant on that normative data type. :tada:
Grahame Grieve (Dec 09 2021 at 18:56):
I think the spec is quite clear on this: A constraint is defined as:
Condition that must evaluate to true
So that's quite clear: if the result of the expression is unknown, it's cannot evaluate to true.
Grahame Grieve (Dec 09 2021 at 18:57):
a validator that treats {} as true is a validator that is not enforcing the constraints as stated
Bryn Rhodes (Dec 13 2021 at 19:31):
@Grahame Grieve @Cooper Thompson @Chris Moesel , the comparison operators are precision robust in that they proceed from years down. It's not that comparisons with different precisions always return null, they return null if that difference means the answer is unknown. In this case, for example:
@2021-12-08 < @2020
This evaluates to false because 2021 is greater than 2020, so we're safe to return false, not null. On the other hand,
@2020-12-08 < @2020
This evaluates to null because we don't know the actual month or day on the right-hand side.
Bryn Rhodes (Dec 13 2021 at 19:35):
For calculating difference and duration, we have those operators defined in CQL, it would be straightforward to promote those definitions to FHIRPath as part of the next version.
Grahame Grieve (Dec 13 2021 at 19:41):
we should do that, yes
Brian Postlethwaite (Dec 20 2021 at 09:28):
This will mean a whole tonne of my fhirpath expressions need heaps more null checks in them. :sob:
Cooper Thompson (Dec 20 2021 at 14:13):
Probably not just yours. I think we probably need to review most of the date-based invariants in the whole spec, since the intuitive thing isn't what happens.
Richard Townley-O'Neill (Dec 21 2021 at 02:00):
@Brian Postlethwaite While reading this thread I notes some related problems with the FHIRPath tester. :sad: https://github.com/brianpos/FhirPathTester/issues/10
Bryn Rhodes (Jan 03 2022 at 20:18):
This will mean a whole tonne of my fhirpath expressions need heaps more null checks in them.
@Brian Postlethwaite , it really shouldn't, the intent is to provide the intuitive behavior. Can you provide some specific examples where you would need it? We have been using this approach in CQL for years with temporal comparisons and we haven't had to use null checks.
Bryn Rhodes (Jan 03 2022 at 20:24):
And there is already a FHIRPath issue related to this, so I've added a proposed disposition to it: https://jira.hl7.org/browse/FHIR-19896
Cooper Thompson (Jan 04 2022 at 20:02):
Sort of related to this, but what should a system do if they have data that is not compliant with the (new/future updated) expression? For example, if we have a start date of 1/2/2022 and an end date of 1/1/2022 (i.e. startDate > endDate). This of course doesn't make sense, but if we got that data from a legacy system that has been retired, there isn't really much we can do.
The options I can think of:
- Don't send the data at all. But this isn't really great, since it means that callers of the API will just be unable to access data that we have.
- Send it in Period anyway, ignoring the constraint, and ignore Lloyd when he yells at us for being non-conformant :D.
- Check every set of start/end timestamps, and if they pass the invariant, send them in Period, and if not, send them in extensions.
Cooper Thompson (Jan 04 2022 at 20:03):
I'm really starting to struggle with FHIR invariants, must-support, and 1..x cardinalities. It seems like they really are a representation of what the community hopes is true in the real world. But if real-world data doesn't align with those hopes, then systems are really SOL if they want to use FHIR. We can always add data validation (if we don't have it already) for new data entered in the system. But legacy data is just something we have to deal with.
Lloyd McKenzie (Jan 04 2022 at 21:46):
Sending garbage data doesn't really help the recipient. One of the two values must, by definition, be wrong. If you don't know which, then you can't really trust either. So sending it as text where at least a human can look and hopefully infer reality is your best/safest thing to do.
Daniel Venton (Jan 04 2022 at 22:03):
I say that conformance is a goal. And it's a fine achievable goal with data collection and recording going forward. But whenever you say, "historical data", conformance becomes a whole lot more challenging. In the case of the official chart says, start day 2 end day 1... clearly wrong. But that is what is in the chart. You have the choice of including the data doing your option 2. (This is often my choice) You can say I know the data is wrong so I won't transfer the data to the FHIR resource copy of the chart (option 1). As you say, now the consumer doesn't have the data at all. Sending them in extensions is fine, but does your consumers know what the extensions are and what do they do with the data that still says, "start day 2, end day 1?" Sure it didn't violate a conformance rule, but it's still wonky. You can put the values in just the .text it's not machine readable and if it's a research project with millions of rows it's unlikely that anybody is going to read them and update with their best guess. This is the problem with historical data, 4 years ago a code/date/etc. was assigned. Today the IG says that code is not allowed. Who goes back and updates 4 year old chart entries so they conform to current rules? Especially if the Dr that wrote the item no longer works there or that was the last time the patient was seen at the facility.
100% conformance with new going forward data is achievable, historical not so much. You can certainly patch it by ignoring non-conformant rows/attributes but are you doing more harm by making the resources conformant?
Lloyd McKenzie (Jan 04 2022 at 23:27):
This isn't about "conformance" so much as "don't send garbage". If something is clearly wrong, yes you can send it, but that puts the work of dealing with it on all recipient systems. If everyone downstream needs to make allowances for garbage data, that gets really expensive. The system that acquired the garbage data in the first place is in the best position to clean it - and so the responsibility should rest on them to do so.
Lloyd McKenzie (Jan 04 2022 at 23:28):
Certainly you can choose to transmit non-conformant instances. However, you'll quite often find that other systems will reject or choke on the data. (And if there are certification requirements, you'll find that you won't pass.)
Daniel Venton (Jan 05 2022 at 13:25):
I agree, the people that own the data are in the best position to clean the data. But who has time/money to "clean" potentially millions of historical records. And if those bad data points require a Dr to asses the patient in order to determine the new correct code, do you call the patient in for a free consult in order to do records maintenance? Is it better to send the Condition of Diabetes with an old obsolete code or not tell the consumer about the diabetes at all (or in a non-machine processable form)?
When it comes to certification, in my experience, you are dealing with current data collection, it's never about "let's pull up data from 4 years ago and see if those are compliant with today's rules."
Cooper Thompson (Jan 05 2022 at 14:14):
The problem with the "don't send garbage" approach is that a lot of useful data is garbage. In fact, the motto of my team for a while was "Garbage in, Miracle out", since we spent so much time handling garbage we got from other systems and trying to make it useful.
In reality, a LOT of data IS garbage, or has parts of it that are garbage. But that doesn't mean it isn't valuable. The question is: should the server be paternalistic and "decide" that the non-conformant data we have isn't useful for any consumer? Or should we expose the data we have, and let the recipient decide if the non-conformant data is useful to them or not?
And of course in the US, the EHRs can't decide that garbage data shouldn't be shared. Info Blocking regulations don't have a direct exception for garbage data (infeasibility might apply, but that feels really extreme and not desirable). So are we supposed to share conformant data using FHIR, and non-conformant data using some other method? That'd be annoying.
Lloyd McKenzie (Jan 05 2022 at 19:17):
Unclean/non-conformant data breaks systems. The rules that FHIR sets are rules that systems are expected to be able to count on when processing FHIR instances. If the data violates those rules, bad things can (and often will) happen. Certainly systems are free to expose whatever data they wish in whatever shape it happens to be in. However, they can't call that data FHIR conformant and they can't count on it being safely processed by downstream systems.
Brian Postlethwaite (Jan 05 2022 at 20:49):
And draft status commits should be able to ig ore that type of invariant too.
Richard Townley-O'Neill (Jan 07 2022 at 00:38):
(deleted)
Lloyd McKenzie (Jan 07 2022 at 18:01):
I'm not clear on that @Brian Postlethwaite. It might make sense for mandatory information to be missing for a draft. But clearly nonsensical data doesn't make any more sense for a draft than for something that's final.
Brian Postlethwaite (Jan 07 2022 at 19:46):
Our server only draws the line at data type level value, and I think that's it. Invariants are definitely off.
As expressions could be checking conditional mandatory fields (very often) or cross field in which case either side could be wrong, and intent is to write an in progress draft.
Grahame Grieve (Jan 10 2022 at 00:30):
@Cooper Thompson I really like "Garbage in, Miracle out"
but I don't have any good solution to this problem at all.
Vassil Peytchev (Jan 10 2022 at 02:14):
Have there been any discussions about having different invariants for POST/PUT vs GET? Or legacy data exceptions?
Lloyd McKenzie (Jan 10 2022 at 02:34):
We could consider introducing a standard Resource.implicitRules element that says "legacy data, quality rules may not hold" and possibly have the validator turn errors into warnings when it's set.
Cooper Thompson (Jan 10 2022 at 14:33):
I do like the idea of letting the server explicitly declare that it knows the resource isn't fully conformant, and possibly with a reason why. The only downside is that the server needs to do validation so that it knows to declare that, and running validation can have a performance impact. I wonder if we could have "strict" and "lax" modes, and have a way for the client to specify which it wants. In lax mode, the server can just send what it has (possibly fudging some conformance constraints), but in strict mode it would need to explicitly declare if something it is sending is non-conformant.
Lloyd McKenzie (Jan 10 2022 at 15:45):
We can't relax the expectation to be fully conformant. If you spit out v2 vertical bar data or a CSV file in a FHIR interface, that's not valid - end stop. What we could potentially do is define a set of constraints that are 'tolerable' if violated by legacy data and - if specifically asked for - invalid data could be sent. The default would have to be that the data must be strictly valid because that's the requirement now.
Cooper Thompson (Jan 10 2022 at 15:55):
I think that could really just be part of the definition of "lax" mode. If "lax" mode only allows you to skip stuff like invariants, cardinalities, must support, value set bindings, etc., but the rest of the FHIR structure still needs to be valid, I think that'd be about right.
René Spronk (Jan 10 2022 at 15:59):
If were to introduce "garbage mode" I'm pretty sure it'd be used far and wide, and mostly without a good reason, which is a reason not to go there. If relevant in your specific context, you could use a meta flag to indicate such a mode. No need for any new functionality, I'd say.
Lloyd McKenzie (Jan 10 2022 at 16:02):
You'd still need to include a modifier extension to highlight the fact that the instance contained garbage.
Cooper Thompson (Jan 10 2022 at 16:23):
Just for the purposes of this conversation, let's define "garbage" as "useful, possibly clinically relevant data that doesn't conform to the optimistic hopes of the FHIR spec (or relevant IG)".
Cooper Thompson (Jan 10 2022 at 16:27):
Which means "garbage mode" == "best option in the industry to send useful clinical data".
Lloyd McKenzie (Jan 10 2022 at 16:35):
A start that's after an end is "garbage". At least one of the two values is definitively wrong and there's no way to tell which. It's not about optimism and, with properly defined input constraints, it's a situation that should not ever happen. It's like a date of Feb. 30 or an age of -10 years. Some things just don't make sense and there's no way for an electronic system to reliably do anything with the data.
Lloyd McKenzie (Jan 10 2022 at 16:35):
I.e. it's not "useful clinical data".
Daniel Venton (Jan 10 2022 at 16:40):
Once you start populating modifierExtension, I think you are guaranteeing that the data will be ignored unless this customization is understood by all consumers.
Cooper Thompson (Jan 10 2022 at 16:41):
Lloyd McKenzie said:
I.e. it's not "useful clinical data".
But if the period is invalid, and that is required in the resource, then we can't send the entire resource. The rest of the resource might be useful.
Lloyd McKenzie (Jan 10 2022 at 16:41):
Right. Well, in this case, the data isn't safe to be consumed unless the consuming system is prepared to handle non-compliant data, so rejecting/ignoring the data if the modifier isn't understood/accepted is probably the correct thing for systems to do.
Cooper Thompson (Jan 10 2022 at 16:41):
And in R4, since per-1 is kinda busted, the data actually is useful. The flaw is in the FHIR spec.
Michele Mottini (Jan 10 2022 at 16:43):
Having working FHIR servers to connect to that spit out syntactically correct FHIR is very very useful, a big improvement on having to implement a load of custom interfaces. Having the FHIR you get from those servers conformant to some implementation guide is a nice to have
Lloyd McKenzie (Jan 10 2022 at 16:43):
If the period is mandatory, it could potentially be handled with a data-absent-reason. And if there are situations where it's reasonable for the period to not be known, then the cardinality should certainly be relaxed in the core spec. It would only be appropriate for period to be mandatory if the belief was that the resource couldn't possibly be useful if the period was unknown.
Michele Mottini (Jan 10 2022 at 16:44):
(where syntactically correct = can be loaded with standard libraries)
Cooper Thompson (Jan 10 2022 at 16:46):
I'm also considering IGs here. A large part of my concern is is due to all the 1..1 and MS tags I saw in IGs this ballot cycle. Either we need to explain to IG authors that anytime they make something 1..1 they are blocking information sharing, or we need an industry-wide solution to handle real-world data.
Lloyd McKenzie (Jan 10 2022 at 16:49):
Complying with an IG is always optional. It's up to an IG whether it wants to allow legacy data or non-'clean' data. Some IGs only care about net-new data or want to protect systems from atypical content. Others need to be able to handle everything. It would be useful though for each IG to have a section that specifically discusses its strategy around legacy, external or other data of potentially poor quality.
Cooper Thompson (Jan 10 2022 at 16:49):
René Spronk said:
If were to introduce "garbage mode" I'm pretty sure it'd be used far and wide, and mostly without a good reason, which is a reason not to go there. If relevant in your specific context, you could use a meta flag to indicate such a mode. No need for any new functionality, I'd say.
The .NET library has a SkipConstraintValidation setting. So there already is a lax mode. It just is up to the client to set that in their handling. In the real world, I think clients MUST set that setting if they want to get the data they care about. So I'm just saying we need this to be a FHIR thing not just something all the FHIR libraries do.
Daniel Venton (Jan 10 2022 at 17:17):
Cooper Thompson said:
I'm also considering IGs here. A large part of my concern is is due to all the 1..1 and MS tags I saw in IGs this ballot cycle. Either we need to explain to IG authors that anytime they make something 1..1 they are blocking information sharing, or we need an industry-wide solution to handle real-world data.
If an IG makes a field 1..1 where base is 0..1, I wouldn't think that the IG is saying all X resources must have that data. Instead only resources that do have that value are for the use case that the IG describes. Making profile a filter. GET base/X?_profile=IGX.... Server would say, all X where field not null.
Lloyd McKenzie (Jan 10 2022 at 17:20):
Right. Not all instances that a server has will necessarily meet the requirements of a given IG - and that shouldn't be a problem.
Yunwei Wang (Jan 10 2022 at 18:16):
That requires server to support _profile search parameter. I don't know which sever actually supports that.
Alexander Zautke (Jan 10 2022 at 18:23):
And even the ones that do, will probably only look for claims in meta.profile
Cooper Thompson (Jan 10 2022 at 18:35):
I wonder if IG authors really understand this. Is it intuitive that doing a Patient?_profile=IPS will automatically filter out all patients that don't have a birthDate or gender or an associated DAR (IPS Patient Profile)? I've seen adding 1..1 be a common pattern in other IGs too, where I don't think the IG authors are using that to intentionally exclude data from their profile use cases. I don't think we're currently set up for success to kick this to IG authors. Either we need a lot of education with IG authors on what adding requirements actually means (it means they are filtering data), or we need a more formal "out" (lax mode). Or we just do nothing and force everyone to be non-compliant in order achieve real-world use cases.
Jean Duteau (Jan 10 2022 at 18:38):
This is a very interesting discussion that I've been following. Almost all of my IGs are taking existing specifications and replicating them in FHIR, so when I'm making something mandatory, it is because the underlying business process (i.e. the existing spec) has it as mandatory. I do understand how this can make interoperability harder, but when the server requires that piece of data -birth date or gender, as an example - my IG and profile saying that those elements are 1..1 is making it clear to the client that they need this or they can't communicate.
I wonder if most IGs and IG authors are like this - "this is mandatory because we need it for the business process we are describing". For IGs that are not about a specific process, the guidance around "don't make things mandatory unless you have to!" is obviously very important.
Stephen MacVicar (Jan 10 2022 at 18:54):
Yunwei Wang said:
That requires server to support _profile search parameter. I don't know which sever actually supports that.
Or which IGs actually require that the server support it.
Yunwei Wang (Jan 10 2022 at 19:13):
@Cooper Thompson you complained about "garbage in". But when you propose a "lax mode", isn't that a "garbage in" for down stream systems. "lax mode" is not a solution but just kicking can down the road.
Paul Church (Jan 10 2022 at 19:16):
The tricky thing about relying on _profile search is that it's only going to work if the server has actually tagged every resource with all of the profiles you care about - it's entirely possible that a resource meets the requirements for profile X, but is not tagged with that in meta.profile because the server didn't validate profile X.
Cooper Thompson (Jan 10 2022 at 19:19):
Yunwei Wang said:
Cooper Thompson you complained about "garbage in". But when you propose a "lax mode", isn't that a "garbage in" for down stream systems. "lax mode" is not a solution but just kicking can down the road.
Kinda. But once you have garbage, what else would you do? You either need to propagate the garbage, or you don't. And if you don't propagate it, are you Info Blocking?
Cooper Thompson (Jan 10 2022 at 19:20):
(or you invoke Content and Manner exception and use something other than FHIR since FHIR can't handle it)
Daniel Venton (Jan 10 2022 at 19:33):
Paul Church said:
The tricky thing about relying on _profile search is that it's only going to work if the server has actually tagged every resource with all of the profiles you care about - it's entirely possible that a resource meets the requirements for profile X, but is not tagged with that in meta.profile because the server didn't validate profile X.
You don't have to have tagged every resource. You could tag every resource if that is how your search works. Say I do a query /Observation?_profile=XYZ.
If the server is SQL and the query is "SELECT * FROM Observations where profile contains (XYZ)" - then yes you need them tagged
But if the server converted that _profile parameter "SELECT * FROM Observations where code = 123 and status = complete and value > 10" - then they don't need to be tagged.
But to @Yunwei Wang point, who has time to implement every different _profile search value and the matching data values into their servers?
Paul Church (Jan 10 2022 at 20:37):
I have mixed feelings about the whole "propagating garbage" issue because it gets to the heart of whether we're propagating data with the intent that it's actually machine-readable to a standard where you can do ML, decision support, visualizations, applications on top of it, or whether it's good enough that a human can maybe guess what it's supposed to mean if they're lucky. I don't want to deny people their records but I would rather they were meaningfully machine-readable or having those records in your possession is kind of a dead end.
Ironically, the Google FHIR implementation has options to disable required fields, disable referent type validation (i.e. a Reference(Patient) field referring to something that isn't a Patient), and disable FHIRPath constraints from the base spec - because for better or worse, some customers want to do these things.
Richard Townley-O'Neill (Jan 11 2022 at 01:21):
Cooper Thompson said:
Sort of related to this, but what should a system do if they have data that is not compliant with the (new/future updated) expression? For example, if we have a start date of 1/2/2022 and an end date of 1/1/2022 (i.e. startDate > endDate). This of course doesn't make sense, but if we got that data from a legacy system that has been retired, there isn't really much we can do.
The options I can think of:
- Don't send the data at all. But this isn't really great, since it means that callers of the API will just be unable to access data that we have.
- Send it in Period anyway, ignoring the constraint, and ignore Lloyd when he yells at us for being non-conformant :D.
- Check every set of start/end timestamps, and if they pass the invariant, send them in Period, and if not, send them in extensions.
Another option is to use an extension in Period to carry the garbage, say
periodAsString: it can carry the start and end in a format useful for humans and there will be no validation problems. Of course it will not be searchable, but it is garbage. And the rest of the resource will be usable.
Lloyd McKenzie (Jan 11 2022 at 04:33):
I definitely prefer options that stick non-conformant data in narrative, extensions, or something else that conveys the "potentially relevant" data that is not computably useful.
Cooper Thompson (Jan 24 2022 at 15:01):
I'd forgotten this was discussed previously in this argonaut historical data thread.
Last updated: Apr 12 2022 at 19:14 UTC
