Stream: implementers
Topic: Querying with :excludes vs :not ?
Josh Mandel (Jul 13 2020 at 16:16):
Today in FHIR it's hard to query for data that doesn't match a specific value. For example, say you want to query for data that *aren't marked with a "restricted" or "very restricted" security label. You might try:
GET Observation?_security:not=R,VR
... but this query is subtly broken (and @Keith Boone explains many of the reasons in his "Set theory much" blog post). Briefly, this query will match a resource that has a VR security label if the resource also has some some other label like a "sensitivity category" code of http://terminology.hl7.org/CodeSystem/v3-ActCode|HIV indicating that it's very restricted data about HIV. The HIV code means there's a security label present that isn't R or VR, so this Observation gets included in the response to the :not search, even though we didn't want it to.
Josh Mandel (Jul 13 2020 at 16:17):
Have we thought about defining a different modifier to convey this intent -- something like :exclude instead of :not, that would allow queries like:
GET Observation?_security:exclude=R,VR
John Moehrke (Jul 13 2020 at 18:17):
to be clear this is not a problem specific to _security..... and actually ConfidentialityCode vocabulary is a bad example as it is a non-overlapping set of 6 codes, so to say 'not this' can also be said as 'only that'. This vocabulary is unusual in that it is explicitly designed as a non-overlapping set of 6 codes. (Note part of this design was also assuming all data had to have a confidentiityCode value (1..1), we did not put that requirement into FHIR core to avoid massive duplication of 99.99% of all resources being forced to have the code Normal. Had we required (1..1) then 'only that' would have been equivalent to 'not this'. Sadly current core requirements on .meta.security do lead to situations you are pointing out.
Josh Mandel (Jul 13 2020 at 18:59):
Good point. I've updated the topic to make this clear.
Grahame Grieve (Jul 13 2020 at 19:14):
you could do this with a value set query, I believe
Josh Mandel (Jul 13 2020 at 20:52):
Can you show me what that would look like @Grahame Grieve ? How do I define a valueset so that Observation?_security:in=https://my-value-set returns only observations that lack a R or VR label? The ValueSet query will only find observations that match something, right?
Grahame Grieve (Jul 13 2020 at 21:45):
yes like that, but using :not-in instead of :in
Josh Mandel (Jul 13 2020 at 21:59):
So if https://my-value-set has 3 codes (A, B, C), then ?_security:not-in=https://my-value-set acts differently from ?_security:not=A,B,C?
Josh Mandel (Jul 16 2020 at 18:43):
@Grahame Grieve can you weigh in on this?
Grahame Grieve (Jul 16 2020 at 23:55):
no I think you're right - it will return any resources that have at least security tag that it not in the value set
Josh Mandel (Jul 17 2020 at 17:48):
So is it worth proposing a new :excludes modifier for these kinds of cases? Or better to just leave this as a known challenge? I could see it either way...
Grahame Grieve (Jul 17 2020 at 22:11):
not sure. we could. Let's see - can we do it with _filter?
Observation?_filter=not (security in https://my-value-set)
I think that's what you want - because with richer syntax, you can control the handling of the not
Josh Mandel (Jul 18 2020 at 03:24):
That's true; is that an argument against proposing a modifier? Many of our most recent improvements to search are things that _filter "already covers"
Grahame Grieve (Jul 18 2020 at 03:38):
it is an argument, yes. I'm not sure how strong it is, as you say
Josh Mandel (Jun 30 2021 at 19:09):
I'm revisiting this with a new use case of excluding data; from search results we're looking at examples in the Argo Write project where patient-supplied data may have a specific tag; it's important to have some way to include or exclude such data from a search -- and today, there's no way to ask for data without a certain tag.
GET Observation?_tag:excludes=http://terminology.hl7.org/CodeSystem/common-tags|unreconciled
would be a fairly natural thing to do. I'll submit a Jira issue.
Josh Mandel (Jun 30 2021 at 19:23):
Wait, reading now I see the following note tucked away in an example at http://hl7.org/fhir/search.html#token:
Note that this search does not return "any document that has a section that is not an Allergies and adverse reaction section" (e.g. in the presence of multiple possible matches, the negation applies to the set, not each individual entry)
... this seems to imply that :not already behaves like the :excludes I'm asking for -- but nobody point this out, and @Grahame Grieve you even said the opposite:
it will return any resources that have at least security tag that it not in the value set
Grahame Grieve (Jun 30 2021 at 19:35):
yes i was wrong, sorry
Josh Mandel (Jun 30 2021 at 19:50):
No, that's a great outcome. My use case is already supported. I'll give this a shot against some reference servers.
Last updated: Apr 12 2022 at 19:14 UTC
