Stream: terminology
Topic: CodeSystem URI as ValueSet URI
Michael Lawley (Nov 02 2021 at 01:52):
The Terminology Service page (maturity level 4) - http://build.fhir.org/terminology-service.html#validation has the following to say:
Every code system has an implicit value set that is "all the concepts defined in the code system" (CodeSystem.valueSet). For some code systems, these value set URIs are defined in advance (e.g. for LOINC, it is http://loinc.org/vs). However, for some code systems, they are not known. Clients can refer to these implicit value sets by providing the URI for the code system itself.
We used to support this in Ontoserver, but ended up disabling it because some infrastructure tooling complained
Grahame Grieve (Nov 02 2021 at 06:54):
hmm. I don't think anyone's doing this
Michael Lawley (Nov 02 2021 at 21:56):
It's come up for me because 1. I was reading the spec, and 2. UCUM doesn't define such an implicit valueset uri so would appear to be a prime candidate for this usage
Grahame Grieve (Nov 02 2021 at 22:33):
well, should we retire it. or make it work?
Robert McClure (Nov 05 2021 at 21:48):
This thread and a request for a value set with all the ICD-10-CM codes in it prompted a lively internal discussion. I believe that Michael found the only, nicely hidden, reference to what is to my mind commonly mentioned as "the implicit value set" meaning "all the codes in the code system." I've had some concern about the confusion that using the same url to represent both a code system and a value set could cause given that this is intended to be the permanent canonical identifier for the object specified. But let's keep in mind the identifier is not expected to resolve so perhaps it means either/both? I also understand there are implicit tooling implications that seem to be creating havoc.
After some discussion with @Lloyd McKenzie we've agreed as a strawman to propose this:
- It is permissible for a code system canonical url can be cast (my word) as a value set canonical url
- When this is done, the implied value set resource SHALL be interpreted to have the following:
- Valueset.url = codesystem.url
- Valueset.status = codesystem.status for the code system version used for any operations
- valueset.immutable = true
- valueset.compose.inactive = We need to have a separate discussion on what "MeaningIfMissing" should say about this, but I would say it should be "false"
- valueset.compose.include.system = the code system base canonical url
- valueset.compose.include.version SHALL NOT be populated so that it (as an immutable value set) is never defined to be specific to a particular version. We may want to think about how this works for all the SCT editions.
Thoughts? @Grahame Grieve @Michael Lawley @Rob Hausam @Carol Macumber @Ted Klein @Carmela Couderc
Grahame Grieve (Nov 05 2021 at 22:54):
using the same url to represent both a code system and a value set could cause given that this is intended to be the permanent canonical identifier for the object specified
so we should always be clear that we are not (and never are) using the same URL to represent both. And it's not documented that way. We're saying that for a code system, there is either an explicit or implicit value set that means 'all the codes in the code system', and that therefore it's acceptable to provide a code system URL as a proxy for that value set. Irrespective of whether we or you think that's a good idea, we must be clear that it's a proxy, not an alternative identifier. (like people who put medications on the problem list as a proxy for the underlying condition and side effects of treatment)
btw. the 'all codes' value set includes inactive codes - it must. Otherwise agree about it how it's constructed. If you don't want inactive codes, you don't want 'all codes in the system' and using the code system URL as a proxy for the value set you do want isn't the thing you want to do
Ted Klein (Nov 05 2021 at 23:02):
Finally home and looking at this. Note we have a whole raft (>100s) of value sets in THO (imported from v3 coremif last official release December 2019, and from v2) that are defined with a compose.include only specifying the system (thus including all codes in the expansion). The URI values are all over the place relative to the CodeSystem URI. Many (most?) of these have additional metadata, ie the other elements in ValueSet resource with various values, many of which differ from similar or implied values in the CodeSystem referenced in the compose. Some (many? most?) have values that are required for conformance to VSD; I think (but need to check) that some of these are not in the CodeSystem resource specified in the ValueSet.compose; unknown what the effect/meaning of this might be. Some (a few I think?) of these value sets have their URI specified in CodeSystem.valueSet element; also unknown what this means wrt how the tooling handles it, and what might be the expectation when these resources are imported from the generated package for servers. Maybe it is just I am ignorant of too many things, but it appears there are still a number of disconnects and ambiguities.
Lloyd McKenzie (Nov 06 2021 at 04:07):
Right. I never asserted that the Valueset.url would be the CodeSystem.url - the implicit ValueSet doesn't have a URL (and can't be stuck in a registry or otherwise discovered).
Patrick Werner (Nov 06 2021 at 11:01):
Hmm. Isn‘t the implicit VS.url defined in CodeSystem.valueSet?
Patrick Werner (Nov 06 2021 at 11:04):
I‘m concerned about the proposed „proxy pattern“
This would allow binding an element to a CodeSystem. This is an error i see beginners doing a lot.
IG publisher/ validator creates currently errors on this, helping FHIR beginners.
Allowing the proxy url means, that a validator can‘t throw an error in this case - as the user could have meant to use the cs proxy url.
Robert McClure (Nov 06 2021 at 15:04):
All, the above proposal was intended to be provocative to force clarifications, so let's get all this straight.
@Grahame Grieve This:
we must be clear that it's a proxy, not an alternative identifier.
Is a distinction without a difference unless you clearly - for all, not just highly technical designers and implementers - clarify how to easily distinguish when "a proxy" is ok and when it is not.
@Lloyd McKenzie
the implicit ValueSet doesn't have a URL (and can't be stuck in a registry or otherwise discovered)
If the implicit value set does not have a url, then how are we referencing it? That statement in the context of how FHIR is currently designed makes no sense to me. I keep thinking that in essence "the implicit all codes value set" that is defined as the code system url is actually not a FHIR value set. Instead, it seems to be a shorthand reference to an expansion of all the codes for a specified code system (can be version specific?) that can be treated like the result of $expand of an all codes value set and used for validation. If that is on-target, then I'd like to see an explicit list of the places this can be used: Binding? Max value set (I'd very much like this) , other?
Lloyd McKenzie (Nov 06 2021 at 16:43):
CodeSystem.valueSet is a reference to an explicit valueset that serves the same function as the implicit one - though it could potentially have significantly more metadata.
The implicit value set gets generated "on the fly" when someone points to a CodeSystem instead of a ValueSet in a place where only ValueSets are allowed to be referenced. The ValueSet itself doesn't have a URL because it is not storeable/discoverable/retrievable as a distinct entity. If you reference the canonical for a code system any place where the data type is canonical<ValueSet>, then that's taken as an instruction to auto-generate the implicit value set.
And, as I think about it more, if the reference in the canonical is to a specific version of the code system, then the implied value set should also be tied to that version of the code system.
Grahame Grieve (Nov 06 2021 at 21:22):
An implicit value set does implicitly have a URL, but you probably don’t know what it is. (Explicitly!)
Grahame Grieve (Nov 06 2021 at 21:23):
Think we clearly say that using a proxy is only OK in one particular context? Where do we currently say that?
Grahame Grieve (Nov 06 2021 at 21:24):
(I could look it up but I’m out in the country today wth limited access - surprised zulip is working)
Michael Lawley (Nov 06 2021 at 21:49):
"All codes in the code system" means that ValueSet.compose.inactive must be true.
Michael Lawley (Nov 06 2021 at 21:53):
CodeSystem.valueSet is a reference to an _explicit_ valueset that serves the same function as the implicit one - though it could potentially have significantly more metadata.
@Lloyd McKenzie where is it stated that the ValueSet must be explicit?
Michael Lawley (Nov 06 2021 at 21:57):
At issue here for me are code systems that do not have a value defined for CodeSystem.valueSet.
I would be comfortable, to handle the "it's a proxy, not an alternative identifier" constraint, that it is only valid to bind to CodeSystem.url as an alternative IF CodeSystem.valueSet is undefined. I think this would usefully handle @Patrick Werner 's issue and is in alignment with the existing wording I quoted at the beginning of this thread.
Lloyd McKenzie (Nov 06 2021 at 23:09):
There's zero value in relisting the CodeSystem url given that it's a given that url can always be used
Robert McClure (Nov 08 2021 at 17:12):
My understanding of this discussion, particularly Lloyd, is that we allow the use of the code system url to be used in places that expect a valueset canonical url. When this is done, the returned value set needs to be something formally defined, and my provocative proposal above was to clarify what that thing is. I sense we've not settled on something and I'd like to see a full update to my proposal so we all understand. In particular I believe the valueset.url must be populated - so with what?
I'm not sure how to use Michael's desire to do an either/or with codesystem.valueset. given that if we allow this typecasting, it can occur anywhere and there is no guarantee that the server needing this "all codes" thing will have a code system with that populated.
Grahame Grieve (Nov 08 2021 at 19:21):
it doesn't have to be everywhere. In particular, it can not be in resources, because of the way that Reference() is defined. It can only be in parameters on the terminology operations
Michael Lawley (Nov 08 2021 at 21:24):
Does that mean you can't bind to such valuesets nor use them as the scope for ConceptMaps? That seems quite limiting.
Michael Lawley (Nov 08 2021 at 21:28):
@Robert McClure I would not expect to be able to search for an implicit ValueSet and get back a representation of it (partly because one of the use-cases for implicit resources is to identify things that are beyond the scope of FHIR to define explicitly -- http://snomed.info/sct?fhir_vs=refset is an example).
Grahame Grieve (Nov 09 2021 at 07:29):
Does that mean you can't bind to such valuesets nor use them as the scope for ConceptMaps? That seems quite limiting.
I believe so
Patrick Werner (Nov 09 2021 at 11:32):
Grahame Grieve said:
Does that mean you can't bind to such valuesets nor use them as the scope for ConceptMaps? That seems quite limiting.
I believe so
just create an include everything VS of this CS. Or am i missing a point
Michael Lawley (Nov 11 2021 at 00:01):
Yes, it's always possible to create an explicit ValueSet, but then a) it's another thing to manage, share, etc, and b) it forces everyone to go off and create their own version of essentially the same thing. Eg a ConceptMap between SNOMED unit codes and UCUM codes would need to also define (and pass around) an all-codes UCUM ValueSet rather than just being able to reference an implicit one.
Lloyd McKenzie (Nov 13 2021 at 15:16):
I would expect you to be able to bind and use them in concept maps, but NOT be able to store them in registries. ValueSet.url is not mandatory. So when you generate the thing in memory, you don't have to have one.
Patrick Werner (Nov 15 2021 at 16:00):
Michael Lawley said:
Yes, it's always possible to create an explicit ValueSet, but then a) it's another thing to manage, share, etc, and b) it forces everyone to go off and create their own version of essentially the same thing. Eg a ConceptMap between SNOMED unit codes and UCUM codes would need to also define (and pass around) an all-codes UCUM ValueSet rather than just being able to reference an implicit one.
i understand the use-case and like the idea (wasn't aware of implicite VS). Is this supported by any tooling/product beside ontoserver at the moment?
Michael Lawley (Nov 15 2021 at 20:19):
The basic non-ECL ones tend to be supported by terminology servers that have SNOMED support. I believe tx.fhir.org also has good coverage. You can try some of them out and compare with this tool: https://ontoserver.csiro.au/vstool
Patrick Werner (Nov 17 2021 at 11:31):
The IG Publisher and Java Validator are not allowing binding to a CS:
The valueSet reference http://hl7.org/fhir/encounter-status on element Encounter.status points to something that is not a value set (CodeSystem)
Grahame Grieve (Nov 17 2021 at 19:10):
no. the way the definitions work, it's invalid to substitute a reference to a ValueSet with a CodeSystem in a reference or a canonical data type. It's only possible to do so in parameters to tx server calls, and that's the scope of what's being discussed here
Robert McClure (Nov 17 2021 at 19:13):
@Grahame Grieve said:
it doesn't have to be everywhere. In particular, it can not be in resources, because of the way that Reference() is defined. It can only be in parameters on the terminology operations.
This actually would work for me, but I'm not clear that it would work for others and it is clearly not what folks thought they could do with implicit value sets. If we restrict the use of an implicit value set to a parameter in operations that allowed value sets as a parameter, then I think all my "how do you represent it" concerns go away. We'd not want folks to persist it, but perhaps we'd say that an implicit value set cannot have valuest.url populated. then for sure you can not do normal value set things with it.
Would this work?
Grahame Grieve (Nov 17 2021 at 21:05):
hey no
Grahame Grieve (Nov 17 2021 at 21:06):
there's some real confusion here. Implicit Value Sets are really value sets, and you can use them in value set references - everywhere
Grahame Grieve (Nov 17 2021 at 21:06):
and implicit value sets have real well defined URLs
Grahame Grieve (Nov 17 2021 at 21:07):
it's using CodeSystem URLs as proxies for value sets that may or may not be explicitly defined that's only possible in terminology service parameters
Rob Hausam (Nov 17 2021 at 23:47):
I guess it's this last point that I'm not so sure about. If the code system url represents an implicit value set for "all codes" in the code system, then I think, at least for the sake of consistency, it should follow the same rules as any other implicit value set (as noted above). And if isn't a url that references the implicit value of all codes in the code system (either as specified in CodeSystem.valueSet or by convention), then I think we shouldn't assume that it can be used for that purpose - anywhere. I think it probably adds even more possibility for misuse and confusion if we allow the code system url to represent some sort of "implicit value set lite", that can be used in one place, but not others.
Grahame Grieve (Nov 18 2021 at 02:11):
If the code system url represents an implicit value set for "all codes" in the code system
it does not. It cannot 'represent' an implicit value set.
Grahame Grieve (Nov 18 2021 at 02:12):
a value set may be inferred from it - whether explicit or implicit.
Grahame Grieve (Nov 18 2021 at 02:13):
I don't feel strongly that it should be able to used like this with terminology service parameters. it might be a bad idea given that many in the community still can't differentiate between code systems and value sets, or it might be a good idea as a convenience short cut
Grahame Grieve (Nov 18 2021 at 02:13):
but I'm very sure that you can't use a code system URL in place of a value set URL in a reference
Grahame Grieve (Nov 18 2021 at 02:14):
there's a compromise: define a different parameter that can be used in place of a value set url
Rob Hausam (Nov 18 2021 at 02:38):
I'm fine if we keep them completely separate. It's just that, as has already been pointed out, the spec in 4.7.5 Value Set Validation describes using the code system URI as an "implicit value set" - and doesn't say anything about "this only applies to terminology service operations" (though the statement does appear on that page) and it doesn't make any distinctions from any other type or use of an "implicit value set". So we at the very least need to make the scope of what we want this to apply to (if we want to use it at all) much clearer.
Grahame Grieve (Nov 18 2021 at 02:46):
well, the wording is narrower than that:
Grahame Grieve (Nov 18 2021 at 02:46):
Clients can refer to these implicit value sets by providing the URI for the code system itself
Grahame Grieve (Nov 18 2021 at 02:47):
so:
- "clients" - it's not a thing in a resource, only a client (of the terminology service)
- the wording is clear: it's a way to refer to them.
Grahame Grieve (Nov 18 2021 at 02:47):
never the less, it could be clearer, still. Or we could remove it since that wording predates defining the CodeSystem resource itself
Rob Hausam (Nov 18 2021 at 02:54):
I get your point about "client" - but I think most of us (including me) have tended to gloss over that when reading this. So, as you say, we either need to make it clearer, or remove it. Given that there's not much (if any) support or requests for it, the latter may be the easiest and least confusing. And since that page is not (yet) normative, this could be a good time to make that change.
Lloyd McKenzie (Nov 18 2021 at 15:40):
What is an "implicit" value is not pointing to the CodeSystem as a ValueSet?
Bret H (Nov 18 2021 at 15:45):
can someone tell me if there is a difference here between 'implicit' and 'intensional' value set? https://www.hl7.org/fhir/valueset-example-intensional.html Seems really similar conceptually to me. ?
Lloyd McKenzie (Nov 18 2021 at 16:30):
Intensional value set is defined by a filter expression. Implicit value set (as I understand it) is a value set that isn't formally defined and can't be found by searching a registry, but can still be passed as a URL to a terminology server and cause it to behave "as if" it were a formally defined value set.
Bret H (Nov 18 2021 at 17:31):
why would you do that rather than use an intentional value set?
Bret H (Nov 18 2021 at 17:32):
The intentional value set definition can be made fairly vague
Bret H (Nov 18 2021 at 17:33):
e.g. filter by existence in the codesystem
Rob Hausam (Nov 18 2021 at 17:56):
This is the explanation from the Using SNOMED CT with FHIR page (SNOMED CT has the most well-developed set of implicit value sets):
Implicit value sets are those whose specification can be predicted based on the grammar of the underlying code system, and the known structure of the URL that identifies them. SNOMED CT has two common sets of implicit value sets defined: By Subsumption, and By Reference Set. These implicit value sets do not use complex queries. Implicit value sets can also be defined using an expression constraint. The implicit value set capability allows a single URL to serve as a value set definition, and can serve as the basis for the $expand operation and for other value set references.
Implicit value sets are basically a convenience. For this collection of relatively simple and predictable value sets, this capability avoids the need for someone (either manually or programmatically) to create and maintain them as individual FHIR resources and to install them on multiple terminology servers. And for servers that support the capability it gives users a simple and predictable way to access the value set content.
Bret H (Nov 18 2021 at 18:03):
@Rob Hausam umm, maybe I missing something but sounds like the implicit definition above is using a filtering by Subumption and by reference set within SNOMEDCT, which could be done with an intensional value set. can you help me further?
Grahame Grieve (Nov 18 2021 at 18:34):
that's because @Lloyd McKenzie's definition is wrong:
Implicit value set (as I understand it) is a value set that isn't formally defined and can't be found by searching a registry
Grahame Grieve (Nov 18 2021 at 18:36):
not at all. an implicit value set may exist and may be found in a registry. Or not. But it's content can be predicted from it's URL. For this reason, implicit value sets are mostly intensional (or they just include all the code system)
What's useful about implicit value sets is that you can use them and a teminology server can understand without someone having to try and instantiate an infinite amount of intensional value sets on the server
Lloyd McKenzie (Nov 18 2021 at 21:44):
And the only type of implicit value set URL is a code system URL - is that correct?
Rob Hausam (Nov 18 2021 at 21:57):
No. The SNOMED CT page I linked to above describes 5 different SNOMED CT implicit value set url patterns. That's the most complex that we have, but other code systems can do similar things when appropriate.
Grahame Grieve (Nov 18 2021 at 23:12):
And the only type of implicit value set URL is a code system URL - is that correct?
no a code system URL is not an implicit value set. I can see that I'm going to have to keep saying this repeatedly
Lloyd McKenzie (Nov 19 2021 at 04:33):
Then I'm confused. But if this is something that only magically manifests when invoking terminology services and can't be used anywhere else, then I guess I don't really care...
Davera Gabriel (Nov 19 2021 at 18:04):
in the mix here, and despite what the definitions in HL7 may have for the term: "intensional" I do think there is a need for rendering a value set whose members meet the criteria expressed in a set of rules and may differ structurally from value sets rendered from an established (static?) mechanism in a registry or a code system.
Grahame Grieve (Nov 19 2021 at 18:54):
@Lloyd McKenzie well, how do we clarify this for you. You running around making false statements on this might be the source of the confusion here. A code system URL is not an implicit value set. A client may infer a value set identity (CodeSystem.valueSet) from the code system URL - and that may or may not be implicit. But yes, as you say, it's only for terminology service clients; it's not something that's possible or allowed in the Reference or canonical types
Michael Lawley (Nov 19 2021 at 20:43):
I would be in favour of removing the wording from 4.7.5 because a) it is clearly promoting confusion, and b) we removed this from Ontoserver some years ago with no negative feedback.
Peter Jordan (Nov 19 2021 at 21:16):
Agree with @Michael Lawley. As he states in an earlier post, this section pre-dates the creation of the CodeSystem resource. I only find use for implicit value sets with regard to Code Systems that my Server won't return in their entirety (e.g. SNOMED CT and LOINC); otherwise, it's more client-friendly to create an explicit value set for that purpose.
Robert McClure (Nov 20 2021 at 01:29):
Wow, my understanding was the same as @Lloyd McKenzie but you all seem to have made up a very context-specific meaning for implicit when talking about a value set. The SCT set is just to my eye a SCT-specific Expansion query and since you can not use the syntax to define a value set for any other use, I'm kinda at a loss as why this is all that important. I actually think not having to explicitly define an "all code" value set for every code system would be useful, hence my desire to formalize what it meant. Seems I'm in the minority.
I absolutely agree section 4.7.5 needs to be removed as it seems to make sense to very few people and is interpreted incorrectly by the majority.
And I still need a value set of all ICD-10-CM codes that I hope DaVinci wants for a max binding but I'll see. I assume if not inside USCore, it will be made in the specific IG that needs it. Or THO... @Paul Knapp
Grahame Grieve (Nov 20 2021 at 02:56):
I actually think not having to explicitly define an "all code" value set for every code system would be useful, hence my desire to formalize what it meant.
It would certainly be useful to have a some standard syntax, but the code system URI itself can't be. {url}/valueset would generally work...
Lloyd McKenzie (Nov 20 2021 at 15:02):
To make sure I understand, when invoking terminology services and passing an argument as an HTTP parameter, in an argument that would usually be a ValueSet canonical, it's possible to send a CodeSystem canonical instead. The interpretation in that circumstance is to treat the CodeSystem reference as if it was instead the CodeSystem.valueSet canonical and proceed with the execution of the operation. Is that correct? What if the Code System didn't have a CodeSystem.valueSet? Would that be an error, or would the terminology service 'infer' the existence of a value set that said "all codes from this code system"?
Grahame Grieve (Nov 20 2021 at 20:37):
To make sure I understand, when invoking terminology services and passing an argument as an HTTP parameter, in an argument that would usually be a ValueSet canonical, it's possible to send a CodeSystem canonical instead.
yes
The interpretation in that circumstance is to treat the CodeSystem reference as if it was instead the CodeSystem.valueSet canonical and proceed with the execution of the operation. Is that correct?
yes. In fact, that pre-existed the definition of CodeSystem/$validate-code s
What if the Code System didn't have a CodeSystem.valueSet? Would that be an error, or would the terminology service 'infer' the existence of a value set that said "all codes from this code system"?
if it can infer it - whether stated or not - then it should use that. If it can't (can't think why it wouldn't) then that would be an error
Joshua Wiedekopf (Apr 06 2022 at 10:48):
Hi, i discovered this thread after creating an issue in HAPI FHIR that pertains to this thread. I'm a bit confused as to what the agreed-upon result of this discussion between @Lloyd McKenzie and @Michael Lawley is:
Lloyd: CodeSystem.valueSet is a reference to an explicit valueset that serves the same function as the implicit one - though it could potentially have significantly more metadata.
Michael: where is it stated that the ValueSet must be explicit?
The remainder of the discussion revolved mostly about using the CodeSystem.url as proxy for ValueSet.url in the scope of terminology service operations.
My question is thus: do we have agreement that when a CodeSystem.valueSet parameter is set to a value not referring to an explicit VS known to the server, the server shall allow e.g. expansion of that ValueSet when calling ValueSet/$expand?url=<the-url>? This is the way Ontoserver handles it, and personally, I agree with this behaviour, since maintaining explicit VS just so I can bind all codes to e.g. a ConceptMap.sourceUri is not desirable IMHO.
Grahame Grieve (Apr 06 2022 at 11:49):
it seems reasonable to me
Last updated: Apr 12 2022 at 19:14 UTC
