Stream: IG creation
Topic: Alternate terminology servers
Elliot Silver (Aug 19 2021 at 05:16):
Based last month's terminology server scramble, it looks like it is possible to use a terminology server other than tx.fhir.org for the IG publisher. That raises some questions about organizations hosting their own terminology servers:
- To host a terminology server only for IG build support , what capabilities does it need? Can it be a static website that just supports downloading ValueSet and CodeSystem resources? Does it need to support the full terminology services spec (http://build.fhir.org/terminology-service.html) or does the IG publisher use a subset?
- Is there anyway to pass credentials or a token through the publisher, so the terminology server can determine the user is authorized?
- Does that server have to support all terminology used in the IGs built pointing at it, or can it define only the local terminology and defer other content to tx.fhir.org? Or can publisher be pointed at multiple terminology servers?
Or, am I looking at this the wrong way, and the correct path is for an organization to get their vocabulary into tx.fhir.org, or to publishing an IG with their vocabulary in it?
Grahame Grieve (Aug 19 2021 at 05:33):
in principle, there is an intent to support alternative servers, though not for Hl7 published content. Documenting the requirements is still on my todo list. The publisher makes extensive use of $expand and $validate-code, and there's some very deep requirements around validate-code and particular code systems.
in principle, it may be possible to delegate some terminologies to another server. That's also on my todo list. It's also good to get vocabulary onto tx.fhir.org. That's all there is right now, actually
Elliot Silver (Aug 19 2021 at 05:49):
Grahame Grieve said:
though not for Hl7 published content
What about HL7 affiliates? Are they expected to put terminologies in tx.fhir.org?
The publisher makes extensive use of $expand and $validate-code
That's what I was afraid of. So a static website isn't enough. Even a FHIR server that supports CodeSystem and ValueSet and their associated search parameters (including canonical resolution) isn't enough. It needs to support specific terminology capabilities.
It's also good to get vocabulary onto tx.fhir.org.
This may sound silly, but why? (Other than that's all that is really supported right now.) The demands to keep vocabulary up to date will continue to grow. Will HL7 be able to handle the demand? What about organizations that have IP restrictions on their vocab, and don't want it on a public server?
Grahame Grieve (Aug 19 2021 at 05:52):
What about HL7 affiliates?
The intention is that affiliates can nominate alternative tx servers, but they can't right now
It needs to support specific terminology capabilities.
Yes. It hammers them
It's also good to get vocabulary onto tx.fhir.org.
because then everyione can access it easily.
Will HL7 be able to handle the demand?
the intent is that it will rest on the package infrastructure - so publish it through packages, tx.fhir.org will pick it up
What about organizations that have IP restrictions on their vocab, and don't want it on a public server?
Then of course it won't go on tx.fhir.org, and right now, that's not supported, though as I said, it's on the roadmap
Elliot Silver (Aug 19 2021 at 06:09):
Grahame Grieve said:
the intent is that it will rest on the package infrastructure - so publish it through packages, tx.fhir.org will pick it up
Interesting.
That circles back to an earlier thought I had -- your description of the terminology server requirements makes me think that the IG publisher ignores ~/.fhir/packages for terminology. If I had a package for my, e.g. IP-restricted vocabulary, just adding it there wouldn't do any good.
Grahame Grieve (Aug 19 2021 at 06:44):
no it only load what it's told to load. Add it as a dependency
Peter Jordan (Aug 19 2021 at 06:58):
Grahame Grieve said:
in principle, there is an intent to support alternative servers, though not for Hl7 published content. Documenting the requirements is still on my todo list. The publisher makes extensive use of $expand and $validate-code, and there's some very deep requirements around validate-code and particular code systems.
I'm wondering if those 'deep' requirements relate to use outside of the validation requirements of the IG Publisher? The fact is that no one Terminology Server is going to have all of the Code Systems and the (federated) approach I'd like to see is that a local/national Terminology Server is called by the Validator and re-directs to tx.fhir.org for HL7 vocabularies. This is very difficult to do when the $validate-code requests are POSTs with complex, non-standard, parameters (e.g. expansion profiles) - it would be so much easier if they were GET requests. Are these additional parameters required for validating IGs, or more complex operations such as the build process?
Grahame Grieve (Aug 19 2021 at 07:07):
GET / POST doesn't make any difference. The complexity does... and there's lots of that, because people want features. So the complexity isn't going away.
Grahame Grieve (Aug 19 2021 at 07:07):
but they're not non-standard. The validation part is; I need to work on that, because the validate-code API needs work to support the validator. That's one of the issues. But others relate to expansion parameters that must be supported
Gino Canessa (Aug 19 2021 at 12:07):
Peter, I believe part of the issue is that for anything HL7 publishes, they want to be able to verify it is correct. If I build an IG that validates against my local terminology server, I can say that anything is valid and correct.
Elliot Silver (Aug 19 2021 at 15:10):
Grahame Grieve said:
no it only load what it's told to load. Add it as a dependency
Even if I add a terminology package to my cache, and add it as a dependency in my IG, I can't build the IG using that vocab, because the publisher requires an actual server to validate against, correct?
Peter Jordan said:
The fact is that no one Terminology Server is going to have all of the Code Systems and the (federated) approach I'd like to see is that a local/national Terminology Server is called by the Validator and re-directs to tx.fhir.org for HL7 vocabularies.
I think a federated/distributed approach will be needed, but I don't know that we need redirects. If the publisher has a list of servers, it can try each of them until it finds the one hosting the resources it needs. Supporting multiple servers would also allow for cases where Org A hosts a terminology server with their vocab, Org B hosts a server with their vocab, there is a national server, and the default HL7 server; a redirect-chained approach couldn't handle that.
Jose Costa Teixeira (Aug 19 2021 at 15:21):
when the publisher tries each of them until it finds what it needs - that polling could become a lot of unnecessary traffic, no?
John Moehrke (Aug 19 2021 at 15:23):
unnecessary unless that result is what your IG needs. but, agreed there must be a better way.
Grahame Grieve (Aug 19 2021 at 20:07):
I can't build the IG using that vocab, because the publisher requires an actual server to validate against
The IG publisher has an internal mini terminology server that evaluates things locally when possible
Peter Jordan (Aug 19 2021 at 21:53):
Jose Costa Teixeira said:
when the publisher tries each of them until it finds what it needs - that polling could become a lot of unnecessary traffic, no?
My approach only requires a local/national Terminology Server and the re-direction of requests to tx.fhir.org for any artefacts with HL7 URLs. However, at present, it only does this for GET requests...so work to do, AND I need to understand some of the complex IG validation use cases that Grahame refers to. Are these such that they need to be handled in a single place?
Grahame Grieve (Aug 19 2021 at 22:29):
it's certainly easier for me for the terminology server to handle delegation, and I may investigate that when I work on this
Elliot Silver (Aug 19 2021 at 23:11):
Grahame Grieve said:
I can't build the IG using that vocab, because the publisher requires an actual server to validate against
The IG publisher has an internal mini terminology server that evaluates things locally when possible
What triggers non-local evaluation?
I guess what I'm wondering is if I can distribute a package of my terminology, can I get away without a terminology server?
Peter Jordan said:
My approach only requires a local/national Terminology Server and the re-direction of requests to tx.fhir.org for any artefacts with HL7 URLs.
This only works if you can link your terminology servers into one chain: (local ->) national -> HL7. How about an IG that draws on vocab from two national terminologies, or two private terminology servers? Jose's polling concerns can be dealt with through caching (perhaps), etags, etc.
Grahame Grieve (Aug 19 2021 at 23:20):
right. life is really complex. And it'll never be as simple as 'any artefacts with HL7 URLs' - tx.fhir.org does all sorts of terminologies that you'll run into accidentally.
Grahame Grieve (Aug 19 2021 at 23:20):
What triggers non-local evaluation?
When local evaluation fails. Then the IG publisher bundles up everything it has locally that's relevant and hands it off to tx.fhir.org and that tries.
Grahame Grieve (Aug 19 2021 at 23:21):
So you can have a value set that mixes a local terminology to the IG, and a terminology only known to tx.fhir.org (we have several IGs that do this)
Elliot Silver (Aug 19 2021 at 23:23):
Grahame Grieve said:
What triggers non-local evaluation?
When local evaluation fails.
:rolling_eyes:
Elliot Silver (Aug 19 2021 at 23:25):
So this would be things like a local ValueSet using a tx.fhir.org CodeSystem, and needing to check if the values are actually in the code system?
Grahame Grieve (Aug 19 2021 at 23:28):
sure. Snomed / LOINC / RxNorm, which are too big to load into the IG publisher on the fly
Elliot Silver (Aug 19 2021 at 23:31):
OK. I think I'm getting there.
Taking the opposite approach, what's needed to get terminology hosted on tx.fhir.org?
Michael Lawley (Aug 19 2021 at 23:33):
Previously(?) there were some proprietary APIs that were being used to (as I understood things) temporarily load content into the terminology server. Are these still being used, or only conditionally used if the tx server supports them?
Elliot Silver (Aug 19 2021 at 23:34):
I was thinking less about mechanics and more about policy, etc. but I suppose the mechanics are important too.
Grahame Grieve (Aug 19 2021 at 23:36):
policy wise, you just have to ask, and provide reasonable evidence that you have the authority to ask
Grahame Grieve (Aug 19 2021 at 23:37):
if you have CodeSystem or ValueSet resources, just release them as a package, and ask. Or you can make them as a contribution to https://github.com/FHIR/packages/tree/master/packages/fhir.tx.support.r4
Grahame Grieve (Aug 19 2021 at 23:37):
if they're too big for a FHIR resource, then we need to talk about that
Michael Lawley (Aug 19 2021 at 23:38):
Sorry, I wasn't asking in the context of your question @Elliot Silver . Rather I was keen to know whether there is a barrier to using other tx servers.
Michael Lawley (Aug 19 2021 at 23:51):
@Grahame Grieve wrt too big for a FHIR resource, what is the size limit? We have a FHIR CodeSystem that is more than 300MB in size
Grahame Grieve (Aug 19 2021 at 23:59):
pragmatic. My server doesn't set a limit, it just gets slow
Grahame Grieve (Aug 20 2021 at 00:01):
tx.fhir.org does run some modifications to the terminology service API, and it is on my todo list to document them. The two that stick to mind are:
- more control over validation using a paremeter to tell the terminology server what to validate
- a session mechanism that means that content is only posted to the terminology server once by the validator / IG publisher, instead of every time (makes a lot of difference to performance)
Peter Jordan (Aug 20 2021 at 05:33):
Gino Canessa said:
Peter, I believe part of the issue is that for anything HL7 publishes, they want to be able to verify it is correct. If I build an IG that validates against my local terminology server, I can say that anything is valid and correct.
Sure, that's why I re-direct to tx.fhir.org for HL7 terminologies. BTW: I've just worked on my solution for POSTs which, of course, become 307 GET requests when re-directed, so all the parameters need to go in the query string and that won't be possible where resources such as value sets are in the request payload.
Last updated: Apr 12 2022 at 19:14 UTC
