Stream: uk
Topic: Terminology - SNOMED CT UK Edition
Kevin Mayfield (Jun 13 2020 at 04:14):
I want to push terminology validation in UK implementations (as it drives both SCT and validation).
As the UK onto server is still in beta, what other options do we have.?
Possibly could use the SCT built into the HL7 Validator?
Michael Lawley (Jun 13 2020 at 07:10):
What are your timeframes?
Kevin Mayfield (Jun 13 2020 at 07:19):
as soon as its workable I think we should start recommending it.
I'm pushing validation as something developers should do early on. Terminology support helps that (and helps terminlogy)
Kevin Mayfield (Jun 13 2020 at 07:25):
See also https://chat.fhir.org/#narrow/stream/179177-conformance/topic/FHIR.20Validator.20-sct.20option
Michael Lawley (Jun 13 2020 at 07:26):
I guess I'm wondering why not start with the current beta as it is there now?
Kevin Mayfield (Jun 13 2020 at 07:32):
I've just tried using FHR Validator with this parameter -tx https://ontoserver.dataproducts.nhs.uk/fhir/
Kevin Mayfield (Jun 13 2020 at 07:33):
Get this response
Kevin Mayfield (Jun 13 2020 at 07:33):
FHIR Validation tool Version 5.0.7-SNAPSHOT (Git# 6bc790d98b04). Built 2020-06-07T08:35:37.25+10:00 (6 days old)
Detected Java version: 14.0.1 from /Users/kevinmayfield/Library/Java/JavaVirtualMachines/openjdk-14.0.1/Contents/Home on x86_64 (64bit). 8192MB available
Arguments: Bundle-prescription-order.xml -version 4.0.1 -ig https://packages.simplifier.net/uk.core.r4/-/uk.core.r4-1.1.0.tgz -tx https://ontoserver.dataproducts.nhs.uk/fhir/
Directories: Current = /Users/kevinmayfield/github/nhsconnect/eps/Examples, Package Cache = /Users/kevinmayfield/.fhir/packages
.. FHIR Version 4.0, definitions from hl7.fhir.r4.core#4.0.1
.. connect to tx server @ https://ontoserver.dataproducts.nhs.uk/fhir/
Exception in thread "main" org.hl7.fhir.exceptions.FHIRException: Unable to connect to terminology server. Use parameter '-tx n/a' tun run without using terminology services to validate LOINC, SNOMED, ICD-X etc. Error = Error parsing response message: Unknown Content acceptUnknown @ START_TAG seen ...<fhirVersion value="3.0.2"/><acceptUnknown value="extensions"/>... @1:1239
at org.hl7.fhir.r5.context.SimpleWorkerContext.connectToTSServer(SimpleWorkerContext.java:258)
at org.hl7.fhir.validation.ValidationEngine.connectToTSServer(ValidationEngine.java:718)
at org.hl7.fhir.validation.ValidationEngine.setTerminologyServer(ValidationEngine.java:309)
at org.hl7.fhir.validation.ValidationEngine.<init>(ValidationEngine.java:339)
at org.hl7.fhir.validation.cli.services.ValidationService.getValidator(ValidationService.java:180)
at org.hl7.fhir.validation.Validator.main(Validator.java:175)
Caused by: org.hl7.fhir.r5.utils.client.EFhirClientException: Error parsing response message: Unknown Content acceptUnknown @ START_TAG seen ...<fhirVersion value="3.0.2"/><acceptUnknown value="extensions"/>... @1:1239
at org.hl7.fhir.r5.utils.client.ClientUtils.unmarshalReference(ClientUtils.java:352)
at org.hl7.fhir.r5.utils.client.ClientUtils.issueResourceRequest(ClientUtils.java:237)
at org.hl7.fhir.r5.utils.client.ClientUtils.issueResourceRequest(ClientUtils.java:219)
at org.hl7.fhir.r5.utils.client.ClientUtils.issueResourceRequest(ClientUtils.java:210)
at org.hl7.fhir.r5.utils.client.ClientUtils.issueGetResourceRequest(ClientUtils.java:146)
at org.hl7.fhir.r5.utils.client.FHIRToolingClient.getCapabilitiesStatementQuick(FHIRToolingClient.java:181)
at org.hl7.fhir.convertors.txClient.TerminologyClientR5.getCapabilitiesStatementQuick(TerminologyClientR5.java:94)
at org.hl7.fhir.r5.context.SimpleWorkerContext.connectToTSServer(SimpleWorkerContext.java:256)
... 5 more
Caused by: org.hl7.fhir.exceptions.FHIRFormatError: Unknown Content acceptUnknown @ START_TAG seen ...<fhirVersion value="3.0.2"/><acceptUnknown value="extensions"/>... @1:1239
at org.hl7.fhir.r5.formats.XmlParserBase.unknownContent(XmlParserBase.java:312)
at org.hl7.fhir.r5.formats.XmlParser.parseCapabilityStatement(XmlParser.java:3929)
at org.hl7.fhir.r5.formats.XmlParser.parseResource(XmlParser.java:26381)
at org.hl7.fhir.r5.formats.XmlParserBase.parse(XmlParserBase.java:136)
at org.hl7.fhir.r5.formats.XmlParserBase.parse(XmlParserBase.java:120)
at org.hl7.fhir.r5.formats.ParserBase.parse(ParserBase.java:61)
at org.hl7.fhir.r5.utils.client.ClientUtils.unmarshalReference(ClientUtils.java:345)
Grahame Grieve (Jun 13 2020 at 07:35):
I'll look into it tomorrow sometime
Michael Lawley (Jun 13 2020 at 07:49):
Does it work with https://r4.ontoserver.csiro.au/fhir -- it also has SNOMED UK loaded
Michael Lawley (Jun 13 2020 at 07:50):
Note, the UK Beta is STU3 not R4
Kevin Mayfield (Jun 13 2020 at 12:12):
Yep, that did work.
Grahame Grieve (Jun 13 2020 at 21:05):
@Kevin Mayfield if you run with the parameter -version 3.0 then -tx https://ontoserver.dataproducts.nhs.uk/fhir/ should work
Kevin Mayfield (Jun 14 2020 at 05:06):
Yes the NHSD terminology server works (but then resource validation has errors).
Which leads to: Should the terminology server be the same version as the IG?
Kevin Mayfield (Jun 14 2020 at 07:02):
Researched a little more and what I've found is:
- NHS Digital Terminology STU3 does validate codes in ValueSets
- ValueSet uri's are inconsistent between versions (this is logged)
- FHIR Validator does not like mixing versions (terminology and IG need to be same versions)
- Quality of validation using FHIR Validator and terminology server is very good
- https://r4.ontoserver.csiro.au/fhir/ doesn't contain UK Drug Extension
Kevin Mayfield (Jun 14 2020 at 07:06):
- A terminology server (with UK SNOMED) is required (not using a -tx results in validation failure)
Kevin Mayfield (Jun 14 2020 at 07:07):
May need some workarounds to get a clean validation.
Kevin Mayfield (Jun 14 2020 at 07:11):
- https://ontoserver.dataproducts.nhs.uk is on an old UK SNOMED version (20190807)
Kevin Mayfield (Jun 14 2020 at 07:12):
- https://r4.ontoserver.csiro.au is on the current UK SNOMED version
Grahame Grieve (Jun 14 2020 at 19:26):
FHIR Validator does not like mixing versions
yes.
Rob Hausam (Jun 19 2020 at 15:02):
We are working to load the UK Edition on tx.fhir.org. But the current loader code assumes that all of the content is provided in a single set of files, which isn't the case with the UK distribution. The Clinical and Drug extensions and the International edition are distributed in separate folders, which will need to be combined (accounting for the proper dependencies). I do have access to the distributions through TRUD, so that isn't a problem, but deciding what to load and how best to get it loaded is. Should we load the latest SNOMEDCT2_29.3.0_20200610000001 Clinical Edition release and the SNOMEDCT2_29.2.0_20200513000001 Drug Extension release (which I think is the latest - I need to double check that)? And maybe we should use the BETA releases, as that will be the structure going forward? The releases are pretty frequent, especially for the drugs and with the recent COVID-19 micro-releases, and we probably won't try to stay absolutely current with all of it. We appreciate any advice and help we can get from the UK community in figuring out what is the best way forward on this.
Richard Kavanagh (Jun 19 2020 at 17:43):
@Michael Lawley may have some insights as I know they have worked with the peculiarities of the UK SNOMED edition.
Rob Hausam (Jun 19 2020 at 17:55):
Yes, I've discussed it a bit with Michael, already, and we'll likely discuss it further. I think we're fairly clear on what needs to be done to load the content. Long term I think that's going to need some changes to the loader code. But I think what we most need to figure out right now is what extension/version/format combinations we should target for the initial content load and then how we plan to keep that content maintained (and on what frequency) going forward.
Grahame Grieve (Jun 23 2020 at 21:00):
what needs to change in the loader code?
Rob Hausam (Jun 23 2020 at 21:10):
The code needs to support specifying multiple source folders (snapshot or possibly full) for the content that need to be combined as the source of a complete edition, and then once the content has been combined to run the snapshot extraction algorithm on the combined content to create the edition that needs to be loaded.
Grahame Grieve (Jun 23 2020 at 21:10):
what's the combination rules? Everytime I've looked at something like this, there's redefinitions or versioning issues that mean that it's not just a matter of reading more files
Rob Hausam (Jun 23 2020 at 21:20):
That's the snapshot extraction algorithm. I can locate a reference for you (or Michael can). If you have other things to do (tongue in cheek question), I'm happy to take the lead on working on it (I'm sure with some of your help).
Grahame Grieve (Jun 23 2020 at 21:25):
sure a reference would be good
Michael Lawley (Jun 24 2020 at 06:38):
It's RF2 and because you're combining multiple packages you need to treat the content as if it conformed to the FULL ReleaseType and then compute a SNAPSHOT ReleaseType. I'd like to be able to point at http://snomed.org/rf2 but it seems that the real URL is http://snomed.org/rfs
I had a quick look to try to point at the right page that talks about generating a SNAPSHOT from FULL, but failed -- I'm not sure such a page exists.
You at least need to process the MDRS - Module Dependency Reference Set (https://confluence.ihtsdotools.org/x/M4e2AQ or https://confluence.ihtsdotools.org/display/DOCRELFMT/5.2.12+Module+Dependency+Reference+Set) - which gives you a set of moduleId-effectiveTime pairs for the version you're importing.
Then, for every "component" (identified by the id column) you take the row with the latest effectiveTime value as long as it is NOT GREATER than the effectiveTime from the corresponding pairs above with the same moduleId. (so it is not enough to just take the latest effective time; you always need to first filter against the MDRS pairs).
All of this, and the fact that the files can't be assumed to be sorted in a useful manner, means that you need to load a huge pile of stuff into memory and/or do two passes over the content.
Adam Flinton (Jun 25 2020 at 12:00):
"All of this, and the fact that the files can't be assumed to be sorted in a useful manner, means that you need to load a huge pile of stuff into memory and/or do two passes over the content."
Yup used to blow stuff up when I was trying to combine the UK & UK drugs into one..... eventually gave up as had other things to do.
Jeremy Rogers (Apr 12 2022 at 17:06):
@Rob Hausam WRT your June 2020 post about the difficulty of loading the UK Edition onto tx.fhir.org, because it was scattered across multiple files and zipped archives...you might try your luck again with the New! Simpler! Monolithic! Bundling of the UK extensions content. Its a snapshot-only distro across a single set of the 22-odd files that make up any well formatted SNOMED edition or extension release distro bundle. Content-wise, it therefore includes the union of the relevant International Edition plus (currently) the latest versions of the UK Clinical, Drug and Pathology extensions. Can be found on the dreaded TRUD here: https://isd.digital.nhs.uk/trud/users/guest/filters/0/categories/26/items/1799/releases. New releases expected every 4 weeks, as per the schedule for releases of the UK Drug Extension component within.
Last updated: Apr 12 2022 at 19:14 UTC
