FHIR Chat · Issues around build of the UTG IG pages

Stream: terminology / utg

Topic: Issues around build of the UTG IG pages

Ted Klein (Apr 09 2020 at 22:13):

In the creation of the V3 UTG resources from the coremif source, we have several value sets which have a content logical definition of the form "this head code and all of the subconcepts in the descendent tree, but exclude the head code". The mapping to ValueSet.compose for the are in the following form (this the the one from ValueSet ActRelationshipTemporallyPertainsEnd):

Ted Klein (Apr 09 2020 at 22:13):

Ted Klein (Apr 09 2020 at 22:13):

the IG generates a pages with he following for the Expansion:

Ted Klein (Apr 09 2020 at 22:14):

Content Logical Definition
Definition

Include codes from http://terminology.hl7.org/CodeSystem/v3-ActRelationshipType where concept is-a _ActRelationshipTemporallyPertainsEnd
Exclude these codes as defined in http://terminology.hl7.org/CodeSystem/v3-ActRelationshipType
Code    Display
_ActRelationshipTemporallyPertainsEnd   ActRelationshipTemporallyPertainsEnd    A relationship that defines the relative time of the end source act based on the time of the target act.

This value set includes codes based on the following rules:

Expansion

No Expansion for this valueset (not supported by Publication Tooling)

Ted Klein (Apr 09 2020 at 22:15):

Is the <compose> formulation incorrect for the desired behavior? (a tree of codes but without the top code in the tree). Or is this an expansion operation error of some kind?

Ted Klein (Apr 09 2020 at 22:19):

@Grahame Grieve sorry forgot to tag you in the above. I will begin to list the FHIR item errors here below as in the next hours.

Grahame Grieve (Apr 09 2020 at 22:21):

well, the first question is, why not use descendent-of instead of is-a with an exclude?

Ted Klein (Apr 09 2020 at 22:53):

duh. that is what was done a long time ago with the first translation. do clearly using 'descendant-of would be MUCH smarter..let me mess with it I'm sure that will make the problem both go away and make the resulting resources simpler. I need to look at the coremif; I suspect it is mapped this way because coremif may only support is-a and exclude, which may be why it was mapped this way.

Ted Klein (Apr 09 2020 at 22:58):

@Lloyd McKenzie what do you think?

Lloyd McKenzie (Apr 09 2020 at 22:59):

I have no problem to switching, but I don't understand why no expansion is supported...

Grahame Grieve (Apr 09 2020 at 23:13):

it's on list to debug

Ted Klein (Apr 10 2020 at 00:20):

it also looks like an erroneous error message - in the form of "v3-MaterialEntityClassType error Error from server: Unable to find value set "http://terminology.hl7.org/ValueSet/v3-SpecimenAdditiveEntity" when the value set exists - can be browsed to - but based on the underlying code content and the compose rules its expansion contains zero concepts. This value set in particular is the combination of a half dozen or more other included value sets, all of which exist just fine but some of which end up with no codes in their expansion - correctly, as they are 'all descendants of' but the head code has no descendants. LOL. Great source content. But...methinks it probably should not be reported as an error, should rather be a warning.

Ted Klein (Apr 10 2020 at 00:26):

FHIR content issues. @Grahame Grieve @Lloyd McKenzie as discussed on our call, here is a list as I go through them and verify that it is not an import or other error on the UTG processing side, the list of all of the FHIR errors on the UTG build.. There are hundreds of them. These are all from my local build, hence the source URL of the resources rooted in my local build folder. There are all value sets on the FHIR Terminologies->CodeSystems->External FHIR tab (the ones with canonical URLs rooted at http://terminology.hl7.org

Ted Klein (Apr 10 2020 at 00:27):

The first batch are all of the type: Users/tedklein/Documents/tedhl7/vocabulary/Projects/UTG/gitImages/UTG/input/sourceOfTruth/fhir/attribute-estimate-type.xml
Path Severity Message
CodeSystem/attribute-estimate-type: CodeSystem.valueSet (l238/c75) error URL value "http://hl7.org/fhir/ValueSet/attribute-estimate-type" does not resolve. Where the URL is the inserted implicit value set done by the publisher operation in the code system with the <valueset> element.

Ted Klein (Apr 10 2020 at 00:29):

The second batch are all of the form: /Users/tedklein/Documents/tedhl7/vocabulary/Projects/UTG/gitImages/UTG/input/sourceOfTruth/fhir/audit-event-outcome.xml
Path Severity Message
CodeSystem/audit-event-outcome: CodeSystem error CodeSystem http://terminology.hl7.org/CodeSystem/audit-event-outcome has a 'all system' value set of http://hl7.org/fhir/ValueSet/audit-event-outcome, but doesn't have a matching system (http://hl7.org/fhir/audit-event-outcome) where the code system cannot be found. This may be some recursion thing, since the error is on a code system triggered by the implicit value set which seems to be referencing an unfindable code system of the same name...not sure. But there are quite. few of these.

Ted Klein (Apr 10 2020 at 00:30):

The third batch hopefully are the easiest to deal with; they are all id/url mismatches. An example is: /Users/tedklein/Documents/tedhl7/vocabulary/Projects/UTG/gitImages/UTG/input/sourceOfTruth/fhir/audit-source-type.xml
Path Severity Message
http://terminology.hl7.org/CodeSystem/security-source-type error Conformance resource /Users/tedklein/Documents/tedhl7/vocabulary/Projects/UTG/gitImages/UTG/input/sourceOfTruth/fhir/audit-source-type.xml - the canonical URL (http://terminology.hl7.org/CodeSystem/audit-source-type) does not match the URL (http://terminology.hl7.org/CodeSystem/security-source-type)
CodeSystem.url error Resource id/url mismatch: audit-source-type/http://terminology.hl7.org/CodeSystem/security-source-type
/Users/tedklein/Documents/tedhl7/vocabulary/Projects/UTG/gitImages/UTG/input/sourceOfTruth/fhir/audit-source-type

Lloyd McKenzie (Apr 10 2020 at 00:30):

I don't think your attempt to attach files worked

Ted Klein (Apr 10 2020 at 00:31):

I will go through probably in the morning and make a list of all of those in each of these three types of errors, and put the lists of them here.

Ted Klein (Apr 10 2020 at 00:32):

I just copy-pasted the errors - not attach the xml files. If you wish me to do that, I will for each of the ones that exhibit the problem. Do you want the files as the publisher created them, or as we imported them and gave them as input to the publisher?

Grahame Grieve (Apr 10 2020 at 20:19):

@Ted Klein can we just fix these errors? or are we fated to just keep talking about them?

Ted Klein (Apr 10 2020 at 20:34):

@Grahame Grieve Errrr...ummmm...you asked me to put a spreadsheet of them together in a Google Doc so you could figure out how to fix them consistently and systematically. I sent pretty much the entire day today doing that for you. You and Lloyd, as far as I know, are in charge of this FHIR content, so if you think they should just be fixed..then by all means DO IT. If you are asking me if it is possible for the UTG team (meaning our contracted java guy doing the import from FHIR) to fix these programmatically, then we need to see if there is any funding left for him to do so. I would be extremely pleased if alternatively you just fixed them in the current build on FHIR and told me they were fixed and we just rerun the import of the FHIR material and poof! they are fixed in UTG. Please advise.

Grahame Grieve (Apr 10 2020 at 20:39):

remind me why UTG only includes code systems and not value sets for the FHIR stuff?

Ted Klein (Apr 10 2020 at 20:40):

Because you and Lloyd said for the initial release we must not include the FHIR value sets. I gave only mild pushback since you said we would look into importing them in the second release.

Grahame Grieve (Apr 10 2020 at 20:40):

ok thanks

Ted Klein (Apr 10 2020 at 20:40):

Originally we were planning on importing them

Grahame Grieve (Apr 10 2020 at 21:02):

well, I will work through them. The first issue raises a policy question: http://hl7.org/fhir/ValueSet/attribute-estimate-type is not defined in R4. It was added afterwards, so that's why it doesn't resolve

Grahame Grieve (Apr 10 2020 at 21:03):

I think we have to move the all codes value sets to UTG

Grahame Grieve (Apr 10 2020 at 21:03):

what are you fixing manually?

Ted Klein (Apr 10 2020 at 22:05):

We should discuss with Lloyd. We can import the value sets...but that will delay initial release I am sure. We will need to do an import and then examine for and fix errors. The manual fixes are things that are mostly one-offs that it is more cost effective to fix them manually than tweak the import code for the v2 database or the v3 coremif to do them. Things like changing the is-a plus an exclude to just a descendent-of, fixing a few issues with bad names, etc. But there are some other issues that I think are either publisher issues or perhaps policy issues (or both) and I've been waiting to post them here until we figure out what to do about this big tranche. One example are a few v2 value sets which are explicitly enumerated - a subset of codes of the underlying code system. The build complains because the implicit generated 'all codes' value set is autogenerated with the same name/id as the explicit enumerated one. Can get around it by manually changing the name/id of the explicit one, easy enough - but we should decide the policy first. A few oddball items like that.

Grahame Grieve (Apr 10 2020 at 22:16):

@Lloyd McKenzie

Grahame Grieve (Apr 10 2020 at 22:16):

but why would it delay the initial release?

Ted Klein (Apr 10 2020 at 22:21):

because we have to write the java code to do the import for it. if trivial, then maybe push back by a day. If not trivial, then longer.

Ted Klein (Apr 10 2020 at 22:36):

I would like to hear from @Lloyd McKenzie because originally he made some very compelling points to wait to do it until after we think through the ramifications on the current FHIR processes for developing and updating value sets for IGs in progress.

Ted Klein (Apr 10 2020 at 22:37):

If we want to block maintenance of them in UTG, but have a copy of them to work around the current build issues, we can do that - but beer in mind the dangers of having copies of stuff and keeping them in sync.

Grahame Grieve (Apr 10 2020 at 22:54):

There's no way that IG value can or should happen through UTG. it's too knowledge specific - though vocab needs some process around this. I don't know where the scale comes from though.

Grahame Grieve (Apr 10 2020 at 22:54):

but these value sets I am talking about are core value sets

Ted Klein (Apr 10 2020 at 23:08):

well I am ambivalent. The UTG workflow architecture has the bit that certain identified code systems and value sets are surfaced for browsing on the UTG IG pages, but they cannot be sent through the change process - anything identified is trapped if it is on the list of ballot bound or product family specific objects. If it will make things work much better for core we can do this - but what about the canonical URL? I am all for anything we need to make work to get this thing rolling.

Ted Klein (Apr 10 2020 at 23:09):

If it is a small number of them - less than 100 say - we can put them in manually, it is not hard.

Ted Klein (Apr 10 2020 at 23:28):

The version thing is apparently still not working. There are a number of codes that are in the current ActCode and EntityCode (V3) code systems that are referenced in compose.include items in value sets that cannot be found - but they are there in UTG. The UTG current versions of ActCode and EntityCode are 2.0.0

Grahame Grieve (Apr 10 2020 at 23:33):

219 value sets. I've added them to UTG

Ted Klein (Apr 10 2020 at 23:33):

Are you planning to make this work: Extension url "http://hl7.org/fhir/5.0/StructureDefinition/extension-NamingSystem.url" is not valid (invalid Version "5.0"). Or will this always be an error until R5, and I need to suppress it somehow? Or should I just take the resource and put a copy of it in the local UTG resources place?

Ted Klein (Apr 10 2020 at 23:34):

wow ok. I need to build the manifest for it and the tab for the rendering. let me do that.

Grahame Grieve (Apr 10 2020 at 23:34):

yes I do need to do that

Ted Klein (Apr 10 2020 at 23:35):

Where did you put them in Git? Did you create a folder for them under SoT/fhir?

Grahame Grieve (Apr 10 2020 at 23:35):

I just put them in the fhir directory as ValueSet-{id}.xml. Will PM you the canonical URLs directly

Ted Klein (Apr 10 2020 at 23:35):

I think since we have both code systems and value sets now we should have the folder names follow the pattern of v2 and v3

Ted Klein (Apr 10 2020 at 23:35):

easier for debuggin

Grahame Grieve (Apr 10 2020 at 23:36):

well, I'll do that too then

Ted Klein (Apr 10 2020 at 23:36):

everything is SoT/productFamily/codesystem or ValueSet

Ted Klein (Apr 10 2020 at 23:37):

sorry. I have been working on the rest of the 'easy' manual fixes here and validating with my local build. I have a half dozen things to commit in the v2 and v3 space

Ted Klein (Apr 11 2020 at 00:54):

So I updated my stuff (commit and push) and I am rebuilding...you removed the FHIR stuff completely? working on the fixes?

Lloyd McKenzie (Apr 11 2020 at 02:11):

Two main reasons for avoiding migrating the value sets:

When modifying the core spec, we didn't want work groups to have to jump through UTG hoops just to change the content of a value set that's tightly bound to their model
We wanted to distinguish between value sets intended to be shared and where there was acceptance that they might be used in multiple places and where different parties might make changes (or prohibit changes) from happening vs. those that are intended to be specific to particular FHIR model elements where the only process that should have any say over the content would be the FHIR ballot

Lloyd McKenzie (Apr 11 2020 at 02:13):

I can see us perhaps needing to migrate the value sets that are referenced by the code system - i.e. the immutable ones that say "all codes from code system X". I can't see any good reason to migrate anything else. Is the issue around order of load for the publisher? Presumably we could load the FHIR core value sets after we load the UTG stuff?

Grahame Grieve (Apr 11 2020 at 02:24):

the issue is around versioning - UTG contains code systems defined in R5

Grahame Grieve (Apr 11 2020 at 03:02):

why is this error ignored?

Profiles SHOULD state their own version

Lloyd McKenzie (Apr 11 2020 at 03:57):

We have a challenge w/ UTG. We want the profiles to grab their versions from the IG, but apparently not the value sets and code systems (though I'd strongly prefer if everything did...)