Stream: mapping-framework
Topic: R3 to R4 using validator_cli.jar
Declan Kieran (Dec 06 2021 at 15:52):
I've created an FML map based on
http://www.hl7.org/fhir/medicationrequest-version-maps.html#11.1.9.1
there are some changes, including a Reference group function added where I'd attempted to get the context transform working, and I also changed the copy operations as it was complaining about some of the simple src -> tgt copy mappings. The FML is
map "http://hl7.org/fhir/StructureMap/MedicationRequest3to4" = "R3 to R4 Conversions for MedicationRequest"
uses "http://hl7.org/fhir/3.0/StructureDefinition/MedicationRequest" alias MedicationRequestR3 as source
uses "http://hl7.org/fhir/StructureDefinition/MedicationRequest" alias MedicationRequest as target
uses "http://hl7.org/fhir/3.0/StructureDefinition/Reference" alias ReferenceR3 as source
uses "http://hl7.org/fhir/StructureDefinition/Reference" alias Reference as target
imports "http://hl7.org/fhir/StructureMap/*3to4"
group MedicationRequest(source src : MedicationRequestR3, target tgt : MedicationRequest) <<type+>> {
src.id as id -> tgt.id = id;
src.text as text -> tgt.text = text;
src.extension as ext -> tgt.extension = ext;
src.identifier as identifier -> tgt.identifier = identifier;
src.definition as definition -> tgt.instantiatesCanonical = definition;
src.basedOn as basedOn -> tgt.basedOn = basedOn;
src.groupIdentifier as groupIdentifier -> tgt.groupIdentifier = groupIdentifier;
src.status as status -> tgt.status = status;
src.intent as intent -> tgt.intent = intent;
src.category as category -> tgt.category = category;
src.priority as priority -> tgt.priority = priority;
src.medication : CodeableConcept as vs -> tgt.medication = create('CodeableConcept') as vt then CodeableConcept(vs, vt);
src.medication : Reference as vs -> tgt.medication = create('Reference') as vt then Reference(vs, vt);
src.subject as subject -> tgt.subject = subject;
//HERE - Seems to not copy over
src.context -> tgt.encounter;
//src.context : Reference as value -> tgt.encounter = value "value";
//src.context : Reference as vs -> tgt.encounter = create('Reference') as vt then Reference(vs, vt);
//src.context as context -> tgt.context = context;
//src.context as context -> tgt.encounter = context;
src.supportingInformation as supportingInformation -> tgt.supportingInformation = supportingInformation;
src.authoredOn as authoredOn -> tgt.authoredOn = authoredOn;
//HERE - Seems to not copy over
src.requester as vs0 then requester(vs0, tgt);
src.recorder as recorder -> tgt.recorder = recorder;
src.reasonCode as reasonCode -> tgt.reasonCode = reasonCode;
src.reasonReference as reasonReference -> tgt.reasonReference = reasonReference;
src.note as note -> tgt.note = note;
src.dosageInstruction as dosageInstruction -> tgt.dosageInstruction = dosageInstruction;
src.dispenseRequest as vs0 -> tgt.dispenseRequest as vt0 then dispense(vs0, vt0);
src.substitution as vs0 -> tgt.substitution as vt0 then subst(vs0, vt0);
src.priorPrescription as priorPrescription -> tgt.priorPrescription = priorPrescription;
src.detectedIssue as detectedIssue -> tgt.detectedIssue = detectedIssue;
src.eventHistory as eventHistory -> tgt.eventHistory = eventHistory;
}
group requester(source src, target tgt) {
src.agent -> tgt.requester;
src.onBehalfOf as v -> tgt.extension as vt, vt.url = 'http://hl7.org/fhir/3.0/StructureDefinition/extension-MedicationRequest.requester.onBehalfOf', vt.value = v;
}
group dispense(source src, target tgt) {
src.validityPeriod -> tgt.validityPeriod;
src.numberOfRepeatsAllowed -> tgt.numberOfRepeatsAllowed;
src.quantity -> tgt.quantity;
src.expectedSupplyDuration -> tgt.expectedSupplyDuration;
src.performer -> tgt.performer;
}
group subst(source src, target tgt) {
src.allowed as vs -> tgt.allowed = create('boolean') as vt then boolean(vs, vt);
src.reason -> tgt.reason;
}
group Reference(source src : ReferenceR3, target tgt : Reference) <<type+>> {
src.reference as reference -> tgt.reference = reference;
src.identifier as identifier -> tgt.identifier = identifier;
src.display as display -> tgt.display = display;
}
and the resource to convert is
{
"resourceType": "MedicationRequest",
"id": "123",
"text": {
"status": "generated",
"div": "<div xmlns=\"http://www.w3.org/1999/xhtml\"><p><b>Generated Narrative with Details</b></p><p><b>id</b>: 123</p><p><b>identifier</b>: 12345 (OFFICIAL)</p><p><b>status</b>: active</p><p><b>intent</b>: order</p><p><b>medication</b>: <a>prescribed medication</a></p><p><b>subject</b>: <a>Donald Duck</a></p><p><b>context</b>: <a>encounter that leads to this prescription</a></p><p><b>authoredOn</b>: 01/03/2015</p><h3>Requesters</h3><table><tr><td>-</td><td><b>Agent</b></td><td><b>OnBehalfOf</b></td></tr><tr><td>*</td><td><a>Patrick Pump</a></td><td><a>Organization/f002</a></td></tr></table><p><b>reasonCode</b>: Essential hypertension (disorder) <span>(Details : {SNOMED CT code '59621000' = 'Essential hypertension', given as 'Essential hypertension (disorder)'})</span></p><p><b>dosageInstruction</b>: </p></div>"
},
"extension": [
{
"url": "http://hl7.org/fhir/3.0/StructureDefinition/extension-MedicationRequest.requester.onBehalfOf",
"valueReference": {
"reference": "Organization/f002"
}
}
],
"identifier": [
{
"use": "official",
"system": "http://www.bmc.nl/portal/prescriptions",
"value": "12345"
}
],
"status": "active",
"intent": "order",
"medicationReference": {
"reference": "Medication/med0316",
"display": "prescribed medication"
},
"subject": {
"reference": "Patient/pat1",
"display": "Donald Duck"
},
"context": {
"reference": "Encounter/f001",
"display": "encounter that leads to this prescription"
},
"authoredOn": "2015-03-01",
"requester": {
"agent": {
"reference": "Practitioner/f007",
"display": "Patrick Pump"
}
},
"reasonCode": [
{
"coding": [
{
"system": "http://snomed.info/sct",
"code": "59621000",
"display": "Essential hypertension (disorder)"
}
]
}
],
"dosageInstruction": [
{
"sequence": 1,
"text": "Take one tablet daily as directed"
}
]
}
from STU3 to R4. For the most part it is just copying data from one to the other and that works. There are two structural differences in the stu3 and r4 resources that correspond to
stu3.context -> r4.encounter
r4.requester -> r4.requester
These are not included in the r4 output when running the transform. The command I'm using to run this is
java -jar validator_cli.jar ./medicationrequest-stu3.json -transform http://hl7.org/fhir/StructureMap/MedicationRequest3to4 -version 4.0.1 -ig ./MedicationRequest.map -output ./medicationrequest-r4.json
the json diff of the r4 and the expected r4 is
$ diff medicationrequest-r4.json expected-medicationrequest-r4.json
8a9,12
> "encounter": {
> "display": "encounter that leads to this prescription",
> "reference": "Encounter/f001"
> },
40a45,48
> "requester": {
> "display": "Patrick Pump",
> "reference": "Practitioner/f007"
> },
i.e. the context and requester objects are lost during the transform. It runs without error and the context and requester rules are shown in the output when running (truncated to fit char limit)
Start Transform http://hl7.org/fhir/StructureMap/MedicationRequest3to4
Group : MedicationRequest; vars = source variables [src: (MedicationRequest)], target variables [tgt: (MedicationRequest)], shared variables []
rule : id; vars = source variables [src: (MedicationRequest)], target variables [tgt: (MedicationRequest)], shared variables []
rule : text; vars = source variables [src: (MedicationRequest)], target variables [tgt: (MedicationRequest)], shared variables []
rule : extension; vars = source variables [src: (MedicationRequest)], target variables [tgt: (MedicationRequest)], shared variables []
rule : identifier; vars = source variables [src: (MedicationRequest)], target variables [tgt: (MedicationRequest)], shared variables []
...
rule : context; vars = source variables [src: (MedicationRequest)], target variables [tgt: (MedicationRequest)], shared variables []
rule : supportingInformation; vars = source variables [src: (MedicationRequest)], target variables [tgt: (MedicationRequest)], shared variables []
rule : authoredOn; vars = source variables [src: (MedicationRequest)], target variables [tgt: (MedicationRequest)], shared variables []
rule : requester; vars = source variables [src: (MedicationRequest)], target variables [tgt: (MedicationRequest)], shared variables []
Group : requester; vars = source variables [src: (Reference)], target variables [tgt: (MedicationRequest)], shared variables []
rule : agent; vars = source variables [src: (Reference)], target variables [tgt: (MedicationRequest)], shared variables []
rule : onBehalfOf; vars = source variables [src: (Reference)], target variables [tgt: (MedicationRequest)], shared variables []
...
...success
Done. Times: Loading: 00:13.0413. Max Memory = 3Gb
Any thoughts @Oliver Egger?
Declan Kieran (Dec 07 2021 at 09:38):
Declan Kieran said:
I was able to get some data into an encounter object by adding
src -> tgt.encounter as enc then {
src -> enc.reference = "hello" "rule_a";
} "rule_b";
to the map, but when trying to take it from the context in the STU3 MedicationRequest
src -> tgt.encounter as enc then {
src.context as con -> enc.reference = con.reference "rule_a";
} "rule_b";
It runs without error, but no encounter object in the output. It seems like its not getting the data from the context object...
Edit:
Another interesting observation is when I change the name of reference to something invalid, i.e.
src -> tgt.encounter as enc then {
src -> enc.xreference = "hello" "rule_a";
} "rule_b";
I get a nice error, but when I include the context, i.e.
src -> tgt.encounter as enc then {
src.context as con -> enc.xreference = "hello" "rule_a";
} "rule_b";
I don't get an error (or an encounter object in the output...)
Oliver Egger (Dec 07 2021 at 10:29):
@Declan Kieran Like this the source will be parsed as an R4 resource and most probably it is lenient that there is not an error thrown for context, requester. I think you would need first to do the version conversion with the validator that you have an R4 resource because this would probably adjust the context for R3 and R4 correctly.
Declan Kieran (Dec 07 2021 at 11:08):
Oliver Egger said:
Declan Kieran Like this the source will be parsed as an R4 resource and most probably it is lenient that there is not an error thrown for context, requester. I think you would need first to do the version conversion with the validator that you have an R4 resource because this would probably adjust the context for R3 and R4 correctly.
Thanks @Oliver Egger , I did have the source set as
uses "http://hl7.org/fhir/3.0/StructureDefinition/MedicationRequest" alias MedicationRequestR3 as source
But, yes it doesn't seem to be a stringent as opposed to the R4.
I've ran the same source through the validator_cli using the parameters for version conversion,
java -jar validator_cli.jar ./medicationrequest-stu3-apim.json -version 3.0 -to-version 4.0 -output ./to-version-r4.json
and interestingly I get an error on the context conversion
rule : context; vars = source variables [src: (MedicationRequest)], target variables [tgt: (MedicationRequest)], shared variables []
...Failure: Exception executing transform tgt.encounter = create() as vvv on Rule "context": Attempt to get types for an invalid property 'encounter' on type MedicationRequest
org.hl7.fhir.exceptions.FHIRException: Exception executing transform tgt.encounter = create() as vvv on Rule "context": Attempt to get types for an invalid property 'encounter' on type MedicationRequest
at org.hl7.fhir.r5.utils.structuremap.StructureMapUtilities.runTransform(StructureMapUtilities.java:1746)
at org.hl7.fhir.r5.utils.structuremap.StructureMapUtilities.processTarget(StructureMapUtilities.java:1617)
at org.hl7.fhir.r5.utils.structuremap.StructureMapUtilities.executeRule(StructureMapUtilities.java:1206)
at org.hl7.fhir.r5.utils.structuremap.StructureMapUtilities.executeGroup(StructureMapUtilities.java:1193)
at org.hl7.fhir.r5.utils.structuremap.StructureMapUtilities.transform(StructureMapUtilities.java:1155)
at org.hl7.fhir.validation.ValidationEngine.transformVersion(ValidationEngine.java:663)
at org.hl7.fhir.validation.cli.services.ValidationService.transformVersion(ValidationService.java:248)
at org.hl7.fhir.validation.ValidatorCli.doValidation(ValidatorCli.java:259)
at org.hl7.fhir.validation.ValidatorCli.main(ValidatorCli.java:163)
Caused by: org.hl7.fhir.exceptions.FHIRException: Attempt to get types for an invalid property 'encounter' on type MedicationRequest
at org.hl7.fhir.r5.model.Base.getTypesForProperty(Base.java:313)
at org.hl7.fhir.r5.elementmodel.Element.getTypesForProperty(Element.java:740)
at org.hl7.fhir.r5.utils.structuremap.StructureMapUtilities.runTransform(StructureMapUtilities.java:1642)
... 8 more
The error seems to be saying encounter is an invalid property? Is your matchbox server able to run the -to-version type transform? I have it running in an ide, so I can debug through, but I might also see if I can get a debugger on the validator_cli, unless of course I'm just doing something silly here!
Oliver Egger (Dec 07 2021 at 11:26):
it looks like that you get one step further no? I'm afraid you need to debug to check if you have really R3 and R4 MedicationRequest StructureDefinitions loaded (and then internally converted to R5), i'm not familiar with the conversion functionality of validator_cli, sorry. matchbox does not support version conversion at the moment.
Declan Kieran (Dec 07 2021 at 11:29):
Thanks Oliver, I'll update on any progress I make.
Declan Kieran (Dec 08 2021 at 09:20):
@Oliver Egger I have it working for my example now. It seems that only the StructureDefintions for version 3 are being loaded into the ValidationEngine. In the ValidationCli class (with the main), before the validator is initialised, the definitions string is set to the source, i.e. 3.0.2 in this case. When it loads the npm package, it loads version "hl7.fhir.r3.core#3.0.2"
The function getTargetResourceFromStructureMap searches the initialised ValidationEngine for a StuctureDefinition that matches the target url, i.e. http://hl7.org/fhir/StructureDefintion/MedicationRequest.
The list it searches in the context only has version 3 StructureDefintions, so it uses that and when it gets the properties for the conversion, it doesn't find encounter since that is not in STU3. There is also a HashMap (context.structures.map) in the ValidationEngine object that seems to have all StructureDefintions, i.e. 2, 3 and 4, so I exposed this just by making a few things public (super hacky...) and searched it and found these StructureDefintions for MedicationRequest (which are a mixture of r3 and r4)
r4-MedicationRequest
http://hl7.org/fhir/4.0/StructureDefinition/MedicationRequest|4.0.1
http://hl7.org/fhir/4.0/StructureDefinition/MedicationRequest|4.0
MedicationRequest
http://hl7.org/fhir/StructureDefinition/MedicationRequest
http://hl7.org/fhir/StructureDefinition/MedicationRequest|#0
http://hl7.org/fhir/4.0/StructureDefinition/MedicationRequest
r3-MedicationRequest
http://hl7.org/fhir/3.0/StructureDefinition/MedicationRequest
http://hl7.org/fhir/3.0/StructureDefinition/MedicationRequest|#0
I hardcoded (more hack...) the one with the name
http://hl7.org/fhir/4.0/StructureDefinition/MedicationRequest
It complained about not finding id (which was in the source input), but when id was removed it worked for the rest of the source example. i.e. context was converted to encounter and the structure of requester was updated, which is what I expected originally.
So not sure if this is more a package management problem, i.e. http://hl7.org/fhir/StructureDefintion/MedicationRequest should be r4. Or maybe something needs be added to the ValidationEngine to ensure when the target is being handled, it doesn't use source definitions. I'm not sure how to configure the npm packages yet, but I'm sure someone in nhsd will help me with that. If it becomes apparent a code fix might be appropriate, would it be worth making a PR for it?
edit:
For to mention as it didn't seem to be relevant to make it work in the end, but on the creation of some Elements, there is an exception thrown and ignored
Method threw 'java.lang.NullPointerException' exception. Cannot evaluate org.hl7.fhir.r5.elementmodel.Element.toString()
org.hl7.fhir.r5.elementmodel.Element. does have an implementation of toString(). It's maybe not related, but thought it might be worth mentioning...
Nikolai Schwertner (Mar 05 2022 at 15:41):
I too have been trying to test the R3->R4 maps via the FHIR validator, but it does appear to be broken. So far I have:
wget https://github.com/hapifhir/org.hl7.fhir.core/releases/latest/download/validator_cli.jar
wget -O medicationrequest-stu3.json http://hl7.org/fhir/stu3/medicationrequest0331.json
java -jar validator_cli.jar ./medicationrequest-stu3.json -transform http://hl7.org/fhir/StructureMap/MedicationRequest3to4 -version 3.0.2 -ig hl7.fhir.xver.r4#1.2.0 -output ./medicationrequest-r4.json
Sadly, this fails with org.hl7.fhir.exceptions.FHIRException: No matches found for rule for 'Reference to BackboneElement' from http://hl7.org/fhir/StructureMap/MedicationRequest3to4, from rule 'agent'.
I believe the issue may be similar to what others have encountered (the transformations package does not seem to properly reference the R4 structure definitions, if I understand correctly). Any ideas how to solve this? Are there any unofficial updates to the hl7.fhir.xver.r4 package anywhere beyond 1.2.0?
@Lloyd McKenzie @Declan Kieran @Oliver Egger @Vadim Peretokin @Gino Canessa @Alexander Zautke
Lloyd McKenzie (Mar 05 2022 at 19:17):
@Grahame Grieve
Declan Kieran (Mar 07 2022 at 10:39):
@Nikolai Schwertner
I've had a go at transforming this resource using the following (note using -to-version instead of specifying -transform)
java -jar validator_cli.jar medicationrequest-stu3.json -version 3.0 -to-version 4.0 -output medication_request_r4.json
In the demo I did here
https://github.com/declankieran-nhsd/hapi-fhir-3to4-demo
I had to remove the ID's from all the resources, not sure I put this in the readme, so apologies for that! Otherwise it fail with an error like this
rule : Resource-id; vars = source variables [src: (MedicationRequest)], target variables [tgt: (MedicationRequest)], shared variables []
...Failure: Exception executing transform tgt.id = create() as vvv on Rule "Resource-id": null
org.hl7.fhir.exceptions.FHIRException: Exception executing transform tgt.id = create() as vvv on Rule "Resource-id": null
Not handling ID's makes sense though from a migration perspective, but I suppose there might be uses cases where you'd want to maintain the same ID for a resource in a new system.
Another thing I found was it didn't handle contained resources, so I just removed those from the input examples. Again, sort of makes sense as you'd probably want to traverse though contained/linked resources at a different layer.
So to transform the example you've used, I extracted the contained medication resources and put it in its own file and it successfully transformed using
java -jar validator_cli.jar medication.json -version 3.0 -to-version 4.0 -output medication_r4shell
For the rest of the example (without the contained), I reran the validator_cli
java -jar validator_cli.jar medicationrequest-stu3.json -version 3.0 -to-version 4.0 -output medication_request_r4.json
But I've got an error about doseAndRate
rule : doserate; vars = source variables [src: (Dosage)], target variables [tgt: (Dosage)], shared variables []
condition [dose.exists() or rate.exists()] for dosageInstruction=Dosage[4 children] : true
Exception in thread "main" java.lang.Error: Unrecognised name doseAndRate on dosageInstruction
at org.hl7.fhir.r5.elementmodel.Element.makeProperty(Element.java:471)
Which is odd as there isn't a doseAndRate element in the example and it's cardinality is 0..*. It also doesn't seem to be in the map here https://www.hl7.org/fhir/medicationrequest-version-maps.html
So I removed the dosageInstruction and the rest of it transforms. I haven't checked if the transform is correct, but by line count it looks like stuff carried across..
22 medication.json
26 medication_r4.json
66 medication_request_r4-nodosage.json
64 medicationrequest-stu3-nodosage.json
To fix the dosage error, you might need to start playing with the FML and generating a new StructureMap to see if you can get round the error, but someone else might be able to correct me on that. You'd then need to put the StructureMap into the xver package (again, for the way I'm doing it, there might be better ways)
The modified xver package I'm using is here
You need to unpack that to the directory ~/.fhir/packages/hl7.fhir.xver.r4#1.2.0, as this is where the validator_cli will read it from
This will give you the modification to the target URL's that will make it run like it does in the pipeline in the repo mentioned.
Example run
https://github.com/declankieran-nhsd/hapi-fhir-3to4-demo/runs/5101135359?check_suite_focus=true
Pipeline yaml
https://github.com/declankieran-nhsd/hapi-fhir-3to4-demo/blob/main/.github/workflows/validate-transforms.yml
Nikolai Schwertner (Mar 07 2022 at 18:09):
@Declan Kieran Thank you for sharing.
It looks like
java -jar validator_cli.jar medicationrequest-stu3.json -transform http://hl7.org/fhir/StructureMap/MedicationRequest3to4 -version 3.0.2 -ig hl7.fhir.xver.r4#1.2.0 -output medicationrequest-r4.json
and
java -jar validator_cli.jar medicationrequest-stu3.json -version 3.0 -to-version 4.0 -output medicationrequest-r4.json
produce the same result.
Your steps worked on my end as well with the medication request example. I still had to manually remove the requester attribute to squash the agent error. Unfortunately, the conversion is not very exciting without the dosage and requester as the stu3 and r4 versions are identical under these conditions. The only difference I noticed was for the contained Medication (extracted into a separate resource) where the isBranded attribute is converted into an extension.
I tried converting the example at https://www.hl7.org/fhir/STU3/medicationexample0316.json but that gives Exception executing transform tgt.strength = create() as vvv on Rule "amount": Attempt to get types for an invalid property 'strength' on type BackboneElement. I tried a couple other examples from different resource types only to run into an assortment of errors.
I think the conclusion is that the hl7.fhir.xver.r4 package would need to be fixed before it can be used with the validator tool to convert the sample resources convincingly.
Declan Kieran (Mar 07 2022 at 22:15):
@Nikolai Schwertner Yes I agree, I think there is more work to be done on the maps, but for the resources I was able to transform, the transforms appeared to mostly correct out of 794, although my method for automatically comparing the transforms had some limitations, i.e. it only examined keys. However, it was quite good in general, 573 out 794 with no issues detected by my script
https://github.com/declankieran-nhsd/hapi-fhir-3to4-demo/tree/main/examples
and of the ones where there was something detected, most of them are invalid data or elements that were renamed. This would suggest that the StructureMap utilities can a handle good range of complexity. Also the conversion of FML to StructureMap appears to be rock solid.
The isBrand you mentioned being moved to an extension is something that I also saw happen in the examples I tested and shows the maps are pretty well developed. isBrand is only in STU3 and not in R4, so it is maintaining the data in case you might want to "round-trip" (see https://build.fhir.org/versions.html#extensions). There is however probably even more work still to do there on making that complete.
In terms of the work that needs done to make it production level, I'd say the maps that are there would be a good starting point to develop on for whatever workflows you're looking to implement. If you're using everything, then of course that's not helpful! But in developing your own maps, it seems like a good opportunity to provide back to general community as this is definitely something that will be beneficial across companies and organisations, and it would make sense to share this to allow for cross site collaboration. Not sure what the process would be to feed that sort of stuff back @Lloyd McKenzie ?. NHS Digital are currently looking to provide maps for their various profiles, which will eventually be made public I believe.
Also, there were unit tests within the core libraries for the maps, but they are no longer being maintained. This fix might help, but I'm guessing given the code is old, there'd probably be more issues there.
The script that automatically checks the transforms in the repo I used for testing, doesn't refer to any mappings to do it. I was thinking this would be a useful way to approach it, so that it could be used to fix any issues in the mappings. If things are renamed it highlights that as possibly lost or renamed, as there is no way of determining that without referencing a mapping of some sort. However once I have specific maps that are well tested for the particular set of resources I'm currently looking at, I think I'll extend the code to reference the maps to give me a cleaner test report, but its actually handier for testing at the moment as it is. Definitely still work to do on this, and its a wild tricky problem!
Lloyd McKenzie (Mar 07 2022 at 23:42):
This is more of an @Grahame Grieve question than a 'me' question. Because we're doing an R4B release, there's an opportunity to fix the maps. Whether there's energy/bandwidth, I don't know.
Last updated: Apr 12 2022 at 19:14 UTC
