FHIR Chat · FHIR Mapping language in real world applications · mapping-framework

Stream: mapping-framework

Topic: FHIR Mapping language in real world applications

Yannick Börner (Dec 06 2020 at 15:17):

Hi everyone, we are currently using the mapping language very extensively to map from different source formats to FHIR and back, for instance CSV, v2, vcf and custom json. During development of these maps, we ran into syntactic and semantic limits of the Mapping Language in quite a few cases. As far as I understood, the mapping language was originally intended to map from FHIR to FHIR so it comes as no surprise that this was the case. I was wondering whether this is still the desired use case or if there are some pushes to also use the language for other formats apart from FHIR?

If so, I would be happy to share some of the experience from the past couple of months. We identified a few concepts that we needed in a lot of cases for the mappings.

Oliver Egger (Dec 06 2020 at 15:40):

we use the FHIR mapping language for CDA to FHIR (and back) and for other non-FHIR based formats to FHIR, so we would be very interested in your experiences!

Lloyd McKenzie (Dec 06 2020 at 15:47):

Improvement is always welcome. Share away!

Lin Zhang (Dec 06 2020 at 23:38):

Eager for such experiences:clap:

Yannick Börner (Dec 07 2020 at 09:35):

To start, I'll just share some key concepts that we needed in many cases:

1. Global and local let-statements
This is kind of the aforementioned let-statement discussed by @Alexander Zautke above. It is a variable you can store resources or primitive values in. We implemented it and it proved to be vital in quite a few situations. Currently, it is a local statement but an additional global statement would be even more helpful. A few use cases:

• Reusing extracted values in many different places (global and local)
• Defining constants (global)
• We currently use it to fake an if condition (see below) (local)
• Easy reuse of parameters between groups (global)
• Quickly creating copies of resources that are only slightly different from the original (local)

2. If condition
This was mainly used to check if a value exists and then proceed with some operation. The first question that might arise might be: "Why didn't you just use 'where' conditions with FHIR Path?" and the answer would be: "These were situations where if one or more of multiple values exists, it should proceed only once, for instance with creating a resource. The 'where' conditions would force a foreach-loop on us that executes for every true condition in a chained condition." This is our current workaround for this very case:

// Check whether resource should be created
let resourceShouldBeCreated = create('Boolean');
operations then SetBooleanToFalse(operations, resourceShouldBeCreated);

operations.data as data, data.values as values where "blockindex = 0 and groupindex = 0 and itemid = 'id_ba_siteid'
                                    or blockindex = 2 and groupindex = 0 and itemid = 'id_2435'
                                    or blockindex = 2 and groupindex = 0 and itemid = 'id_2458'" then
{
    operations then SetBooleanToTrue(operations, resourceShouldBeCreated);
};

operations where "%resourceShouldBeCreated.valueBoolean = true" -> ...

With an if-condition that executes a single time and evaluates if 'at least one of the conditions is true', this would be way more readable and maintainable and there would be no need for the let statement.

3. For loop with index
In real-world applications, you often use an index within data. Especially if the data has no hierarchy. The FHIR Mapping Language currently does not support something like this. Again, we used our let-statement as a workaround:

let index = create('RepeatIndex');
src then InitIndex(src, index);
src.fields as fields, fields where "id = 'field-' +  %index.Index.toString()"
{
    src then IncrementCaseIndex(src, index);
};

This works, however a proper for loop with a built-in index would be preferable for improved readability and less overhead with the let statement and inclusion of FHIR Path.

What do you think about these issues? Have you encountered something similar? @Oliver Egger Did you run into any limitations yet during CDA mappings?

Grahame Grieve (Dec 07 2020 at 09:55):

@Keith Duddy interested in your comments

Grahame Grieve (Dec 07 2020 at 09:55):

@Yannick Börner the fact that there is no sequence in the language is key to a number of implementation features, including the ability to run the transform at the meta level

Gustav Vella (Dec 08 2020 at 07:47):

Grahame Grieve said:

Yannick Börner the fact that there is no sequence in the language is key to a number of implementation features, including the ability to run the transform at the meta level

Explaining the presence of something with the absence of something else is nearly spiritual. And Yes, It’s easier to say what declarative programming is not than to say what it is. But does that really help? So are you saying that we are abusing the language by doing the things we are doing with it? do you see such cases covered with the approach taken with XSLT extensions? More generally: Where do you see this thing going? What's the plan?

Keith Duddy (Dec 08 2020 at 08:11):

Thanks for calling me out Grahame - I get emails when @ named, as I'd been out of touch with the Mapping Language's progress for some time.

For context, I should first say that Grahame's ambition for the language was to allow for mapping between FHIR & FHIR, but also to allow for any tree structured inputs and outputs to be possible, as long as the engine could deduce the correct structural match between a FHIR resource and, say, an XML document tree, or HL7 CDA document tree. Having come from the UML/MOF world, with my experience of the QVT language(s), I had limited myself to thinking only about transformations between metamodels expressed in the same (meta)language (effectively Eclipse Modelling Framework -EMF, which implements EMOF). But if source and target models are all structured as trees, then I could find no reason why an engine implementation couldn't create them with assumed equivalences between the element containment for the tree structure in any input or output format. I know one of Grahame's early prototypes allowed FHIR to XML mappings (and vice versa?).

@Yannick Börner , I can see the usefulness of having "let" variable declarations. In the equivalent concept to "group" in the QVT language, e.g.

group tutorial(source src : TLeft, target tgt : TRight)

QVT has a "relation", and it allows for any number of variables to be declared, which would include both the source and target (or sometimes for building intermediate structure, not even any target, just a built up runtime relation that could be looked up in another rule). This example includes source, 2 variables and target:

(this relation assumes that an "empty" T is already created for every S in a AnSforEachT relation/rule)

relation PopLowerAndUpperTfromS(src : Tleft, var1 : int, var2 : int, tgt: TRight) {

forall S:SomeType in src, T:TgtType in tgt
where AnSforEachT(S, T) and
S .interestingValue < 10 and S.interestingValue > 4
make var1 = S .interestingValue and
var2 = interestingValue + 12 and
T.lower = var1 and // or T.lower = S.interstingValue
T.upper = var2
}
[sorry for the poor psuedo-syntax for QVT - it's been a while]

Then upon population of T.lower and T.upper, the engine would store a trace relation for each unique matching S:SomeType in src:

[I use @ to mean "pointer-to"]

(@SomeS, 6, 18, @Sometgt)
(@AnotherS, 4, 16, @Anothertgt)

etc... so that after the transformation runs, you can see which S in src matched, and what values were transferred to the target, without having to follow the pointers in the S and T to see the results.

As you can see, in the rule body, vaibles are really syntactic sugar, as I could have just spelled out the expressions assigned to var1 and var2 in each part of the rule expression - the main point was to store derived values calculated during execution in the traces. Then other rules/relations could make use of them later, by binding them to declared variables: [Here "_" means we don't care what the value in a stored relation trace is.]

relation HighRangeKfromS(src : Tleft, tgt: TRight) {
let upper: int;
forall S:SomeType in src, K:RelatedType in tgt
where SrelatedToK(S, K) and
PopLowerAndUpperTfromS(S, _, upper, _) and
upper > 15
make K.isHighRange = true
}

We allowed declaration of as many variables as you wanted inside relations, with "let" declarations (like "upper" above). These also had to be used to check the index of a list, for example to make sure you didn't map the same element in a list twice. As there was no iteration, this was probably a bit clumsy, and had to rely on the index being embedded in a trace relation, but serves a similar purpose to your use of "let" declared variables.

I can't recall any global variables - which may have been useful for constants, but because the engine determined the order of execution of rules, based on dependency analysis, I don't think we could have depended on the state of anything global if there were multiple possible rule execution orders implied by the structure. e.g. if "SomeS" was matched first, then setting a global variable based on the values of SomeS would yield a different outcome than if "AnotherS" was matched first.

I don't know if that's helpful, as I'm afraid I don't know what the current engine implementations do, and I had a quick look at the latest state of the Tutorial, but Group if left TBD...

cheers,
...|<

Yannick Börner (Dec 08 2020 at 09:59):

Grahame Grieve said:

Yannick Börner the fact that there is no sequence in the language is key to a number of implementation features, including the ability to run the transform at the meta level

I see the point here but I fear this restriction could hinder wider adoption for the use cases I described above. Especially when dealing with larger datasets sometimes you simply have to be aware of your current position within the data. Could you elaborate on the transformation at the meta level? I have no experience regarding this and would like to understand the use case.

Yannick Börner (Dec 08 2020 at 13:13):

@Keith Duddy That is an interesting take on the concepts I mentioned from the view of QVT. Global let statements would require the engine to employ delayed execution to make sure one already has an instance of the constant available on your syntax tree for example. There are quite a lot of strings attached to this, so let's throw this idea overboard for now.

As far as I can tell, we both agree that local let statements are indeed helpful and you employed them as well in the QVT language. @Grahame Grieve Do you see any huge red flags that would prevent the local let statement from being added to the standard?

Gustav Vella (Dec 11 2020 at 09:15):

Yannick Börner said:

As far as I can tell, we both agree that local let statements are indeed helpful and you employed them as well in the QVT language. Grahame Grieve Do you see any huge red flags that would prevent the local let statement from being added to the standard?

@Yannick Börner since there seems to be some consensus here on the local let statement the best way to go is to submit a request: https://jira.hl7.org

Oliver Egger (Dec 12 2020 at 10:37):

there is already a ticket for that and was voted on and set to resolved - change required ... see https://jira.hl7.org/browse/FHIR-21642

Alexander Zautke (Dec 15 2020 at 09:30):

@Oliver Egger The resolution of that ticket states that the let statement can only be used on top of the mapping file, meaning that we couldn't have local let statements.

So what I'm reading from the messages above is that instead of introducing local let statements, they should rather be (primitive) targets of a group, correct? @Yannick Börner What that work for you, too?

Gustav Vella (Dec 15 2020 at 19:45):

Hi There, @Oliver Egger thanks. I was unsure whether I understood correctly but @Alexander Zautke confirmed the scope of that ticket. Also thanks to @Keith Duddy for that very insightful feedback. We at Healex were considering having a call to discuss the problematic cases / limitations encountered - @Yannick Börner only shared one issue - and the specific suggestions proposed here coming Monday/Tuesday, December 21st/22nd 2020. That would be Mo 21:00h (Berlin), Mo 15:00h (Boston) and Tue 07:00h (Sydney). Would there be interest to join that call? If so just a thumbs up and I'll share the Teams invitation. The choice of timeslot is the only inclusive slot suitable a 3 continent call (leaving some chance for @Grahame Grieve joining)

Oliver Egger (Dec 15 2020 at 21:03):

Alexander Zautke said:

The resolution of that ticket states that the let statement can only be used on top of the mapping file, meaning that we couldn't have local let statements.

I agree, sorry, i didn't note the subtle change in resolution ...

Yannick Börner (Dec 16 2020 at 05:47):

@Gustav Vella Sounds good!

Yannick Börner (Dec 16 2020 at 05:56):

@Alexander Zautke With regards to the difficulties as discussed by Keith, I think local let statements are the way to go at first. They are easier to implement and introduce next to no issues. However, a global let statement would be great as well as it would allow for constants and shared variables between groups. It would need to be thought through meticulously though as previously mentioned.

Alexander Zautke (Dec 17 2020 at 13:20):

Not answer to the question that I asked ;) We have now a ticket that has been voted on with a clear resolution. Before we try to add new features, I would like to evaluate the current limitations. What would not be possible if we use the empty target variables instead of local let statements?

Alexander Zautke (Dec 17 2020 at 13:20):

Regardless of how easy it seems to implement it...

Yannick Börner (Dec 17 2020 at 15:01):

Ha, you're right @Alexander Zautke! I do think the functionality could be substituted with target variables. However, for nested group calls, it'll become a parameter nightmare. You would have to pass a parameter for each target variable through the entire execution chain. Though it should work, it will make the maps extremely bloated.

A target with a custom structure definition holding multiple variables at once could be feasible in this case. But that is not really a clean solution either.

Gustav Vella (Dec 20 2020 at 19:06):

Gustav Vella said:

We at Healex were considering having a call to discuss the problematic cases / limitations encountered - Yannick Börner only shared one issue - and the specific suggestions proposed here coming Monday/Tuesday, December 21st/22nd 2020. That would be Mo 21:00h (Berlin), Mo 15:00h (Boston) and Tue 07:00h (Sydney). Would there be interest to join that call? If so just a thumbs up and I'll share the Teams invitation. The choice of timeslot is the only inclusive slot suitable a 3 continent call (leaving some chance for Grahame Grieve joining)

As promised, here's the link for tomorrow's call https://teams.microsoft.com/l/meetup-join/19%3ameeting_ZDY5NmE0YzctZDM5My00OWYxLTlhNjctY2IyMDUzOThjM2Uy%40thread.v2/0?context=%7b%22Tid%22%3a%22f09354e8-9d00-42a1-98c8-58639e2a8f91%22%2c%22Oid%22%3a%227a865118-743f-4058-ab6a-0b9cae52d351%22%7d

Anyone wishing to can join.

Last updated: Apr 12 2022 at 19:14 UTC