Stream: mapping-framework
Topic: StructureMap
Grahame Grieve (Feb 20 2019 at 09:05):
I am working on an implementation using StructureMap. While doing so, I've realised that there's a missing feature.
Grahame Grieve (Feb 20 2019 at 09:08):
so when you are writing a transformation, you are defining the relationship between the source and the destination structures. One (difficult but very useful) feature of the way the mapping language works is that there is no state - no variables that exist out of local context. All the rules define their dependency carefully, so that you know what inputs are required.
Grahame Grieve (Feb 20 2019 at 09:08):
this makes for both efficient transforms - Rules can be run in series in any order, or in parallel, as long as dependencies are satisfied. Also, it means that you can run the transforms, and given a known set of inputs, you can describe the possible set of outputs. That's pretty cool
Grahame Grieve (Feb 20 2019 at 09:09):
but the mapping language assumes, when it's doing that, that the destination is empty, and the content is being pushed into the destination without overwriting anything.
Grahame Grieve (Feb 20 2019 at 09:09):
actually, that's not quite true. We say:
Grahame Grieve (Feb 20 2019 at 09:13):
.. no, apparently we don't say it. But there's no stated rule that the starting target has to be empty. If it's not, it really doesn't make any difference; the target is populated from the source, and any mapped fields in target will be ... changed...
Grahame Grieve (Feb 20 2019 at 09:14):
it's obvious what to do if the cardinality of the target is ..1: you overwrite the existing value if you have one. (aside: is there any reason to want to say 'clear this element if there's nothing in the source'?)
Grahame Grieve (Feb 20 2019 at 09:15):
it's not so obvious if the cardinality is a ..* - you can just say that the mapping automatically adds new objects to the list... but that's actually not what you want
Grahame Grieve (Feb 20 2019 at 09:17):
take the case of transforming from a PID segment to a Patient resource, and processing the XTN entries in PID-17 (I think it is)
Grahame Grieve (Feb 20 2019 at 09:18):
you have a list of phone numbers and fax and email etc. It's ok to talk about the first time, when you transform from pid to Patient, and stick it in your database, no problems
Grahame Grieve (Feb 20 2019 at 09:18):
but what happens when you get an update?
Grahame Grieve (Feb 20 2019 at 09:19):
You can still talk about the mapping, but somewhere, you need to say criteria are so that home phone number replaces the existing home phone number, instead of just adding a new one. (assuming there's only one...)
Grahame Grieve (Feb 20 2019 at 09:21):
so it seems to me that you need some kind of match expression that specifies which target in a list the source transform into
Grahame Grieve (Feb 20 2019 at 09:21):
something like
Grahame Grieve (Feb 20 2019 at 09:23):
pid-17 where type= home and system = phone matching target.type=home and target.system= phone -> ContactPoint then { rules }
Grahame Grieve (Feb 20 2019 at 09:23):
(well, something like that - the v2 details are not quite right, but it serves to show the principle)
Grahame Grieve (Feb 20 2019 at 09:24):
so that's what I think the missing feature is: a rule to specify what matches
Grahame Grieve (Feb 20 2019 at 09:25):
but I ducked an issue: do you want to delete anything from the target list? (kind of like, do you want to delete the entry for a ..1 element?
Grahame Grieve (Feb 20 2019 at 09:26):
I don't think that the matching thing solves this....
Grahame Grieve (Feb 20 2019 at 09:26):
this has an interesting consequence, btw, because it also applies when you start from empty
Grahame Grieve (Feb 20 2019 at 09:27):
and what it means is the outcome, if you construct the rules the right (wrong) way, is the outcome depends on the order in which the rules are executed. It may also, for the same reason, prove too hard to evaluate speculatively.
Grahame Grieve (Feb 20 2019 at 09:28):
An alternative to all this is to invent some new language that specifies the 'update rules' when committing an update to a resource.
Grahame Grieve (Feb 20 2019 at 09:29):
of course, on the RESTful api we have patch, but this is different - subtly, but importantly - from a transform rules
Grahame Grieve (Feb 20 2019 at 09:30):
so having got this far, and maybe talked myself out of my original idea.... has anyone seen any language that deals with update semantics?
Grahame Grieve (Feb 20 2019 at 09:31):
@Keith Duddy @Josh Mandel I'd appreciate your opinions... thanks
Paul Church (Feb 20 2019 at 19:52):
We generally think of mapping on streaming data as a problem of f(new data, existing resources) -> updated resources, as compared to the static mapping problem of f(data) -> resources. This requires a prefetch step to figure out what existing resource(s) are relevant and retrieve them if they exist (else the mapping results in a create operation). Unfortunately this gives up a lot of the nice stateless properties, but trying to do f(new data) -> "description of mutations" pushes a lot of complexity into how you encode update semantics and how the server applies them.
Inside the mapping, any ..* field needs merge logic describing what field(s) are the primary key on which the old and new lists should be joined, which your matching rule handles. A deletion could look like type=X, system=Y, and all other non-primary-key fields null; when merging the lists any existing entry with type=X, system=Y should be dropped - although this carries a bunch of limiting assumptions, it hits some common use cases.
Grahame Grieve (Feb 20 2019 at 20:16):
I think you're agreeing with where I ended up - the merging rules should be separate from the mapping rules
Paul Church (Feb 21 2019 at 00:46):
Yes and no - the merging rules either end up in the server as part of a patch-like operation that knows how to apply deltas, or they end up in the mapping framework (at least, attached to the mapping framework as a second step) in a way that prevents it from being stateless. It sounds like you're leaning towards the former as preferable?
Grahame Grieve (Feb 21 2019 at 00:51):
well, they are dependent on the business rules of a particular input - how it manages ids etc. So they can't be in the server as a context-less operation.
Grahame Grieve (Feb 21 2019 at 00:51):
and the rules have to be expressed somewhere. So they are paired with the mapping, since that's also tied to the context
Grahame Grieve (Feb 21 2019 at 00:51):
I lean to the second option then, though I do not know what 'attached' means in that sense
Keith Duddy (Feb 21 2019 at 02:19):
Hi @Grahame Grieve and @Paul Church ,
Paul, I think you nicely summarised what my first response would be to the question above (i.e. update is f(old-state, new-resources) -> new-state. I have always worked with functional mappings, and so each model in or out of a transformation is immutable. That sounds very restrictive, but any two models can be related by a versioning link that says "this is my [immutable] predecessor", which automatically preserves the history of the updates made to any resource/model. Then a diff can be made on the two versions to see what changed (which is also invaluable when debugging complex mapping rules), or finer-grained traces can be stored, to avoid structural diffing - see below for why this might be useful.
That said, there is a lot of research out there on bi-directional transformations and in-place updates which provide frameworks for thinking about your questions [Proceedings of the International Conference on Model Transformation - ICMT has dozens of articles over the years on these topics]. The answers are almost always not as simple as you'd like - i.e. cardinality of * may or may not have anything to do with the semantics of an update - this would need to be specified without relying on that to provide any assumptions. If 1 element is in a multi-valued slot, the * cardinality says nothing about whether it should be added to or replaced. That is in the semantics of the update, and would be explicitly specified in the mapping. (Conversely, a cardinality of 1 with a mapping rule that tries to add an additional element should raise an exception, rather than just be assumed to be a replacement for previous single element.)
Another issue to think about is that a container with a simple value in it may change, based on new inputs, and a simple structural comparison may make it seem that no update was made, e.g. in the case:
{x:112[n:1, name:"smith], x:113[n:3, name:"jones]} -> forall x.n set n=1 -> {x:112[n:1, name:"smith"], x:113[n:1, name:"jones]}
The value of n in x:113 has clearly been updated from 3 to 1, but n in x:112 _has also_ been updated, from 1 to 1, and this should preferably be recorded in the transformation "diff"(or trace) rather than relying on simple structural comparisons. Implementations I have been involved in will keep a trace between each pair of changed values in 2 immutable versions of a model, regardless of whether the change resulted in a simple value update or not.
Of course, the value of this trace information depends greatly on what the semantics of x and n are and why we want to know that they've been transformed, and when (even if it's incidentally an identity transformation). But in something as generic and wide ranging as a FHIR resource model, there will almost always be some use case for this.
Keith Duddy (Feb 21 2019 at 02:48):
[Aside to the in-place update approaches: my experience, at least in the group that worked on QVT at OMG was that the in-place-update people were mostly Telecoms or SCADA people who had to push tiny new bits of state into devices with limited expensive persistent memory, which could not possibly store all previous states, and would sometimes change state at mircosecond scale intervals... perhaps this will also be true of some medical equipment that will push out small resources using FHIR (or consume sensor data structures, or whatever). However, my impression in the main is that a FHIR resource will mostly be supported by a modern n-tier server, with the processing and storage capacity to keep all relevant [immutable copies of] state, and links between them at various granularities.]
Last updated: Apr 12 2022 at 19:14 UTC
