Stream: implementers
Topic: Resource updates and inconsistent references
Brian Reinhold (Oct 01 2019 at 14:15):
If one does an update of a resource, its version will change. However, there could be resources that already exist on the server that reference that resource. However, there is no way to indicate which version of the resource that reference is pointing to. Is one forced to compare times of last updates to resolve that? It seems painfully complex, perhaps requiring a significant number of downloads to find the correct version of the referenced resource. Would it make sense to replace updates with a differential like we do in Structure definitions, where the 'base' resource is the resource being updated?
This might be useful for Point of Care devices where the Device can change its status and a new set of Observations get generated based upon the new status. The Observations generated on the original status would point to the 'base' in this case, and the new ones to the Device with a differential flag and a reference to the base. At least it would be both efficient for readers and still take little space on the server. If the differential flag is set, NO element will be required except the reference to the base (which may be another differential). Maybe we require at least one element to be populated that is different from that in the base just to be sure 'empty' differentials are not uploaded.
Alexander Kiel (Oct 01 2019 at 15:37):
According to the spec literal references can be versioned:
References can be version specific - that is, a reference may point to a specific version of a resource. e.g.:
<target>
<reference value="http://example.org/fhir/Observation/1x2/_history/2" />
</target>
Brian Reinhold (Oct 01 2019 at 16:05):
According to the spec literal references can be versioned:
References can be version specific - that is, a reference may point to a specific version of a resource. e.g.:
<target>
<reference value="http://example.org/fhir/Observation/1x2/_history/2" />
</target>
Good to know. Question: I understand that not all servers support versioning in the sense of being able to access older versions of a resource. Is this correct? Perhaps such servers do not do updates? If they do support versioning then I should be able to GET an older version using the same URL
Michele Mottini (Oct 01 2019 at 16:46):
I understand that not all servers support versioning in the sense of being able to access older versions of a resource. Is this correct?
Yes, correct
Perhaps such servers do not do updates?
They can do updates - they just modify the one and only version of the resource
Grahame Grieve (Oct 01 2019 at 20:45):
the new ones to the Device with a differential flag and a reference to the base. At least it would be both efficient for readers and still take little space on the server
I find this a very slippery idea. It might save space but the cost of differentials/deltas is very high risk - as anyone who's ever tried to make them work has experienced.
Grahame Grieve (Oct 01 2019 at 20:45):
generally, you could try doing what you're doing with versioned references but I think that's also a very difficult idea.
Grahame Grieve (Oct 01 2019 at 20:46):
How does it matter that the observation points to the right definition of the device?
Brian Reinhold (Oct 01 2019 at 23:34):
the new ones to the Device with a differential flag and a reference to the base. At least it would be both efficient for readers and still take little space on the server
I find this a very slippery idea. It might save space but the cost of differentials/deltas is very high risk - as anyone who's ever tried to make them work has experienced.
Never having implemented such I can't say. But I was thinking more along the lines that a 'differential' of an Observation would just be a new Observation resource with only changed elements populated (and something to point to the existing 'base' Observation resource). Clearly one would need to point to the base and there would have to be something to indicate that it was a 'differential'. This would be useful for devices that stream a sequence of measurements where the only thing that changes between the measurements is the valueQuantity.value and the effectiveDateTime, All other fields are the same. Some of the tests with the measurements being uploaded by some acute care devices systems overwhelmed the server. Shrinking the size of the Observation resource would help. But I can see many issues with that as well. Searches would certainly be more challenging.
Grahame Grieve (Oct 02 2019 at 00:07):
I think that the server should deal with that. Where you testing with one of the standard open servers?
Brian Reinhold (Oct 02 2019 at 09:07):
I think that the server should deal with that. Where you testing with one of the standard open servers?
From what I understand it was a HAPI FHIR server. This work was being done by those working on IEEE SDC (Service-oriented Device Connectivity). The volume of measurement data these devices can generate is astounding. As I understand it a lot of this stuff is passed around using V2 messaging but it is a little out of my realm! However, one of the goals of Devices On Fhir is to have a consistent representation of all types of medical device data in FHIR from PHD to PoCD to SDC.
Grahame Grieve (Oct 02 2019 at 09:45):
HAPI is fairly slow out of the box; it would be interesting to know if they could have optimised it
Brian Reinhold (Oct 02 2019 at 09:45):
According to the spec literal references can be versioned:
References can be version specific - that is, a reference may point to a specific version of a resource. e.g.:
@Alexander Kiel If I were to put, for example, a reference to a Device resource holding the info about the device that generated the Observation data as Observation.device.reference="Device/46/_history3" how would I search for this resource when there are 10 versions? The id of the resource would just be 46 so I could not just grab the value and search for it as an id; it would not exist. If the server supports vreads I suppose it would work... I assume I could otherwise search on both the id and the meta.version? But would it exist on the server? When I dump a set of Patient resources from a HAPI FHIR server I see the latest version of a given Patient but not any earlier ones. I also tried searching with the full history as in a VREAD but just got the latest version.
Grahame Grieve (Oct 02 2019 at 09:47):
because history overwrites. So there's only one current copy - that's how it's deined.
Brian Reinhold (Oct 02 2019 at 09:51):
@Grahame Grieve At the most recent connectathon we were hoping to get some insight into that in one of our HCD meetings. We talked a little with Rick Geimer about that. I was hoping to grab @James Agnew for the discussion but could not get a hold of him. I also never got a clear answer from the SDC folks about how they were uploading the data which could also make a difference.
Alexander Kiel (Oct 02 2019 at 12:17):
Alexander Kiel If I were to put, for example, a reference to a Device resource holding the info about the device that generated the Observation data as Observation.device.reference="Device/46/_history3" how would I search for this resource when there are 10 versions? The id of the resource would just be 46 so I could not just grab the value and search for it as an id; it would not exist. If the server supports vreads I suppose it would work... I assume I could otherwise search on both the id and the meta.version? But would it exist on the server? When I dump a set of Patient resources from a HAPI FHIR server I see the latest version of a given Patient but not any earlier ones. I also tried searching with the full history as in a VREAD but just got the latest version.
I'm not super familiar with FHIR Search but I assume that search only works over the most recent version of each resource. History is more for audit purposes. You can view the history of a resource using GET [base]/[type]/[id]/_history{?[parameters]&_format=[mime-type]} and you can filter it by _since and _at but I don't think, you can search it.
I'm here with @Grahame Grieve that you don't should use versioning for something like differential Observations. At the end FHIR is mostly a data exchange format and not primarily a storage format. Most generic servers will store resources directly as they are. But if you have time series data, a special storage format might be necessary. If you look at monitoring solutions like Prometheus they have quite advanced Storage Solutions for time series data.
Brian Reinhold (Oct 02 2019 at 12:35):
@Alexander Kiel I was not thinking of versions for 'differentials'. The use of versions is to solve a different issue. Consider a PoCD medical device that is sending observations. Then the device changes in some way, and an update of the Device resource is made. The new Observations will point to this updated Device, but so will the old Observations made pre-update unless version information is found and getting the Device resource as it was before the update can retrieved. Of course this can be solved by creating a new Device resource instead of an update. PoCD was trying to avoid creating a new resource for such dynamic changes.
The differential issue was to see if there was a way to more efficiently 'store' data. A sequence of Observations might only change in the value and time stamp, everything else is the same. If a vital sign its even worse as the requirement to follow the vital signs profile means there will be a repeated category and LOINC entry in every resource in addition to the MDC entry.
The IEEE specs do support Real TIme Sample arrays (RTSAs) for periodic time series which would be very efficient. But not all devices use that. If the streaming is relatively slow (say once per second), RTSAs tend NOT to be used. Of course the gateway could always buffer such measurements and create an RTSA for upload into the FHIR valueSampledData but doing so would require a gateway that is pre-programmed for a specific device's behavior. That is far from ideal. I do that for some devices but if a new device has similar behavior my gateway will handle it as individual measurements because it doesn't know that it streams at frequency X.
Grahame Grieve (Oct 02 2019 at 12:42):
search only works over the most recent version of each resource
yes
Grahame Grieve (Oct 02 2019 at 12:44):
it's still not clear to me how a change to the device changes the observation. It's still the same actual device...
Grahame Grieve (Oct 02 2019 at 12:44):
if there was a way to more efficiently 'store' data
Alexanders point is that FHIR has nothing to do with storing data. It's just the API. Servers can store data however they want
Brian Reinhold (Oct 02 2019 at 12:47):
it's still not clear to me how a change to the device changes the observation. It's still the same actual device...
@Grahame Grieve Its certainly not an issue in the PHD world where the device is static. But in the PoCD world there might be modes of operation that change the sensitivity or something like that which is not expressed in the measurement contexts. @John Rhoads ?
Grahame Grieve (Oct 02 2019 at 12:48):
I would think that if that changes the interpretation of the observation, it should be a data in the observation. You still haven't explained how that would be
Lloyd McKenzie (Oct 02 2019 at 14:11):
The data in the record would need to be fully expanded for search, inclusion in documents, etc. If you wanted to adopt a convention where transmission from the raw device to a particular server used the same data as was present on the previous reading from that device, you could use "implicitRules" to set that expectation. However, it means that the raw device measurements would have to be sent to a specialized service that recognized that particular implicitRule and would handle the expansion. It also means that if a measurement ever got lost in the sequence, you'd be in deep trouble.
Alexander Kiel (Oct 02 2019 at 14:55):
@Brian Reinhold I don't mean using SampledData for more compact storage. It's more like I could imagine a server which stores series of normal Observations with valueQuantity and effectiveDateTime in a more compact way internally.
Last updated: Apr 12 2022 at 19:14 UTC
