Stream: implementers
Topic: Transaction with history
Sean McIlvenna (Jan 31 2019 at 01:26):
Is there a way to indicate the history of resources in a transaction? For example, can I have a transaction that has three versions of the same resource? And if so, is the server supposed to respect the meta properties, such as the .meta.lastUpdated and .meta.versionId properties?
Lloyd McKenzie (Jan 31 2019 at 01:27):
You can use a transaction to update something 3 times, but the meta will reflect what happened on the server the data's being created on, not where it came from.
Sean McIlvenna (Jan 31 2019 at 01:28):
so there's no way to get the three different versions to have the correct "lastUpdated" property?
Sean McIlvenna (Jan 31 2019 at 01:28):
or to ensure that the three different versions are processed in the correct order?
Sean McIlvenna (Jan 31 2019 at 01:29):
Seems to me that this should be possible so that systems could backup a FHIR server using a Bundle
Sean McIlvenna (Jan 31 2019 at 01:29):
backup AND restore
Lloyd McKenzie (Jan 31 2019 at 01:29):
I think entries are processed in order unless there's a reason they can't be. But certainly there's no way to set lastUpdated to a historical value.
Lloyd McKenzie (Jan 31 2019 at 01:30):
If we want that, it'd need to be a distinct operation.
Sean McIlvenna (Jan 31 2019 at 01:30):
ok... so you think I should submit a change request to create a "$restore" operation?
Lloyd McKenzie (Jan 31 2019 at 01:30):
Sure, why not :)
Sean McIlvenna (Jan 31 2019 at 01:30):
that $restore operation could be either a Bundle transaction OR a specific resource, I suppose
Sean McIlvenna (Jan 31 2019 at 01:31):
/$restore
AND
/Composition/1/$restore
Sean McIlvenna (Jan 31 2019 at 01:41):
https://gforge.hl7.org/gf/project/fhir/tracker/?action=TrackerItemEdit&tracker_item_id=20332
Grahame Grieve (Jan 31 2019 at 03:56):
would this wipe the internal history? why would you do this across the API? It just seems like an internal thing to me
John Moehrke (Jan 31 2019 at 15:06):
This certainly brings up concerns with integrity of the history, if history can e re-written without a trace. Clearly this operation would need to be privileged, and it should clearly have strong authorization and audit logging behavioral requirements. behavioral requirements that can be fulfilled by a security layer such as SMART-on-FHIR, or other. behavioral logging requirements that could utilize AuditEvent or other.
Would there be a Provenance impact? Seems these historic versions of a resource could have original Provenance restored as well; but one might ask if the $restore operation should create one new Provenance indicating the $restore activity and pointing (.target) of the version specific resources impacted (restored).
Sean McIlvenna (Jan 31 2019 at 15:14):
@Grahame Grieve , because as far as I can tell, none of the existing FHIR servers implement a form of backup/restore short of what's allowed by the supporting database. If this existed in the spec, it might be more likely to be supported by FHIR servers. Standardizing a more-thorough export/import format/process for a FHIR server doesn't seem like a bad thing to me.
Grahame Grieve (Jan 31 2019 at 18:36):
Standardizing a more-thorough export/import format/process for a FHIR server doesn't seem like a bad thing to me.
But overwriting history seems like a step further than that
Sean McIlvenna (Jan 31 2019 at 21:14):
then let's adjust the $restore operation so that it only ADDS to the history
Sean McIlvenna (Jan 31 2019 at 21:14):
so, if a resource already has history, the existing history is preserved, and added to
Sean McIlvenna (Jan 31 2019 at 21:15):
maybe...
Sean McIlvenna (Jan 31 2019 at 21:15):
dunno, might have to think about that more
Sean McIlvenna (Jan 31 2019 at 21:15):
nah, I don't like that... I take it back
Sean McIlvenna (Jan 31 2019 at 21:15):
let's suggest that the $restore operation be secured with role-based authorization
Sean McIlvenna (Jan 31 2019 at 21:16):
and call it good :)
Sean McIlvenna (Jan 31 2019 at 21:16):
then, if the server implementer wants to allowing anonymous users to overwrite history, that's up to them
Paul Church (Jan 31 2019 at 21:31):
We implemented this in the Google Cloud FHIR server - our (non-spec) bulk import operation will accept history bundles and synthesize the corresponding history. It's quite useful for integration testing and a few niche use cases but creates complex issues for anything other than populating an empty store to a known state.
If the resource already exists, the imported history will get interleaved with the existing history - and that could easily represent an inconsistent sequence of mutations. Any client holding on to an etag may get confused, because the history can be mutated without touching the current version. Doing a backup/restore of data that includes versioned references is even more difficult. We have audit logs outside of the FHIR store itself so we can potentially maintain the integrity of the audit trail but it's always going to look more confusing than "normal" operations.
It's a nice implementation-specific feature but is there actually a use case where users want interoperability between servers of backup/restore including resource history?
Grahame Grieve (Jan 31 2019 at 21:32):
wow that's head hurting territory
Paul Church (Jan 31 2019 at 21:42):
It's just one of those early ideas that needs some iteration. If the operation couldn't touch resources that already exist or existed in the history, it would be a lot better behaved.
Grahame Grieve (Jan 31 2019 at 21:49):
or if you just reset the repository completely
Sean McIlvenna (Jan 31 2019 at 21:51):
actually, that's kinda the scenario I want to accommodate
Sean McIlvenna (Jan 31 2019 at 21:52):
completely wiping out the repository
Sean McIlvenna (Jan 31 2019 at 21:52):
and getting everything that was in the repository that was there before the wipe
Elliot Silver (Jan 31 2019 at 21:52):
If you are doing incremental backups, I would expect you need to do incremental restores.
Grahame Grieve (Jan 31 2019 at 21:53):
I know - that's what we need - a healthcare specific standard for doing incremental backup restores across a RESTful API
Elliot Silver (Jan 31 2019 at 21:54):
OK, so is there a need to do any of this in an interoperable way?
Paul Church (Jan 31 2019 at 22:33):
I think there is a case for interoperable import including history - bulk data $export is trying to give the client everything the server knows about some patients. But what if the application logic on the receiving end uses the resource history? How do you capture and transfer that? It's not a big stretch to say it's an ndjson file containing history bundles. (Alternately, the answer could be "bad application, don't do that".)
Grahame Grieve (Jan 31 2019 at 22:36):
or the answer could be... that's ok, but we don't believe it deserves formal standardization
John Moehrke (Feb 01 2019 at 14:04):
Having the ability so that one can fill a demo server, seems like a function of a reference implementation that has a feature of being easy to use as a demo server..... Even as a reference server this capability should be OFF by default, and it should take extraordanary effort by the sysadmin to enable it.... I don't think it is strange to have features like this... but they are system features, they don't need to be defined in a standard.
John Moehrke (Feb 01 2019 at 14:08):
Where ever it is defined, it should be defined as a highly privileged interface, and the use of that interface should be highly logged. Thus if it 'accidentally' got enabled on a medical-records dataset, it could be detected as to being enabled and all uses of the interface determined so they could be rolled back. This doesn't help the patient-safety risks that may have happened during the time that bad data was in the system, but it does make it detectable.
Sean McIlvenna (Feb 01 2019 at 16:12):
By concluding that, you're concluding that no FHIR server will ever want to (in a standardized way) transfer data with history between FHIR servers.
Lloyd McKenzie (Feb 01 2019 at 16:14):
The issue is that "history" means two things:
1. What happened to this business record over time
2. What did the data on this particular server look like at a particular point in the past?
FHIR's _history is only trying to represent #2. #1 is handled with Provenance
Lloyd McKenzie (Feb 01 2019 at 16:15):
Reloading history thus only makes sense when you're doing a restore or when you're simulating a single server with multiple boxes
Last updated: Apr 12 2022 at 19:14 UTC
