Stream: implementers
Topic: Search by _lastUpdated for deleted resources?
Tim Berezny (Mar 15 2019 at 13:22):
If I do a search for resources by a _lastUpdated date range, would i expect to get back any information about resources that were DELETED in that date range? If so, what kind of data would I expect to indicate it was deleted?
John Moehrke (Mar 15 2019 at 13:27):
First you need to understand that the effect of a DELETE operation is very policy dependent. Especially in databases that are supporting Treatment. Often times a need to stop-using is supported by setting the resource status element to entered-in-error. Sometimes a Delete does leave behind history, which might work as you are thinking; but it is possible a real-delete will also remove all history. There is a possibility that when the resource was deleted, that a Provenance record of that delete was recorded leaving only a Provenance resource behind.
Florian Apolloner (Mar 15 2019 at 13:37):
It probably depends on your requirements, but when one were to stick to the spec as it is written (http://hl7.org/fhir/http.html#delete) I'd say the resource shouldn't show up in search results (to quote: "... and that the resource is no longer found through search interactions."). But that is just me playing with FHIR since two days or so…
Tim Berezny (Mar 15 2019 at 13:39):
Interesting, ok. In the case I'm looking at, it would be for deleting a HealthcareService (a more significant delete then just setting as not active).
I have the choice to decide how it's implemented, it sounds like two options you mention are a history search or a provenance search. We don't CURRENTLY have a provenance resource set up for our records, but this may be a reason to add it.
Where would i look for how to implement with a _history search, I can't quite pinpoint how to identify with that method that the resource was DELETED.
Would publishing a HealthcareService via a subscrition make it clear if a resource was deleted?
Florian Apolloner (Mar 15 2019 at 13:45):
Where would i look for how to implement with a _history search, I can't quite pinpoint how to identify with that method that the resource was DELETED.
Ok, so _history works as follows (to the best of my understanding): Assume you have a resource with the following operations: POST, PUT, PUT, DELETE (so creation, two updates, and a delete). The history would look like this:
<entry> <resource> <Patient> <id value="23424"/> <!-- IMPORTANT: no extra data here (aside from metadata), since this record is deleted --> </Patient> </resource> <request> <!-- DELETE: this was a delete --> <method value="DELETE"/> </request> </entry> <!-- another <entry> here like the following for the second update --> <entry> <resource> <Patient> <id value="23424"/> <!-- other data submitted here for the update --> </Patient> </resource> <request> <!-- PUT: this was the first update --> <method value="PUT"/> </request> </entry> <entry> <resource> <Patient> <id value="23424"/> <!-- other data submitted here --> </Patient> </resource> <request> <!-- POST: this was a create --> <method value="POST"/> </request> </entry>
If you now where to limit the history to one entry it should have "DELETE" as method set. One other option would be a simple GET towards the resource which should return http status code 410 if the resource is deleted and the system supports that.
Florian Apolloner (Mar 15 2019 at 13:46):
Would publishing a HealthcareService via a subscrition make it clear if a resource was deleted?
This strongly depends on the type of the subscription. http://hl7.org/fhir/STU3/subscription.html cleary says "Note that the search criteria are applied to the new value of the resource. The consequence of this is that there is no notification when a resource is deleted, or when a resource is updated so that it no longer meets the criteria. ". As far as I understand this sentence, this would mean that you would only get a subscription notify if your search params would filter for just the resource type (!!) or only the id (or other values in metadata). Everything else like identifier etc is imo gone on a deleted resource.
Florian Apolloner (Mar 15 2019 at 14:00):
Thinking about it I was trying to remember why I did show the deleted resource with an id in the first history entry. I think this came to me as a result of http://hl7.org/fhir/STU3/http.html#vread which says that servers SHALL return an id there. So for "convenience" reasons I've kept the same scheme for the entries in _history
Grahame Grieve (Mar 15 2019 at 18:42):
a deleted resource is deleted. @John Moehrke is a misleading to say that the efect is policy dependent. If a resource is deleted, the server has to act like it's deleted, if it accepts the DELETE operation. OTOH, Whether a history is available depends on server capabilities (and then policy)
Grahame Grieve (Mar 15 2019 at 18:44):
once a resource is deleted, it's considered gone except for _history - so search and subscription don't help. Hence for this reason, you need _history for this one
John Moehrke (Mar 15 2019 at 18:46):
I don't disagree, just want to start from the fact level that delete is often verbally said in ways that are not technically true. A technical DELETE is a RESTful DELETE. In that case, it is gone... If you knew the _id value, then one can search on Provenance for Provenance.target that hold that _id value; and look for Provenance.activity=DELETE -- http://build.fhir.org/provenance.html#removal
John Moehrke (Mar 15 2019 at 18:47):
once a resource is deleted, it's considered gone except for _history - so search and subscription don't help. Hence for this reason, you need _history for this one
what does the last version in the history look like to indicate that the histoirical line is ended (DELETE) vs active?
Grahame Grieve (Mar 15 2019 at 18:57):
Bundle.entry.request.method = DELETE. Bundle.entry.response not present
Florian Apolloner (Mar 15 2019 at 19:02):
Bundle.entry.request.method = DELETE. Bundle.entry.response not present
That seems to be in contrast with what implementations do. For instance the demo server at http://hapi.fhir.org/ will return:
"fullUrl": "http://hapi.fhir.org/baseDstu3/Patient/1427169", "resource": { "resourceType": "Patient", "id": "1427169", "meta": { "versionId": "2", "lastUpdated": "2019-03-15T16:15:54.229+00:00" }, "text": { "status": "generated", "div": "<div xmlns=\"http://www.w3.org/1999/xhtml\"><table class=\"hapiPropertyTable\"><tbody></tbody></table></div>" } }, "request": { "method": "DELETE", "url": "http://hapi.fhir.org/baseDstu3/Patient/1427169/_history/2" }
Is this wrong? I am asking because I have implemented something similar which currently causes a few issues with conformance tests at touchstone.
Grahame Grieve (Mar 15 2019 at 19:06):
yes I think that's wrong. @James Agnew. The relevant piece of standard is http://hl7.org/fhir/http.html#history
Paul Church (Mar 15 2019 at 19:10):
Just to clarify, did you mean Bundle.entry.resource not present? In the given example, Bundle.entry.response is actually empty.
Florian Apolloner (Mar 15 2019 at 19:10):
Mhm, then in my opinion (not that it is worth much, I literally started with FHIR this week), it would be great if the standard could mention that. I personally implemented it like this because http://hl7.org/fhir/http.html#vread returns resources even for deleted elements (at least as far as I understand it: "The returned resource SHALL have an id element with a value that is the [id], and a meta.versionId element with a value of [vid].").
While _history is not exactly the same as vread it seemed to make sense consistency wise.
Florian Apolloner (Mar 15 2019 at 19:11):
@Paul Church Yes, in my implementation and examples I did put them as empty because that is how I understood the spec.
Grahame Grieve (Mar 15 2019 at 19:14):
@Paul Church yes I did mean Bundle.entry.resource is not present. typo - thanks for picking it up
Grahame Grieve (Mar 15 2019 at 19:15):
there is no resource to return for a vread on a deleted resource. What does test.fhir.org return on that case?
Paul Church (Mar 15 2019 at 19:19):
Google Cloud does the same thing that HAPI is doing here (resource populated, response empty), primarily because inside the implementation we want the deleted stub to have a resource entry so we can filter the history uniformly on meta.lastUpdated and with any access control looking at meta.security. But that doesn't prevent us from rewriting the history bundle if necessary to conform.
Paul Church (Mar 15 2019 at 19:21):
I notice that http://hl7.org/fhir/STU3/http.html#history does not have the history example for a deleted resource, and that was the spec we were initially working from.
Florian Apolloner (Mar 15 2019 at 19:23):
there is no resource to return for a vread on a deleted resource. What does test.fhir.org return on that case?
http://test.fhir.org/r3/Account/1/_history?_format=json
it returns no resource on deleted entries. That said, it records double deletes -- that is interesting :D
Grahame Grieve (Mar 15 2019 at 19:44):
that's a bug. as is this: http://test.fhir.org/r3/Account/1/_history/2
Florian Apolloner (Mar 15 2019 at 19:45):
Oh and it also errors out with a server error on a vread for a deleted vid:
http://test.fhir.org/r3/Account/1/_history/3?_format=json
-- from the looks of it it ries to parse the none existant resource :)
{"resourceType" : "OperationOutcome","text" : {"status" : "generated","div" : "<div xmlns=\"http://www.w3.org/1999/xhtml\"><p>Error parsing JSON source: Unexpected content at start of JSON: Eof at Line 0 (path=[])</p></div>"},"issue" : [{"severity" : "error","details" : {"text" : "Error parsing JSON source: Unexpected content at start of JSON: Eof at Line 0 (path=[])"},"diagnostics" : "(000000000047358B){FHIRServer.exe} [000000000087458B] Unknown function at TMethodImplementationIntercept + $3CF9BB\r\n(00000000040B0592){FHIRServer.exe} [00000000044B1592] Unknown function at TMethodImplementationIntercept + $400C9C2\r\n(00000000040B2440){FHIRServer.exe} [00000000044B3440] Unknown function at TMethodImplementationIntercept + $400E870\r\n(0000000000049157){FHIRServer.exe} [000000000044A157] Unknown function at __dbk_fcall_wrapper + $2C0F7\r\n(0000000000049A5B){FHIRServer.exe} [000000000044AA5B] Unknown function at __dbk_fcall_wrapper + $2C9FB\r\n(0000000000011576){FHIRServer.exe} [0000000000412576]\r\n(00000000000115C1){FHIRServer.exe} [00000000004125C1]\r\n(000000000047358B){FHIRServer.exe} [000000000087458B] Unknown function at ){KERNEL32.DLL} [00007FFB68DF84D4] BaseThreadInitThunk + $14\r\n(000000000006D851){ntdll.dll } [00007FFB6B56E851] RtlUserThreadStart + $21\r\n"}]}
Tim Berezny (Mar 18 2019 at 23:11):
The HAPI method seems useful to me, but it sounds like... it's implemented incorrectly? If I read this thread correctly, it looks like deleted resources should be treated like they were never there (with the exception of _history), and then that doing a search for Provenence resources, looking for deleted HealthcareServices would be the truest way to identify that a Healthcare Service has been deleted?
Grahame Grieve (Mar 18 2019 at 23:20):
yes to the first part
Grahame Grieve (Mar 18 2019 at 23:20):
there's no true way to identify a healthcare server that has been deleted except by monitoring the history
Brian Postlethwaite (Mar 19 2019 at 00:53):
More typically they will be marked as inactive...
Florian Apolloner (Mar 19 2019 at 07:25):
there's no true way to identify a healthcare server that has been deleted except by monitoring the history
I've seearched the #subscriptions stream but couldn't really find a reasoning for this. Any hints on why it is like this?
Grahame Grieve (Mar 19 2019 at 08:00):
because the subscription works on search, and search doesn't work on deleted resources.
Grahame Grieve (Mar 19 2019 at 08:01):
Florian Apolloner (Mar 19 2019 at 08:23):
Ok, it seems my english is not good enough to bring the point over, so I will try again. I do understand that subscriptions are based on searches and as such cannot show anything which no longer matches after an update/delete. What I was wondering is mainly why this was implemented/written like this. I'd imagine that systems would like to know about deletes hence this seems like an oversight to me prompting my question about the reasoning behind it.
Also technically speaking a deleted patient is still a patient, as such couldn't a subscription for resource_type = Patient still list it? Or is the assumption really that a deletion removes everything (which is obviously not the case since the history for that resource and id is still there).
Sorry if my questions appear to be annoying, I am trying to grasp the concepts and ideas to give me a better understanding.
Grahame Grieve (Mar 19 2019 at 10:04):
Josh proposed it that way for simplicity when he first proposed. The problem with deleted resources was the first thing I said to him about it, but any other approach is significantly more complicated, so we stuck with that proposal. We've discussed a more complex way that gives way more control, but feedback from implementers hasn't been enough to push it over the line
Florian Apolloner (Mar 19 2019 at 11:46):
Ah okay, thanks!
James Agnew (Mar 25 2019 at 09:22):
FYI- The way HAPI shows that empty patient in a DELETE entry when browsing history was an artifact of the way history works internally in the library. It's a bug that's existed almost since the first version of HAPI somehow.
JUst committed a fix: http://hapi.fhir.org/baseDstu3/Patient/1694175/_history
Last updated: Apr 12 2022 at 19:14 UTC
