Stream: implementers
Topic: GDPR and hard delete
Jack Liu (May 10 2018 at 01:21):
Hi there,
Apologies if this topic has already been discussed but I couldn't find any information regarding this issue.
The Art. 17 of GDPR requires the service to allow user to delete their data. The current delete operation in FHIR Rest API is a soft delete so it seems like we will need to support hard delete of the resource and its history in order to be GDPR compliant. I didn't see any special rule only applicable to healthcare data so I am assuming hard delete will be a requirement just like for any other personal data.
Is there any discussion about this already and what is the guidance on supporting GDPR requirements?
Thanks
René Spronk (May 10 2018 at 07:36):
See the security/privacy stream: https://chat.fhir.org/#narrow/stream/111-Security-and.20Privacy/topic/GPDR
Lloyd McKenzie (May 10 2018 at 09:49):
The RESTful delete is allowed to be a soft delete, but servers are allowed to execute a hard delete if they choose.
John Moehrke (May 10 2018 at 11:53):
As Lloyd points out, FHIR has defined a DELETE operation; it is up to systems to look to policy on what DELETE means. Thus in a GDPR environment, DELETE could be a hard delete, in a different environment DELETE could just set a security tag that prevents most from seeing it.
John Moehrke (May 10 2018 at 11:56):
There is an identified gap that it would be nice to have a deleteMe operation that would not just delete the obvious FHIR Resources, but would also forward the request to deleteMe to other organizations that are known to have received information about that data. This could leverage the AuditEvent for export events. This is just a thought at this point, no one has started that project.
John Moehrke (May 10 2018 at 11:56):
Gaps like this will be the thing we discuss starting at the GDPR track this saturday at FHIR Connectathon. http://wiki.hl7.org/index.php?title=201805_GDPR
Jose Costa Teixeira (May 10 2018 at 13:12):
@Jack Liu keep in mind that from a GDPR perspective, the patient's right to be forgotten is not absolute. There will be some logic to determine which/whether data can be deleted.
Jose Costa Teixeira (May 10 2018 at 13:17):
@John Moehrke I don't know much about security and authentication which seems to be a large chunk of the meeting you are proposing. I can contribute with practicalities of GDPR implementation, but not sure if that is what you want. Saturday is a hard day for me to attend but depending on the time, I can bring some info.
Jack Liu (May 11 2018 at 19:16):
Thanks all. The information is very useful.
René Spronk (May 12 2018 at 08:13):
I envisage we'll need an operation, e.g. $gdpr-delete to deal with a) the logic of whether one should do a hard delete or not (given local [EU country] legal requirements as to persistence of clinical data) and b) having to notify all third parties that have received a copy of your data that they should also erase the data, based on the AuditEvent record. It's a cascading process: if a citizen requests system A to delete item D, and A is aware the data has been sent/queried by application B, then A should call the $gdpr-delete operation on B. If B is aware it has made the information available to C, then B has to use $gdpr-delete on application C.
We'll need to distinguish between a DELETE and a gdpr-delete, one can't and shouldn't assume that all DELETEs will be gdpr erasure requests.
Grahame Grieve (May 12 2018 at 08:15):
also need a consent profile, please
Grahame Grieve (May 12 2018 at 08:15):
there's a spread of actions in health, across casual tracking through to real healthcare provision
Grahame Grieve (May 12 2018 at 08:16):
I have a copy of a legal opinion HL7 got on it's own obligations under GDPR
Grahame Grieve (May 12 2018 at 08:16):
which reminds me - we need a way to audit that the patient has been informed about their rights etc
Grahame Grieve (May 12 2018 at 08:16):
a profile on Audit Trail
John Moehrke (May 12 2018 at 12:27):
The use case of 'record that the subject has been informed about their rights' is in the Consent resource today.
Grahame Grieve (May 12 2018 at 12:28):
really? why? I don't follow that logic to use the Consent resource
John Moehrke (May 12 2018 at 12:28):
Why not?
Grahame Grieve (May 12 2018 at 12:44):
you're just recording that you informed the patient of something. I didn't think that was a proper use of consent
John Moehrke (May 12 2018 at 12:46):
It has been discussed. Need it in USA as it is all that HIPAA asks for. I did just add a CR to make it more clear. The recording of the date/time and conditions of the informing act is still useful.
Jose Costa Teixeira (May 12 2018 at 12:51):
Keep in mind that for gdpr, each consent must be explicit and atomic
Jose Costa Teixeira (May 12 2018 at 12:53):
For each purpose you may have to get consent or not , but that consent works for one purpose.
Jose Costa Teixeira (May 12 2018 at 12:54):
"I've read my rights" does not cover gdpr
Alexander Mense (May 12 2018 at 12:59):
There are two different things ...
a) you might need explicit consent for specific PoU and for others you may not because of legal obligation of the processor to process the data (6c)
b) in any case you have to inform the data subject "at the time when personal data are obtained" about controller, PoU, categories of data concerned, ... (12, 13, 14)
Jie Fan (May 23 2018 at 15:16):
Just want to revive this thread, the spec says "For servers that maintain a version history, the delete interaction does not remove a resource's version history. From a version history respect, deleting a resource is the equivalent of creating a special kind of history entry that has no content and is marked as deleted. Note that there is no support for deleting past versions - see notes on the history interaction." in the delete section, I am wondering what is the rational behind not allowing deleting past versions? It feels to me that occasionally we'll need to delete some of the old versions.
Grahame Grieve (May 23 2018 at 18:05):
most servers will keep a history, in a version control sense. What that makes clear is that the spec version of delete is not the same as GDPR's fantasy of erasure
Grahame Grieve (May 23 2018 at 18:05):
further discussion - see https://chat.fhir.org/#narrow/stream/111-Security-and.20Privacy/subject/GPDR
John Moehrke (May 23 2018 at 18:59):
I'm surprised at this statement in the FHIR specification. It seems we have discussed before that the actions taken on a Delete would be dependent on policy. That the FHIR specification simply indicates how a delete is communicated. The FHIR specification should not indicate the functionality. Where is this text specifically, which page?
Grahame Grieve (May 23 2018 at 19:00):
I don't think we're making policy rules there. We don't say a server has to retain the history, just that the interaction is not understood to delete past versions
John Moehrke (May 23 2018 at 19:02):
what does that mean? And if it mean what I think you just said, that sounds very different than the text Jie mentions.
Jie Fan (May 23 2018 at 19:05):
It's here: https://www.hl7.org/fhir/http.html#delete, the last but one paragraph
Jie Fan (May 23 2018 at 19:12):
Sorry for any confusion, but my question was not less about GDPR, I think there should be some way to delete past versions. It's true that most of time one would not want to delete them, but sometimes such an API is useful.
Grahame Grieve (May 23 2018 at 19:22):
I don't think it sounds different.
Grahame Grieve (May 23 2018 at 19:23):
we would have to change the text if we define an $erase operation, which would remove everything. But we should not change delete to do so.
John Moehrke (May 24 2018 at 14:01):
I tend to disagree. The paragraph is asserting a specific policy. I don't disagree that this is a very likely policy. I think Delete is so specific to policy that we need to be clear that the action actually taken will depend on local policy, and authority of the requesting actor.
John Moehrke (May 24 2018 at 14:09):
I created GF#17254 to further track this formally to a conclusion.
Grahame Grieve (May 24 2018 at 17:49):
I don't know what you think the policy we're making is
Grahame Grieve (May 24 2018 at 17:49):
we're just clarifying what the meaning of the interaction is
Jie Fan (Aug 01 2018 at 00:33):
John and Grahame, where can I check all the GDPR related progress made so far? I am most interested in the "right to be forgotten" part (especially on how it works for historical versions).
Jie Fan (Aug 01 2018 at 00:45):
By the way, how do you like the idea that providing an API at the spec level to purge a specific resource and all its historical versions? A mechanism to handle the "right to be forgotten" case is critical to my team in short-term, and we probably need a solution before an official one lands in the spec. We thought about the patient central approach, i.e. remove all resources associated with a patient, and decided it was less ideal because the server cannot accommodate all possible purging logic, in addition, it is hard to maintain reference integrity from an implementation perspective.
Jose Costa Teixeira (Aug 01 2018 at 00:52):
Just to be sure: why do you want to delete all data from a patient? I ask because the right to be forgotten is not.absolute, and from a gdpr perspective I don't see an obvious case where one would hard delete all patient data.
Grahame Grieve (Aug 01 2018 at 04:23):
secondary records need to support purge. We don't have time to add it for R5 anymore, but HAPI already supports it - I believe it's $purge...
Brian Postlethwaite (Aug 01 2018 at 06:36):
I wonder if that should record an AuditEvent that the $purge occurred... :thinking_face:
John Moehrke (Aug 01 2018 at 13:17):
So, although there are Medical Records retention requirements that will keep an EHR data safe... There are still other reasons to want a $deletePatient operation.
1. First of all, the Medical Records retention period does have a sunset. Even if it is 100 years past death. At some point that Medical Records regulation reason to not Erase expires. At that time one MUST Erase the data. This is mandated by GDPR, when the data is no longer needed for the collection purpose it must be eliminated.
2. the fact that FHIR is not limited to PurposeOfUse of Treatment... One might be using FHIR as a database to a Research project, and that project might have the patient come back and demand to be Erased.
John Moehrke (Aug 01 2018 at 13:18):
The Security wg would very much like to define this operation. Yes it would include audit requirements, and options for how to record that.
John Moehrke (Aug 01 2018 at 13:19):
However, we need interested parties to engage with us to work on this. I only have so much time available. RIght now is summer, so most Europeans are on holiday so not available to help.
John Moehrke (Aug 01 2018 at 13:19):
Which gets back to the overall progress on GDPR... Europeans are on holiday...
John Moehrke (Aug 01 2018 at 13:21):
@Grahame Grieve why is it too late to get into "R5"? It seems we could get it into R4 ballot if we could get someone to propose it. Why has HAPI not proposed their solution?
Grahame Grieve (Aug 01 2018 at 13:23):
I meant R4. The problem is that I have no idea how we’re going to get all the changes made as it is. There’s just no time...
John Moehrke (Aug 01 2018 at 13:28):
whew... should be okay to work on it in R5 timeframe. Having someone like HAPI experimenting is good.
Jose Costa Teixeira (Aug 01 2018 at 13:49):
I advise my gdpr customers to avoid jumping to hard erase. Had to avoid this 'GDPR scare' a few times. Hence my question. I think capability to erase data should be there, but I'd always frame things in terms of "right to be forgotten is not universal" and "anonymize instead of deleting"..
Jose Costa Teixeira (Aug 01 2018 at 13:51):
Since the question was about right to be forgotten, my opinion is - rtbf does not automatically erase all data. I'd advise lots of caution with that.
John Moehrke (Aug 01 2018 at 13:52):
Yes, the discussion always starts with calming them down with a reminder that GDPR Erase (and right to be forgotten) do not override other regulated reasons to maintain the data, such as Medical Records retention ... but you must remind them that is a timeframe with a sunset, and after that sunset the data must be erased. This is the big news to healthcare, as the data tends to NEVER be erased.
Jose Costa Teixeira (Aug 01 2018 at 13:54):
@Jie Fan apologies for bursting in. I just hope it is clear that gdpr does not say that "when the patient wants to delete all their data, the hospital must do a hard delete". I actually think specific case will never or seldom happen. Data most likely will have to be retained
Jie Fan (Aug 01 2018 at 14:17):
Grahame, what do you mean by secondary records?
Jie Fan (Aug 01 2018 at 14:22):
OK, we (Google) would love to engage on this, or even propose our solution, but we have immediate need on such an operation, even R4 might be too late for us. We want to be aligned with the TBD spec solution as much as possible (it'll be great if we could agree on the URL that's going to be used for the solution, but it can be hard as I understand)
John Moehrke (Aug 01 2018 at 14:39):
I suspect Grahame was referring to 'secondary records' as in use of FHIR that is not for "Treatment". This is commonly referred to as secondary use.
Yes, please suggest what you would like it to be. When someone suggests something, it tends to be what is chosen. Yes, HL7 is a consensus organization, so the consensus might come up with different solution. This is almost always with good reason. Even a poorly written solution put into the ballot could result in a recognized R4 operation (at STU). However delay, and it becomes an R5 project. Alternative is it becomes an IG with a path to core, which might be a good place for it to be developed. I am not sure what effort is involved for Security WG (my workgroup) to add a Draft level Operation. It seems it should not be too hard. Something like http://build.fhir.org/resource-operation-convert.html, even without examples.???
Anand Mohan Tumuluri (Aug 01 2018 at 16:16):
Interesting that HAPI supports a $purge API. We also settled on the same exact verb i.e. /Patient/[id]/$purge. @James Agnew Can you please confirm if HAPI supports a $purge API? I was not able to see it anywhere in the documentation or code.
James Agnew (Aug 01 2018 at 16:50):
FYI the HAPI operation is called $expunge. There is info on how it works here: https://smilecdr.com/docs/current/fhir_repository/deleting_data.html#the-expunge-operation .. still need to backport the documentation into HAPI's doc set, but the operation works the same.
Jie Fan (Aug 01 2018 at 18:09):
@James Agnew the $expunge operations fits our use case well, is this what you are going to propose?
James Agnew (Aug 01 2018 at 18:15):
I wouldn't say that I'm proposing it for anything at this point.. We implemented it in response to a specific need that a group of HAPI FHIR users had.
It has some quirks for sure (it'll only delete up to N records, and N is capped to avoid overloading the database) and it will not expunge data that hasn't already (and separately) been deleted the normal way. Personally I think these are both good decisions but who knows...
When is the GDPR connetathon stream happening? Baltimore? Happy to participate.
John Moehrke (Aug 01 2018 at 18:56):
it sounds like $expunge is more of a 'empty waste basket'?
John Moehrke (Aug 01 2018 at 18:59):
I think for GDPR like use-cases the verb $erase seems to be the logical verb for the operation. just because that is the verb used in GDPR. Else $forget, but that seems like a strange one... What kind of parameters? I think we are mostly here talking Patient, would that be too limiting? Or would that help clarity? There would certainly be STU experimentation needed around some of the fringe data, like data that is shared with another patient (mother/child). There would need to be a declaration of who is responsible for checking that the operation should actually work, or not -- is it the caller or service that is responsible?... lots to STU expriment with
René Spronk (Aug 02 2018 at 08:35):
$erase is fairly non-ambiguous, and the wording does indeed fit with the gdpr. If & when one needs to do a hard erase (and at some point in the lifecycle of the data one will have to do so), then such an operation makes sense.
When one defines what the operation does, that's where the caveats are: if one has a resouce X (not a patient resource) referenced from the EHR of a patient P1, and it's also referenced from the EHR of a patient P2, then X should not be erased. There may be meta flags on a resource which indicate 'used in ongoing legal proceedings' etc. These are the usual caveats well known in ILM and ECM in general. Note that HAPI's expunge operation doesn't extend to 'all data associated with patient resource P1', but just to 1 Patient resource P1, without following references, so the scope of the operation will have to clarified.
John Moehrke (Aug 02 2018 at 12:52):
I don't think we should confuse $erase with a correction of an instance of data. Like found in the right to be forgotten, this can be just one report that is being 'forgotten'. Different from GDPR Erase all traces of an individual including that the individual even existed. Thus it would seem the first proposal for $erase would have one input parameter of the Patient to erase... this then can't be confused with more targeted corrections. Right?
René Spronk (Aug 02 2018 at 13:21):
? - my question is whether one should design /Patient/1/$erase in such a way that it only erases Patient/1, or all resources part of the EHR of Patient/1 ? It all depends on whether one wishes the client or the server to have control over this.
John Moehrke (Aug 02 2018 at 13:38):
and my response was... the operation is a request by a client to erase all record of that Patient... I say this because this is the GDPR need, and correction of some piece of data can be done with normal CRUD operations on that piece of data. Hence why it s an operation, rather than just a flavor of CRUD (Delete). As an operation it can also do all the softer (yet required) things. As an operation it might also execute site specific implementation such as de-identification rather than delete, or isolating some data needed by some other regulated action while removing visibility for the non-regulated uses... So, I am proposing the hard part be put into the service, not the client. That is my response to your question... Others can provide their response... you can reject my response. i is just a response
Elliot Silver (Aug 03 2018 at 17:30):
I'm not a GDPR expert by any means (I'm not sure I even qualify as a newbie), but is there a reason that FHIR needs to support/standardize any of these purge mechanisms? What is the interoperability need here?
Jose Costa Teixeira (Aug 03 2018 at 17:41):
For gdpr I would actually look at more atomic operations like finding data related to a patient under a given legal grounds of processing, or data stored at or before a given date...
Jose Costa Teixeira (Aug 03 2018 at 17:45):
Patients don't want only to be forgotten, they want to see what data there is, and why
John Moehrke (Aug 03 2018 at 18:04):
Elliot, the fact that some have found it useful to have an operation does not mean that ALL will find it useful. However if there is some critical set of need, that is what should drive the creation of the operation... The fact it is being discussed shows some interest. I don't think that this is critical evidence of need... But enough need to discuss is a low bar... someone must still come forward with critical evidence of need, and that has not yet happened.
John Moehrke (Aug 03 2018 at 18:04):
Jose, that is a different thread... don't change the subject of a thread that has a clear purpose.
Elliot Silver (Aug 03 2018 at 18:08):
@John Moehrke Agree. My thought was just that having a purge capability, and being able to invoke a purge in an interoperable way are two distinct things. I understand why some are wanting the capability. I don't yet understand the interoperable cross-system need.
Jose Costa Teixeira (Aug 03 2018 at 18:09):
Sure. I am looking at the interoperability needs for gdpr and I am happy Elliott asked what is the need. Discovery is imho a precursor for "hard delete". But that is just my advice. Don't want to distract from the hard delete topic, just frame it.
John Moehrke (Aug 03 2018 at 18:10):
I would agree that what we are talking about could be used beyond GDPR specific Erase need... but then again, we are not done talking... so talk away... None of this is clear as needing to be a defined operation... it all clearly needs to be done. I simply expect these unusual administrative tasks to be done using some service specific method
John Moehrke (Aug 03 2018 at 18:12):
Jose, why is $everything not sufficient? http://build.fhir.org/patient-operation-everything.html
Lloyd McKenzie (Aug 03 2018 at 20:40):
$everything doesn't disambiguate based on purpose of use. GDPR would need to filter on what's tied to the patient and is outside (or inside?) particular purpose-of-use boundaries.
Lloyd McKenzie (Aug 03 2018 at 21:22):
I think the interesting problem therefore isn't so much query, but how to assert/track/expose purpose of use on different records (or data elements - shudder)
John Moehrke (Aug 04 2018 at 15:03):
You can tag each resource with PurposeOfUse if you like... see discussion on the security-labels page http://build.fhir.org/security-labels.html#core
Jose Costa Teixeira (Aug 08 2018 at 17:59):
@John Moehrke thanks for this. Are the Purposes of use coded in the labels? In practice, purposes can range from defined valuesets ("To validate the reimbursement as defined in internal procedure XXXX") to ad-hoc purposes ("i really need this because my boss is away")
John Moehrke (Aug 08 2018 at 21:47):
The Lables can carry purposeOfUse vocabulary, yes. When they do this, they are explicitly labeling that data as having been Gathered for that specific PurposeOfUse. More comonly it is understood that a whole FHIR Server is for a specific purposeOfUse, so there isn't redundant labeling. This voabulary can also be put on a Bundle to indicate that the data is being released for that set of PurposeOfUse... And a specific request to access data can carry a set of purposeOfUse to indicate that the intention of the request is to use the data for those purposes.
Lin Zhang (Dec 22 2020 at 14:00):
@James Agnew I'm trying to figure out how to use the operation $expunge to physically delete "test" resources by type.
John Moehrke (Dec 22 2020 at 15:50):
start a new thread... and likely this is a HAPI question.
Last updated: Apr 12 2022 at 19:14 UTC
