Stream: implementers
Topic: Epic Bulk Data Export questions?
John Silva (Aug 04 2021 at 18:49):
Recently noticed that Epic is planning on providing Bulk Data Export in August 2021 (now?):
https://fhir.epic.com/Documentation?docId=fhir_bulk_data
It references supporting the 1.0.1 Bulk Data Export specs --- but it doesn't show supporting NDJSON and the example on the page shows Accept: application/fhir+json not the required application/fhir+ndjson. (http://hl7.org/fhir/uv/bulkdata/export/index.html#query-parameters); is this an oversight on the documentation page? (a compliant server is allowed to support other formats but "Servers SHALL accept the full content type of application/fhir+ndjson" )
Also, for the Group export, who or how are the Groups defined? Is that done in the Epic EHR or can it be done by a Epic FHIR API creating the Group resource?
David Pyke (Aug 04 2021 at 18:51):
It says right in the docs:
The format of the bulk data files is ndjson. The ndjson format is similar to JSON, but is newline-sensitive. For more information, refer to the ndjson specification. Resource instances are included in the bulk data file through a search by Patient ID or through a reference in a previously gathered resource instance. The files do not differentiate by patient, so results for all patients are included in each resource file.
John Silva (Aug 04 2021 at 18:57):
OK, thanks! I missed that but I saw this (seemingly incorrect) requirement for HTTP headers:
Your request must include the following headers:
Accept: application/fhir+json
Prefer: respond-async
Cooper Thompson (Aug 04 2021 at 20:09):
The Bulk FHIR spec requires application/fhir+json on the export request, not ndjson. That accept header controls the OperationOutcome response format to $export, not the file format.
Cooper Thompson (Aug 04 2021 at 20:10):
Our file format is ndjson.
Cooper Thompson (Aug 04 2021 at 20:11):
Groups are defined out-of-band by the admins at the healthcare orgs using Epic.
Cooper Thompson (Aug 04 2021 at 20:13):
P.S. Accept headers for kickoff was discussed recently here: https://chat.fhir.org/#narrow/stream/179250-bulk-data/topic/Bulk.20Export.20Headers.20-.20required.20for.20servers.20or.20clients.3F
John Silva (Aug 04 2021 at 21:21):
@Cooper Thompson - thanks for these answers.
So, is this available yet on any Epic systems or does "August 2021" mean the end of the month? Is it available on the Epic sandbox(es) in AppOrchard? [I guess that might become obvious if we see the ability to enable the Bulk Data APIs as described in the Epic Bulk Data Export documentation.]
OK, Groups are defined "out-of-band" -- but how would an integrator/developer find these since there doesn't seem to be an Epic API search for the Group resource? Does an Epic admin, after they create a Group, have to capture the Group ID string and send it to the developer?
BTW, the Epic Bulk Data Documentation page has some error that causes this popup to come up:
Screen-Shot-2021-08-04-at-5.19.41-PM.png
Cooper Thompson (Aug 04 2021 at 21:26):
Aug 21 is our release (Epic versions are named based on the quarter they come out). In the coming weeks we'll have our sandboxes up to that version. Healthcare organizations will be updating to the Aug 21 release over the next several months (the exact timelines are up to the healthcare organizations).
Cooper Thompson (Aug 04 2021 at 21:27):
And for Group IDs, you are exactly right - the healthcare organization's admin (not Epic staff) will define the group, and send you the group ID via (for example) email.
Cooper Thompson (Aug 04 2021 at 21:27):
And yeah, I noticed that popup. I'll work on getting that fixed.
John Silva (Aug 04 2021 at 21:35):
It's going to be interesting to see how this workflow process pans out --- the Group(s) of patients can be very dynamic it would seem. It also seems like the group would (should?) be defined by some search criteria, e.g. (disease) Conditions, CareTeams, Providers, etc.. I assume the Epic Admin interface will provide Epic "super users" (?) the ability to define these groups in a flexible way.
Cooper Thompson (Aug 04 2021 at 21:44):
Our groups are defined by criteria. Technically you could make that criteria be a fixed list of patients, but we expect that most use cases will be using rule-based group definitions (for example, all patients with condition X discharged in the last week, or all patients with a specific insurance plan or payer).
Brendan Keeler (Aug 06 2021 at 04:46):
Really exciting stuff to see this so early in the wild, even with the out of band Group definition and patient limits
Cooper Thompson (Aug 06 2021 at 12:00):
I'm curious if folks think that doing Group definition via the API is ever going to be realistic or reasonable from a data security perspective. Are there cases where a data holder is going to allow a data consumer to arbitrarily define what patients the consumer wants to access without an approval process? The system trust would have to be super high for that to happen I'd think.
Josh Mandel (Aug 06 2021 at 13:40):
Absolutely -- in many cases the api client is managed by and run on behalf of the healthcare system, so the level of trust is as high as possible (it's trust in self).
Michele Mottini (Aug 06 2021 at 13:41):
I think is realistic. Permissions will still be enforced, data consumers won't be able to see things they are not already able to see with normal REST calls
John Silva (Aug 06 2021 at 20:01):
The Epic AppOrchard config already allows an admin to enable/disable a particular READ or SEARCH API per application so it would be possible, if the Group resource had the ability to enable READ/SEARCH/WRITE for those specific applications that need to create Groups. I would suspect that, at a minimum, Group READ would need to be enabled in order to run the Bulk Export in the first place.
Cooper Thompson (Aug 06 2021 at 20:06):
You don't need Group.read to do the export. We consider the read interaction and the $export operation to be different APIs.
John Silva (Aug 06 2021 at 20:09):
OK, but AppOrchard config could expose Group.WRITE as a way to control which applications have the ability to create Groups for the purpose of Bulk Export (and other use cases as well).
[Currently I guess AppOrchard, in fact the Epic API in whole, only allows read/search so allowing Write/Update would be a 'new thing'. I remember checking the CompatibilityStatement and I think it only advertises READ and SEARCH.]
Cooper Thompson (Aug 06 2021 at 20:21):
(deleted)
Last updated: Apr 12 2022 at 19:14 UTC
