Stream: bulk data
Topic: Bulk export for FHIR operations
Yunwei Wang (Apr 22 2019 at 14:44):
I am working on this ValueSet bulk export idea and I have some confusions about the details.
FHIR ValueSet serves as resource for both value set definition (ValueSet.compose) and value set expansion (valueSet.expansion). ValueSet expansion is requested via ValueSet.$expand operation and some terminology servers (if not most) keep valueSet defintion internally.
Consider that, I think a bulk data export request on value set would be like 'export expansions of all CQM value set".
I am thinking using exisiting parameters but twist it a little bit:
$export?_type=ValueSet.$export&_typeFilter=ValueSet%3F$export%3Ffilter%3D'CQM'
or create a new paramter _typeOperation
$export?_type=ValueSet&_typeOperation=export&_operationParameter=filter%3D'CQM'
Grahame Grieve (Apr 23 2019 at 20:54):
we should find some time to talk about this in Montreal
Yunwei Wang (Apr 24 2019 at 15:02):
@Grahame Grieve Do I need to create a tracker? I will not attend Montreal WGM. Please get me posted about any update on this topic.
Josh Mandel (Apr 24 2019 at 19:08):
Applying transformations (e.g. FHIR operations like $expand) to resources prior to export is pushing beyond our initial scope, for sure. My first inclination would be to ask: is this something we'd want for any FHIR query (e.g., synchronously find a Bundle of resources and apply a transform to each before returning) or only in the context of an export?
Yunwei Wang (Apr 24 2019 at 19:40):
@Josh Mandel Understood. But when we talk about bulk export of ValueSet, it makes more sense for exporting the ValueSet expansion rather than ValueSet definition.
Josh Mandel (Apr 24 2019 at 19:41):
And is that true particularly for export, or for other queries that might return multiple valuesets, like a search?
Yunwei Wang (Apr 24 2019 at 19:51):
This is for $export operation. A work flow without $export would be 1) search ValueSet and get a Bundle of ValueSet metadata returned 2) call $expand on each ValueSet in the search result to get expansion (or Bundled transactions).
Josh Mandel (Apr 24 2019 at 20:00):
Understood that we're talking about export here. I'm just wondering whether there is a more general notion of applying an operation to the results of a query that we would want to define at the level of FHIR rather than as a specific feature in $export.
nicola (RIO/SS) (Apr 24 2019 at 21:21):
Just funny coincidence. For terminology we use custom resource Concept - so your case would be just bulk export for us ;)
Yunwei Wang (Apr 25 2019 at 15:42):
@Josh Mandel I don't know about other operations. But this is the problem I met when I tried to apply bulk export to ValueSet.
Yunwei Wang (Apr 25 2019 at 15:44):
@nicola (RIO/SS) What is the Concept resource?
nicola (RIO/SS) (Apr 25 2019 at 15:47):
@Yunwei Wang https://docs.aidbox.app/terminology/concept
Grahame Grieve (Apr 26 2019 at 00:39):
it's on our todo list to consider something like that
Grahame Grieve (Apr 26 2019 at 00:39):
generally, the subject of bulk operations has come up - do $expand on all the value sets that meet some search criteria
Kishor (Sep 18 2019 at 12:43):
Can someone guide me if there are implementations in HAPI FHIR server for FHIR Bulk Data server
Lloyd McKenzie (Sep 18 2019 at 13:03):
@James Agnew ?
Bas van den Heuvel (Sep 18 2019 at 13:05):
We build a proxy server that can be placed in front of Hali.
Josh Mandel (Sep 18 2019 at 13:06):
Is this the one that does everything in memory though? So a good reference implementation but not suitable for large volumes of data?
Bas van den Heuvel (Sep 18 2019 at 13:06):
Exactly
Nick Robison (Sep 18 2019 at 16:40):
@Kishor We have an implementation of a bulk data server that uses HAPI for the interface, but relies on a Postgres database for its persistent queueing. https://github.com/CMSGov/dpc-app
James Agnew (Sep 19 2019 at 19:26):
@Kishor - Bulk data just got merged to master in HAPI FHIR. It's actually live on our public test server now, although there are some nuances to using it. Most importantly it puts the export payloads into Binary resources, and doesn't yet take advantage of externalized binary storage so it's a hog on the database.
We're working to have that cleaned up soon.. Definitely looking for testers, so we'd love to hear your experience.
natus (Sep 26 2019 at 21:11):
@James Agnew is there a chance the bulk data is available for hapi fhir plain server too ?
Kishor (Sep 27 2019 at 19:09):
@James Agnew can you please direct me to HAPI Bulk test server, tried to look for but was not able to get there
James Agnew (Sep 30 2019 at 22:47):
Hi @Kishor - The public test server at hapi.fhir.org supports bulk data now. For example, the following URL initiates a bulk export of claims (hyou have to supply the mandatory Prefer header though so you can't just click the link):
http://hapi.fhir.org/baseR4/$export?_type=Claim
Prasad Meka (Oct 10 2019 at 15:13):
Does any one has client example to export all Patients or claims .
Indu (Oct 21 2019 at 17:41):
I am able to setup hapi-fhir-jpaserver-starter-master locally. But did not find $export when I retrieve the server's conformance statement. Where as public test server has. Is there any difference in Git vs public test server?
Lloyd McKenzie (Oct 21 2019 at 19:16):
@James Agnew
Last updated: Apr 12 2022 at 19:14 UTC
