Stream: bulk data
Topic: Operation triggering the polling
David Pyke (Jan 11 2022 at 14:33):
I'm working on an IG for registry submissions of large amounts of data (thousands of submission Bundles). To do this, I'd like to use an operation to do the initial upload, possibly using a Binary resource as the input with an NDJSON file as the attachment.
This would return an immediate OperationOutcome saying the upload is successful and then attaching a polling endpoint where an NDJON file of OperationOutcomes would be found that would give results of processing each line of the submission file.
The registry is not willing to be fetching the data, they want the upload file.
So, what would the headers look like for each step? Everywhere I look seems to use a GET as the trigger for async and I want to use a POST of a Binary.
I wanted to use OperationOutcome.location to show the polling point but it's deprecated, so I'm assuming it has to be in the 202 Content-Location response
Josh Mandel (Jan 11 2022 at 15:16):
I'm not quite understanding the flow. It sounds like a case for bulk import, but you're saying the bulk import spec is not workable?
David Pyke (Jan 11 2022 at 15:18):
Right, unless I'm reading it wrong. Bulk Import relies on the receiver fetching the data. My receiver wants to have an NDJSON file uploaded, then set a polling point where the NDJSON file of OperationOutcome resources can be fetched, once all the lines in the uploaded file have been validated
David Pyke (Jan 11 2022 at 15:19):
So, my plan is to POST a Binary with the uploaded file
Josh Mandel (Jan 11 2022 at 15:21):
But what are the Operation Outcomes describing? And what's the rationale about "wants to have"?
David Pyke (Jan 11 2022 at 15:21):
The 202 Accepted would have a OperationOutcome with results of verifying the JSON is correct and polling location where the OperationOutcomes that describe if the individual lines validate against the registry data dictionary requirements
David Pyke (Jan 11 2022 at 15:22):
Their workflow is currently based on file submissions. THey don't want to fetch all the Bundles for each encounter that triggered a submission
David Pyke (Jan 11 2022 at 15:23):
THere could be thousands of submissions as a periodic submission set
Michele Mottini (Jan 11 2022 at 15:23):
So, my plan is to POST a Binary with the uploaded file
You need an async custom operation, bulk import does not work like that
David Pyke (Jan 11 2022 at 15:23):
Yep, that's why I'm here
David Pyke (Jan 11 2022 at 15:25):
THe async FHIR docs don't talk about POST as the trigger so I'm a little confused
Josh Mandel (Jan 11 2022 at 15:38):
Who in your design David is responsible for running the validations on each FHIR resource? And who is dereferencing the URLs provided in the bulk manifest?
David Pyke (Jan 11 2022 at 15:38):
It would be the registry that receives the data
Josh Mandel (Jan 11 2022 at 15:39):
But you're just going to synchronously upload gigabytes of data? With no resume capabilities, and no ability to control parallelism?
David Pyke (Jan 11 2022 at 15:40):
That's what they want. We're giving other options but they want this one spec'd out
Josh Mandel (Jan 11 2022 at 15:45):
Is there room to discuss?
David Pyke (Jan 11 2022 at 15:46):
There hasn't been so far. I keep trying, though
Raj Sankuratri (Jan 11 2022 at 17:15):
Hi @Dan Gottlieb , can you post an example of what the response should look like when a bulk data client queries for all the Groups in the FHIR server it is authorized to see?
Last updated: Apr 12 2022 at 19:14 UTC
