Stream: bulk data
Topic: loading-data
Josh Mandel (Jan 27 2018 at 19:14):
@Grahame Grieve As I try loading data from your server into BQ, I'm hitting some schema issues. http://test.fhir.org/r3/Patient/331037 shows a _given where some of the value are "null" (a string) instead of null (the JSON null primitive). Is this an error in your server?
Grahame Grieve (Jan 27 2018 at 19:38):
yes error in my server
Nagesh Bashyam (Jan 27 2018 at 22:56):
Just wanted to ask for some suggestions on how some of you are implementing the actual database filtering of data and the experience
Is it better to implement something like :
Option 1:
For each patient
For each Resource
select data for each patient
stream the data to the file
End
Option 2:
For patients in (List of all patients)
select
Nagesh Bashyam (Jan 27 2018 at 22:59):
Option 2:
For all patients in (list of patients)
For each Resource
select data
stream data
End
In option 1 , you are selecting data for each patient and then accumulating it, the individual queries dont have large IN clauses.
In option 2, you are trying to select data for patients with large IN clauses which looks to be very inefficient.
Just wanted to get a pulse on approaches before i look into actually looking at the performance and optimization
Grahame Grieve (Jan 27 2018 at 23:09):
I do option 2. but my database is optimised to do that
Last updated: Apr 12 2022 at 19:14 UTC
