Stream: bulk data
Topic: Argonaut Announcements
Josh Mandel (Mar 16 2020 at 19:32):
We'll use this topic for official announcements re: the Argonaut 2020 Bulk Data project.
Josh Mandel (Mar 18 2020 at 16:27):
Cross-post for visibility: this channel has details for today's Argonaut Kick-off call (3p ET)
Josh Mandel (Mar 18 2020 at 17:16):
Update: this call has been moved to https://attendee.gotowebinar.com/register/6855905105082259725 (same time == 3p ET today)
Douglas DeShazo (Mar 19 2020 at 12:40):
@Josh Mandel How do I become a member of this group for bulk data? Should I just attend? I represent Cognizant and was part of the R4 team and CDS Hooks/Web Messaging last year and we are interested in continuing that. Particularly for bulk data and how that might affect payers and the new TEN's that are coming. Thanks.
Michele Mottini (Mar 19 2020 at 12:49):
You just became part of it . . .
Josh Mandel (Mar 19 2020 at 13:36):
http://2020.argo.run has a link to the calendar; you should just join the first session. Before the session we will also have a sign up sheet and survey that we will post here as well as there.
Douglas DeShazo (Mar 20 2020 at 13:31):
Thank you both. @Josh Mandel @Michele Mottini
Scott Fradkin (Mar 20 2020 at 15:32):
With regards to the new Bulk Data IG... is the scope export only? Or will there be work done to incorporate bulk import?
Paul Church (Mar 20 2020 at 15:37):
Yes, there's a draft here: https://github.com/smart-on-fhir/bulk-import/blob/master/import.md
Dan Gottlieb (Mar 20 2020 at 15:45):
@Scott Fradkin @Paul Church the current plan is to scope the 2020 Argonaut bulk data effort to an update to the export spec to align with the upcoming EHR certification requirements (of course, that doesn't preclude continuing work on import outside of the Argonaut process)
Scott Fradkin (Mar 20 2020 at 15:49):
@Dan Gottlieb If not Argonaut, who _is_ working on the import IG?
Dan Gottlieb (Mar 20 2020 at 15:55):
@Scott Fradkin As far as I know it's not currently being actively worked on. It was one of the Argonaut projects proposed for this year, but the steering committee chose to prioritize improving the export spec in 2020 and leave work on an import ig for a future date.
Dan Gottlieb (Mar 20 2020 at 16:03):
For folks interested in participating in the 2020 Argonaut bulk data workgroup, we've posted a brief survey where you can sign up and help determine the project's priorities. If you'd like to be involved, please complete the survey at https://forms.gle/V9B9UaRbWK6UoSPs8 before March 26th.
Dan Gottlieb (Mar 20 2020 at 16:03):
Also, if you weren't able to attend the kickoff meeting earlier this week, a recording has been posted at https://www.youtube.com/watch?v=h18HjsPk6Rc and provides context for this survey.
Dan Gottlieb (Mar 20 2020 at 16:04):
Our first working group meeting will take place on April 2nd at 2pm ET - details are at http://2020.argo.run . Looking forward to thinking through improvements to the Bulk Data Export IG together!
Scott Fradkin (Mar 20 2020 at 16:13):
Thanks for the info @Dan Gottlieb
Dan Gottlieb (Mar 26 2020 at 15:00):
If you haven't filled out the brief survey at https://forms.gle/V9B9UaRbWK6UoSPs8 and would like to participate in the Argonaut 2020 Bulk Data workgroup (focusing on v1.2 of the IG), please be sure to do so by the end of day today. Thanks!
Dan Gottlieb (Apr 02 2020 at 13:29):
The first Argonaut 2020 Bulk Data working group meeting will take place today at 2pm ET - please join if you'd like to be involved!
Dan Gottlieb (Apr 02 2020 at 13:29):
https://meet.jit.si/argo2020.bulk
To join by phone instead, tap this: +1.512.402.2718,,3136221482#
Other dial-in numbers are at https://meet.jit.si/static/dialInInfo.html?room=argo2020.bulk
Dan Gottlieb (Apr 02 2020 at 19:04):
Meeting notes are at https://bit.ly/argo20-bulk
Dan Gottlieb (Apr 13 2020 at 15:11):
I drafted language describing an approach to handling file attachments in the next version of the Bulk IG at https://github.com/HL7/bulk-data/issues/63
Dan Gottlieb (Apr 13 2020 at 15:12):
Please add your thoughts (particularly around the open questions listed)!
Dan Gottlieb (Apr 13 2020 at 15:12):
This will also be our discussion topic for the next bulk data Argonaut call on Thursday at 2pm ET, so please join if you'd like to talk this through (note that an early version of the invite listed the call at 2:30pm ET in error).
Paul Church (Apr 13 2020 at 21:05):
re: question 1 - are you suggesting that the server would package the contents into a Binary resource that did not actually exist on the server but exists in the output, or just that the server must include the Binary resource that it has?
Josh Mandel (Apr 13 2020 at 21:26):
The suggestion was that a server would have the option of packaging contents in this way, if it saw fit. Even if the binary resource did not actually exist on the server.
Josh Mandel (Apr 13 2020 at 21:27):
Given the discussion on the GitHub issue, I am leaning away from this model... but it leaves open the question of what we should do when a server does maintain binary resources directly.
Dan Gottlieb (Apr 13 2020 at 21:41):
Are there use cases for including Binary resources in a bulk data response in the absence of direct links from other resources?
Dan Gottlieb (Apr 13 2020 at 21:41):
If not, perhaps servers can just rewrite these referenced Binary resources to files in a location accessible to the client (or expose the underlying FHIR server to provide direct access to the referenced resources).
Paul Church (Apr 13 2020 at 22:01):
I wouldn't want to reject the possibility of unreferenced Binary resources regardless of whether we can immediately come up with a use case - it's just another resource type, you can export all of the other ones.
John Moehrke (Apr 13 2020 at 22:03):
how is this hard size limit being handled with resources in general? Binary is not the only one that could become huge. The genetic sequences are huge, as are many micro observations.
Jenni Syed (Apr 13 2020 at 22:42):
This was the first discussion that was part of the initial USCDI scope of docs/US Core
Jenni Syed (Apr 13 2020 at 22:43):
The bulk data files are already gigs and gigs large since they have entire population records. Binary is just a "larger" single entry that is currently possible
Paul Church (Apr 14 2020 at 03:54):
One of the considerations in rewriting Attachment.url should be what it might look like from the $import side. I think using Binary is actually easier in that case? Otherwise the importer has to be able to piece together what namespace the url is in, find the corresponding file in its input, and rewrite the url back to something else depending on how it handles large blobs.
Dan Gottlieb (Apr 14 2020 at 13:47):
I don't think we necessarily have to treat them differently, but Binary resources are different from other resources - they act more like a proxy to a file store than serialized data (ie, lack of search, exclusion from compartments, access without FHIR mime type, etc).
Dan Gottlieb (Apr 14 2020 at 13:47):
I like the encapsulation of embedding them in ndjson, but per Vlad's comment on GitHub, this could be burdensome to some clients (particularly for group level and patient level export where there can't be unreferenced Binary resources and the client implementations may not be particularly robust).
Lee Surprenant (Apr 15 2020 at 13:32):
Dan Gottlieb said:
I don't think we necessarily have to treat them differently, but Binary resources are different from other resources - they act more like a proxy to a file store than serialized data (ie, lack of search, exclusion from compartments, access without FHIR mime type, etc).
I agree they are different. If they aren't referenced from somewhere, its very hard to know what is in these things (beyond just the mimetype). In DSTU2 days, we had a bear of a time discussing this resource type with our security and privacy group. In STU3 they added the "securityContext" field, but I'm not sure if all the servers implement this or not:
When the client provides the binary to the server itself, it uses the securityContext element (or the matching X-Security-Context HTTP header) to inform the server that the Binary resource should be treated as if it was the other resource. Typically, the other resource is a DocumentReference or similar resource that refers directly to the Binary resource, but that is not mandatory.
John Moehrke (Apr 15 2020 at 13:38):
and that is why the DocumentReference resource is so critical ... to provide that metadata about the Binary. Binary are more manageable, still unusual, when they are described with a DocumentReference
John Moehrke (Apr 15 2020 at 13:39):
some are indeed just a proxy to a file store... where others are published as bits into the FHIR server...
John Moehrke (Apr 15 2020 at 13:41):
There is some interest (there is a CR) in adding a filename to the Attachment datatype or DocumentReference resource... Not that a filename is helpful here, but given you all interest and discussion... I would be interested in perspective on that CR need. I pushed it over to FHIR-I for consideration first as part of Attachment datatype. There a question of core vs core-extension...
Dan Gottlieb (Apr 16 2020 at 13:48):
The next Argonaut 2020 Bulk Data working group meeting will take place today at 2pm ET. Please join if you'd like to be involved! Note that an early version of the calendar invite incorrectly listed the time as 2:30pm ET.
Dan Gottlieb (Apr 16 2020 at 13:49):
https://meet.jit.si/argo2020.bulk
To join by phone instead, tap this: +1.512.402.2718,,3136221482#
Other dial-in numbers are at https://meet.jit.si/static/dialInInfo.html?room=argo2020.bulk
Meeting notes are at https://bit.ly/argo20-bulk
Dan Gottlieb (Apr 27 2020 at 17:42):
Following up on our last workgroup meeting, I drafted an update to the Bulk IG incorporating the simple approach we discussed for the common case of exporting patient documents, as well as an approach for handling "orphan" Binary resources: https://github.com/HL7/bulk-data/pull/64 . Comments on the PR are very welcome, particularly if they come with a proposal of specific IG language :)
Dan Gottlieb (Apr 27 2020 at 17:43):
I also drafted straw man language to address the next two Argonaut use cases at https://github.com/HL7/bulk-data/pull/68 and https://github.com/HL7/bulk-data/pull/69 that we can discuss at this week's workgroup meeting.
Dan Gottlieb (Apr 30 2020 at 14:01):
The next Argonaut 2020 Bulk Data working group meeting will take place today at 2pm ET. Please join if you'd like to be involved! Note that an early version of the calendar invite incorrectly listed the time as 2:30pm ET.
Dan Gottlieb (Apr 30 2020 at 14:01):
Agenda and notes from past meetings are at https://bit.ly/argo20-bulk
Dan Gottlieb (Apr 30 2020 at 14:01):
https://meet.jit.si/argo2020.bulk
To join by phone instead, tap this: +1.512.402.2718,,3136221482#
Other dial-in numbers are at https://meet.jit.si/static/dialInInfo.html?room=argo2020.bulk
Dan Gottlieb (May 13 2020 at 14:30):
We're canceling the Argonaut meeting tomorrow (May 14th) so folks can focus on the FHIR Connectathon. Our next Argonaut Bulk Data Working Group meeting will be on May 28th at 2pm ET.
Dan Gottlieb (May 27 2020 at 13:43):
The next Argonaut 2020 Bulk Data working group meeting will take place tomorrow (Thursday) at 2pm ET.
Dan Gottlieb (May 27 2020 at 13:43):
We'll be discussing managing groups where members change over time, focusing on the proposal at https://github.com/HL7/bulk-data/pull/71/files?short_path=985cef1#diff-985cef1e2f9cd85436f27813aa6de763 (which includes all language from #68), so please try to review it in advance. Also, if time permits, we'll discuss transmitting information on resource deletions, focusing on the proposal at https://github.com/HL7/bulk-data/pull/69/files?short_path=985cef1#diff-985cef1e2f9cd85436f27813aa6de763 .
Dan Gottlieb (May 27 2020 at 13:43):
https://meet.jit.si/argo2020.bulk
To join by phone instead, tap this: +1.512.402.2718,,3136221482#
Other dial-in numbers are at https://meet.jit.si/static/dialInInfo.html?room=argo2020.bulk
Dan Gottlieb (May 28 2020 at 19:12):
Our meeting notes are a bit sparse today :). If you participated in the call, please help edit for folks who couldn't make it at https://bit.ly/argo20-bulk !
Dan Gottlieb (Jun 11 2020 at 14:04):
The next Argonaut 2020 Bulk Data working group meeting will take place today at 2pm ET
Dan Gottlieb (Jun 11 2020 at 14:04):
We'll be continuing (concluding?) our discussion of managing groups where members change over time, focusing on the proposal at https://github.com/HL7/bulk-data/pull/71/files?short_path=985cef1#diff-985cef1e2f9cd85436f27813aa6de763 . Also, if time permits, we'll discuss transmitting information on resource deletions, focusing on the proposal at https://github.com/HL7/bulk-data/pull/69/files?short_path=985cef1#diff-985cef1e2f9cd85436f27813aa6de763 .
Dan Gottlieb (Jun 11 2020 at 14:04):
Meeting is at: https://meet.jit.si/argo2020.bulk
Dan Gottlieb (Jun 11 2020 at 14:04):
Agenda and meeting notes at: bit.ly/argo20-bulk
Dan Gottlieb (Jun 25 2020 at 17:06):
The next Argonaut 2020 Bulk Data working group meeting will take place today at 2pm ET at https://meet.jit.si/argo2020.bulk .
Dan Gottlieb (Jun 25 2020 at 17:07):
We'll be discussing PR #74 - Signal that server may remove downloaded files,
PR #75 - Transient error code guidance, and time permitting, PR #76 - Parameter for clients to indicate metadata resource requirements. Links to these pull requests are in the agenda at https://bit.ly/argo20-bulk .
Dan Gottlieb (Jul 09 2020 at 13:32):
The next Argonaut 2020 Bulk Data working group meeting will take place today at 2pm ET at https://meet.jit.si/argo2020.bulk . Today's agenda is at https://bit.ly/argo20-bulk .
Dan Gottlieb (Jul 22 2020 at 14:43):
Since a number of the usual participants aren't able to make it this week, we're going to cancel this week's Argonaut bulk data call - our next meeting will be on August 6th.
Dan Gottlieb (Jul 22 2020 at 14:43):
In the interim, if your organization is planning to test a client or server at the September Connectathon or is more generally working on a prototype, please add the details to the spreadsheet at https://docs.google.com/spreadsheets/d/18QmoxCp2D_4yIHC9reIxPU-kQDvTwm9X6yXIqxnBKPc/edit#gid=0 . Feel free to fill out columns as "planned for September" or "stretch goal for January" rather than just a "y" or "n".
PKumar (Jul 23 2020 at 11:47):
Ok @Michael Hansen Thanks for the response. I will utilize the references.
Vishak OS (Jul 23 2020 at 20:17):
Dan Gottlieb said:
Since a number of the usual participants aren't able to make it this week, we're going to cancel this week's Argonaut bulk data call - our next meeting will be on August 6th.
is this the link to the meeting invite for Aug 6th:
https://meet.jit.si/argo2020.bulk
2pm - 3pm EST?
Dan Gottlieb (Jul 23 2020 at 20:34):
@Vishak OS yup - the meeting calendar is at http://2020.argo.run .
Dan Gottlieb (Aug 05 2020 at 14:58):
Reminder - the next Argonaut 2020 Bulk Data working group meeting will take place tomorrow at 2pm ET at https://meet.jit.si/argo2020.bulk . The agenda is at https://bit.ly/argo20-bulk .
Dan Gottlieb (Aug 05 2020 at 14:59):
Also, if your organization is planning to test a client or server at the September Connectathon or is more generally planning on a prototype, please add the details to the spreadsheet at https://docs.google.com/spreadsheets/d/18QmoxCp2D_4yIHC9reIxPU-kQDvTwm9X6yXIqxnBKPc/edit#gid=0 (I pre-populated it with data from the May Connectathon and we'll be discussing at the meeting this week). Feel free to fill out columns as "planned for September" or "stretch goal for January" rather than just a "y" or "n".
Vishak OS (Aug 06 2020 at 16:51):
Dan Gottlieb said:
Vishak OS yup - the meeting calendar is at http://2020.argo.run .
Thanks Dan.
Dan Gottlieb (Aug 19 2020 at 18:37):
Reminder - the next Argonaut 2020 Bulk Data working group meeting will take place tomorrow at 2pm ET at https://meet.jit.si/argo2020.bulk . The agenda is at https://bit.ly/argo20-bulk .
Dan Gottlieb (Aug 19 2020 at 18:38):
Also, I did a first pass at updating the conformance resources in the v1.2 IG to align with the new Argonaut features (including defining a code system and value set for the includeAssociatedData parameter). Let me know if anything doesn't look right (pull requests on the branch would be great as well): http://build.fhir.org/ig/HL7/bulk-data/branches/conformance/
Brett Marquard (Aug 26 2020 at 16:19):
This Friday 8/28 from 2-3 PM EDT the Argonaut team will provide a brief update on the active 2020 projects:
- Bulk Data
- Granular Controls
- Patient Lists
We will also outline our goals for the upcoming Connectathon 9/9-9/11/2020.
Please join us if you can!
Join the meeting:
https://meet.jit.si/argo2020
To join by phone instead, tap this: +1.512.402.2718,,3552657320#
Dan Gottlieb (Sep 03 2020 at 14:35):
Reminder - the next Argonaut 2020 Bulk Data working group meeting will take place today at 2pm ET at https://meet.jit.si/argo2020.bulk . The agenda is at https://bit.ly/argo20-bulk .
Dan Gottlieb (Sep 03 2020 at 14:36):
In addition to our usual conversation around open issues on the spec, we'll be discussing plans for next week's FHIR Connectathon.
Dan Gottlieb (Sep 17 2020 at 14:07):
Reminder - the next Argonaut 2020 Bulk Data working group meeting will take place today at 2pm ET at https://meet.jit.si/argo2020.bulk . The agenda is at https://bit.ly/argo20-bulk .
Dan Gottlieb (Oct 01 2020 at 13:41):
Reminder - the next Argonaut 2020 Bulk Data working group meeting will take place today at 2pm ET at https://meet.jit.si/argo2020.bulk . The agenda is at https://bit.ly/argo20-bulk .
Dan Gottlieb (Oct 01 2020 at 13:41):
It should be a quick call (currently, our only topic is to review the pull request resulting from our discussion last meeting), so feel free to add other agenda items if there are topics you'd like to discuss with the group.
Dan Gottlieb (Oct 15 2020 at 14:06):
The only item we have on the agenda for the Argonaut Bulk Data meeting today is reviewing PR 94 that addresses an issue that raised by @Cooper Thompson by loosening the restrictions on the content of error messages returned from a status request.
Dan Gottlieb (Oct 15 2020 at 14:06):
Does anyone have other items they want to discuss with the group?
Dan Gottlieb (Oct 15 2020 at 14:06):
If not, I'd suggest we cancel today's meeting and folks can share thoughts on the PR on GitHub, and provide any updates on prototype implementations on a thread here.
Dan Gottlieb (Oct 15 2020 at 16:03):
Ok - cancelled the meeting this afternoon at 2pm ET. Our next scheduled call is on 10/29.
Dan Gottlieb (Oct 29 2020 at 13:48):
We don't currently have any agenda items for the bulk data call this afternoon. Does anyone have topics they'd like to discuss or should we cancel again for this week?
Dan Gottlieb (Oct 29 2020 at 13:49):
Also, since we've addressed the areas of improvement we identified at the start of the project and are getting to the end of the Argonaut v1.5 effort, I posted a changelog outlining the updates the workgroup has made at http://build.fhir.org/ig/HL7/bulk-data/branches/changelog/changelog/index.html ( https://github.com/HL7/bulk-data/blob/changelog/spec/changelog/index.md ). Please review and let me know if I missed anything or if any of the items are unclear!
Dan Gottlieb (Oct 29 2020 at 16:33):
Ok - cancelled this afternoon's meeting.
Dan Gottlieb (Nov 10 2020 at 14:54):
Since we've addressed the IG enhancements defined by the Argonaut members at the start of the project and are getting to the end of the year, it probably makes sense to wind down the regular working group meetings.
Dan Gottlieb (Nov 10 2020 at 14:55):
Let's plan to keep our scheduled bulk data working group meeting this Thursday at 2pm ET to wrap up and discuss next steps for the project.
Dan Gottlieb (Nov 10 2020 at 14:55):
As a reminder, the full change log for the v1.5 work is at http://build.fhir.org/ig/HL7/bulk-data/changelog/index.html - please let me know if any of the items are unclear!
Dan Gottlieb (Nov 12 2020 at 15:14):
Reminder - the next Argonaut 2020 Bulk Data working group meeting will take place today at 2pm ET at https://meet.jit.si/argo2020.bulk .
Dan Gottlieb (Nov 12 2020 at 15:14):
Feel free to add items to the agenda is at https://bit.ly/argo20-bulk - the current plan is for a brief call to wrap up the Argonaut bulk project work and discuss next steps.
Douglas DeShazo (Mar 23 2021 at 12:23):
Are there any meetings for the Bulk Data project this year or is the focus on Connectathons and testing out the latest IG?
Dan Gottlieb (Mar 23 2021 at 13:16):
We wrapped up the Argonaut Bulk Data IG update project at the end of last year and are in the process of feeding the resulting IG v1.1 changes into the HL7 ballot process. However, I just posted a track page for Bulk Data for the May Connectathon, and expect that folks will be testing updated clients and servers there!
Douglas DeShazo (Mar 23 2021 at 16:53):
Thanks @Dan Gottlieb I attended as many of the meetings last year as I could and wanted to confirm the project meetings did end last year. Will check out FHIR confluence for the bulk data Connectathon track.
Last updated: Apr 12 2022 at 19:14 UTC
