FHIR Chat · January 2022 Connectathon · bulk data

Stream: bulk data

Topic: January 2022 Connectathon

Dan Gottlieb (Dec 14 2021 at 15:41):

The January FHIR Connectathon is coming up soon!

Dan Gottlieb (Dec 14 2021 at 15:42):

For the Bulk Data track we're thinking of focusing on the following areas (certainly open to other ideas as well):

Dan Gottlieb (Dec 14 2021 at 15:43):

• Testing FHIR bulk data clients and servers, particularly for new and updated client and server implementations now that v2 of the Bulk IG is out
• Prototyping implementations of the Bulk Import proposal. @Vladimir Ignatov will be bringing an open source client and server and @Sam Sayer mentioned he may be able to bring a prototype.
• Prototyping FHIR NLP transforms. @Bin Mao will be experimenting with a FHIR notes -> NLP -> structured FHIR data pipeline (and is scheduled to demo his prototype on the afternoon of the first day).

Dan Gottlieb (Dec 14 2021 at 15:44):

If you're planning to participate and bring a server or an open source client, please add or update your entry on the tracking spreadsheet.

Dan Gottlieb (Dec 14 2021 at 15:44):

@Cooper Thompson will Epic be able to provide access to the Bulk Data test environment again?

Dan Gottlieb (Dec 14 2021 at 15:45):

@Michael Turman any updates on Cerner's production path Bulk Data server?

Dan Gottlieb (Dec 14 2021 at 15:46):

@Jason Vogt - I think you mentioned that Meditech is working on Bulk API support - any chance you'll have a test environment ready for the Connectathon?

Michael Turman (Dec 14 2021 at 21:33):

We're working on it - we want to be ready to do some testing at May's connectathon.

Sam Sayer (Dec 15 2021 at 16:23):

Confirming we will have a prototype import client and server. It's a bit barebones atm, but we're looking forward to testing with other implementations.

Cooper Thompson (Dec 15 2021 at 21:03):

Put me down as a "probably" for getting our server up. We won't be active in the track, but I should be able to get stuff set up for folks to play with.

Dan Gottlieb (Dec 16 2021 at 15:52):

@Sam Sayer and @Vladimir Ignatov, excited to hear you'll both be testing bulk import prototypes to the Connectathon! I added a tab to our tracking spreadsheet so there's a central place to record info about the implementations such as relevant repos and how to access hosted versions (if applicable).

Sam Sayer (Dec 17 2021 at 17:02):

Our lead dev @Matthew Gramigna is out this week, but he can make the updates when he's back on Monday

Matthew Gramigna (Dec 21 2021 at 21:05):

@Dan Gottlieb We (MITRE) will be bringing a few pieces of bulk data-related software to the connectathon as @Sam Sayer mentioned. We'll have a FHIR server that implements bulk import pnp spec. We'll also be bringing a testing client built with Inferno (https://github.com/inferno-framework/inferno-core) that has a few tests for the import pnp workflow (kickoff, polling status location w/ exponential backoff, etc.). I can add this info to the spreadsheet. Excited to test things (and hopefully not break everything too badly)

Dan Gottlieb (Dec 22 2021 at 15:30):

Sounds great - it will be interesting to see if it's possible to get an import working between your data consumer and @Vladimir Ignatov's data provider, as well as to try and run your testing tool against Vlad's data consumer!

Vladimir Ignatov (Dec 23 2021 at 20:19):

My setup is a little odd so I don't have specific data provider (any bulk data capable FHIR server should work). I'm uploading a pic to try to visualize that, but hopefully we can clarify more details during the connectathon.
Untitled-drawing.jpg

Matthew Gramigna (Dec 29 2021 at 18:57):

Vladimir Ignatov said:

My setup is a little odd so I don't have specific data provider (any bulk data capable FHIR server should work). I'm uploading a pic to try to visualize that, but hopefully we can clarify more details during the connectathon.
Untitled-drawing.jpg

This looks great. Our architecture is very very similar. Just swap out the bulk import mediator for our Inferno testing client, and swap out S3/FS for MongoDB, seen here bulk-import.png

Dan Gottlieb (Jan 05 2022 at 17:33):

Hi folks - if you're planning to participate in the Bulk Data Connectathon Track next week, please join our orientation call tomorrow (Thursday) at 12pm ET!

Dan Gottlieb (Jan 05 2022 at 17:33):

Details on this call and the track are at https://confluence.hl7.org/display/FHIR/2022-01+Bulk+Data

Dan Gottlieb (Jan 06 2022 at 19:44):

For those who weren't able to attend the orientation call today, slides are here.

Dan Gottlieb (Jan 06 2022 at 19:44):

Also, if you're planning to participate in the track, please add your name to the sign up sheet.

Dan Gottlieb (Jan 06 2022 at 19:44):

Planning to test a bulk data server or client? Add it to the list!

Matthew Gramigna (Jan 11 2022 at 15:48):

@Dan Gottlieb Just FYI, for a quality data bulk import breakout, the MITRE team is leading a different breakout for Clinical Reasoning at 2pm eastern today, so that is our only conflict. Any other time should work

EDIT: and @Sam Sayer has a conflict at 1PM too sorry

Jamie Jones (Jan 11 2022 at 15:48):

Bulk-Data-January-2022-Connectathon.xlsx Snapshot of the Connectathon 29 Tracking Sheet for folks who can't access Google Sheets

Sam Sayer (Jan 11 2022 at 16:47):

@Dan Gottlieb How about tomorrow at 12pm CT/1pm ET?

Dan Gottlieb (Jan 11 2022 at 16:51):

Works for me! I'll tentatively add it to whova. Is there anyone who wants to attend that can't make that time?

Dan Gottlieb (Jan 11 2022 at 17:30):

I added a breakout to discuss using Bulk Data for quality measures to the schedule: https://whova.com/portal/webapp/conne_202201/Agenda/2118739 . It's open to everyone, so please join if this is an area you're interested in or working on!

Paul Bastide (Jan 11 2022 at 18:04):

Hey, I tried using the bulk-data-client project, and hit an issue where the Bearer token was lower case. I opened an issue https://github.com/smart-on-fhir/bulk-data-client/issues/1 (note I changed the code locally to be upper case B. This is more of an FYI, and hte issue is out there just in case there is someone else that might hit this issue

Matthew Gramigna (Jan 11 2022 at 18:28):

Question about Parameters for the bulk import kickoff request:

In the spec, it says that the exportUrl parameter is a string, so our server was using valueString as the value[x] in the FHIR Parameters body. We noticed that the bulk-import-client reference implementation is using valueUrl instead. Is valueUrl desired instead of string? image.png

Dan Gottlieb (Jan 11 2022 at 19:21):

valueUrl seems more correct to me since I think this should always be a url. What are your thoughts - should we update the IG to indicate this?

Raj Sankuratri (Jan 11 2022 at 19:27):

Jamie Jones said:

Bulk-Data-January-2022-Connectathon.xlsx Snapshot of the Connectathon 29 Tracking Sheet for folks who can't access Google Sheets

THank you!

Dan Gottlieb (Jan 11 2022 at 19:28):

@Paul Bastide thanks for catching this and for the issue!

Paul Bastide (Jan 11 2022 at 19:29):

The tool is providing us some interesting input/output - we found one issue with it. So kudos!

Raj Sankuratri (Jan 11 2022 at 19:36):

A question on Patient Attribution: Is the main focus of the Patient Attribution on Claims-based attribution for Bulk Data export?

Matthew Gramigna (Jan 11 2022 at 20:03):

Dan Gottlieb said:

valueUrl seems more correct to me since I think this should always be a url. What are your thoughts - should we update the IG to indicate this?

I think it makes sense to go with the strictest type since it's always guaranteed to be the case, so valueUrl makes sense. Thanks

Dan Gottlieb (Jan 11 2022 at 20:05):

@Raj Sankuratri can you expand a bit on the use cases you're thinking of?

Matthew Gramigna (Jan 11 2022 at 21:05):

Matthew Gramigna said:

Dan Gottlieb said:

valueUrl seems more correct to me since I think this should always be a url. What are your thoughts - should we update the IG to indicate this?

I think it makes sense to go with the strictest type since it's always guaranteed to be the case, so valueUrl makes sense. Thanks

But also related to this.. I think what the spec could benefit from is an example request body for kicking off the $import. just to show the structure of the FHIR parameters resource and how to include params

Dan Gottlieb (Jan 11 2022 at 21:11):

Makes sense - want to propose something in a pull request or issue?

Raj Sankuratri (Jan 11 2022 at 21:23):

Dan Gottlieb said:

Raj Sankuratri can you expand a bit on the use cases you're thinking of?

The use case would be a request for {base}/Patient/$export from a provider organization. My understanding is that I have to make sure the data I am sending belong to only members that are currently under the care of or were under the care of the requesting organization. This we can deduce from the Claims-based Patient Attribution. However, during the CDEX session this afternoon, I learned that the list of Patients is negotiated between the Payor and the requesting provider organization.

Dan Gottlieb (Jan 11 2022 at 21:52):

Ah, got it. The CMS example I mentioned is going the other way - payor to provider. I'm not aware of any provider to payor bulk data implementations yet, though that might overlap with our quality measure discussion break out tomorrow.

Dan Gottlieb (Jan 11 2022 at 22:01):

I'd think the bulk exporting system (EHR or data warehouse) would need a way to add a set of patients to a group via a UI or algorithm. This could be a FHIR group exposed using the Groups API or a cohort in the back end that defines the patients available to a particular client_id for patient level export requests.

Matthew Gramigna (Jan 11 2022 at 22:13):

Dan Gottlieb said:

Makes sense - want to propose something in a pull request or issue?

Yep I'll pull request it :)

https://github.com/smart-on-fhir/bulk-import/pull/14

Bin Mao (Jan 12 2022 at 16:15):

Anyone has success with bulk export from IBM servers https://bulk.cluster1-blue-250babbbe4c3000e15508cd07c1d282b-0000.us-east.containers.appdomain.cloud/fhir-server/api/v4
I have received auth error "Response code 401 (Unauthorized)"

Paul Bastide (Jan 12 2022 at 16:16):

Bin Mao said:

Anyone has success with bulk export from IBM servers https://bulk.cluster1-blue-250babbbe4c3000e15508cd07c1d282b-0000.us-east.containers.appdomain.cloud/fhir-server/api/v4
I have received auth error "Response code 401 (Unauthorized)"

Yes - but it's my server...

are you using the bulk-data-client?

Paul Bastide (Jan 12 2022 at 16:16):

fhirUrl: "https://bulk.cluster1-blue-250babbbe4c3000e15508cd07c1d282b-0000.us-east.containers.appdomain.cloud/fhir-server/api/v4",

    /**
     * The Bulk Data server token URL ("none" for open servers)
     * auth/realms/demo/.well-known/openid-configuration
     * https://bulk.cluster1-blue-250babbbe4c3000e15508cd07c1d282b-0000.us-east.containers.appdomain.cloud/auth/realms/test/.well-known/openid-configuration
     */
    tokenUrl: "https://bulk.cluster1-blue-250babbbe4c3000e15508cd07c1d282b-0000.us-east.containers.appdomain.cloud/auth/realms/test/protocol/openid-connect/token",

    /**
     * The private key (JWK) used to sign authentication tokens. This is not
     * needed for open servers
     */
    privateKey: {
        "kty": "RSA",
        "alg": "RS384",
        "n": "wYrXh4-wDtTS7tPVxNdl0mAyCd_IEUECR1Ew91b7BQYwow2at4L8PEQ1bRaGSxWATuviaD0Y9UhLzJ2DPMRDUzbao1evCkrKs-yY1ox1xPeNl8cE9RMTZIcGcAS4r8O_DlM4z-yNhUENi5vsY1qPPfde28FObAH3wr7e63viU-LUpJxJmTb5dSGWIearyDg9H8wZpQAR9k2GrZxqqS17mzEiEqtxUpiw26ABShqJQ4xgOAQoiE7WV3yvf36AqpcO2WtwwS6xwcLw55oZzou3MT2aGZPmNZkdyisGOU9A01REwrwxw6TVbabrb2kIINfDE-Wyes2ZocTJb_zNUZN5xw",
        "e": "AQAB",
        "d": "Bq-dMnmsQ1bm0olO4TXvtozMLbslVjNAuOX9Iw5GLa5BD-Dwb1Z_EDXrApG8oetkO1W6xI8XxaFxvOfUGM1O5hkHBI2K5nge-Ig7322Kw_spUQz3BuBZ0yc2-bewCaJhf0UwuT4axXex2BjS2bvPJvzcsgrDgseun3Ooj8n_z5X_DreQBSrKNpns6MG-9FQhkTDLt-aaYkrL0njXN3LnxhuvMjrCUEy8jCLaDHpPdiThTxGlIu4z5mw_RraMIU3uesXDJQHuaKN6j7Lqwy0qlnmEy5WchTikAyfAwLWSMAS7GL16SX9Wxqk3WCNbSMlOTy0NdBvdjKIH5akYJjUfQQ",
        "p": "5xfttF1T_iP8TanFDewG7Xln9dz415nj89W6zDx3FxDMzbgQscQTDPdlYuMQaoZKgZ6RmDTNP0YxMilfhIs17sJ7ORaKF6ssraS_CEuBnS65zwzAB6iT5mguX8ZnPF_Qbob2pTKXRxUkXHEYJfVZnbRFvQ9APBabtEg9ABqTaL0",
        "q": "1mbYVkg7s7Jfr0omLBlbKQ4m-hdetxf7bSjEexzeSGIy4bTXvzX6sHUIKrC5KnKW9pOvmcJkxnzsxHbuuEt4q1FB3cxxjvFJAqz6c2xuw6lmefbYYWWKDVecUkT2Kc_n5P4xwg2z-GjLJs_gU64jyfSArdNua_XzMwqwq9LmHtM",
        "dp": "ifXyFexlmWI4XNEOcCpJVHpXQyOBd41K1iXxl749RorkCahqZwXsbaBAadGu2jmDv3A_8UMMiUrJUe37NTC6qOh4EfPPyyOIz717wmL5ZTIhAWfWOHw-l534mXrj5No6n9F469SRFYGcrIdj6D1aG9kkjSLOsVC58d3ydN5oxG0",
        "dq": "0vXNJlDa1bzMo6jdGIU2ipYPSgNWweeKEGWNtum32hcto6KSquVNLvVovMC44Yhw_Fxi63M4P4nKWqH_0D0Kld9VZQ12K0VFJqnXoVzvO_ziBV4amPMVPH2ZJeYPJSMaNNrdUOi0zdcnFaBzRUNSmbPILcGdpAMUcoOxRNA2d9s",
        "qi": "L-GcGJ5s6XGofwfFE0HRdGg6UEwDuHUJo6hwksG9nc-AqLcNIvJ15Q90dfaqRN1UUza2zYiy7W1I3Vq4Z6OHguXQmdDHTCqCrGJLprvm9x473y2I55MVJ2JsjwPJePVNrJB_KF0zDYY6uYSSL8nTxLmXLBhxvLpKjBb3lkLDpBg",
        "key_ops": [
            "sign"
        ],
        "ext": true,
        "kid": "6cf70879258f9c656bb7ccc65802d099"
    },

    /**
     * This is not needed for open servers
     */
    clientId: "infernoBulk",

Paul Bastide (Jan 12 2022 at 16:16):

These are the key settings

Bin Mao (Jan 12 2022 at 16:24):

Thank you, I was using the bulk-data-client. Guess I just need the key and try it again.
Couldn't find the key in the server sign up sheet

Paul Bastide (Jan 12 2022 at 16:25):

Oh we are using the defaults for https://bulk-data.smarthealthit.org/keys/RS384.private.json

Bin Mao (Jan 12 2022 at 16:43):

Paul Bastide said:

Oh we are using the defaults for https://bulk-data.smarthealthit.org/keys/RS384.private.json

Hi @Paul Bastide I still get the following key error from bulk data client
Are you testing with this client https://github.com/smart-on-fhir/bulk-data-client
or this one https://github.com/smart-on-fhir/sample-apps-stu3/tree/master/fhir-downloader

request: GET https://bulk.cluster1-blue-250babbbe4c3000e15508cd07c1d282b-0000.us-east.containers.appdomain.cloud/fhir-server/api/v4/Patient/$export
request headers: {
  'user-agent': 'Bulk Data Client <https://github.com/smart-on-fhir/bulk-data-client>',
  accept: 'application/fhir+json',
  prefer: 'respond-async',
  authorization: 'bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJBVVVlWEN5TnFmU2QzcnZMNzNiUTJpd1p5XzE0SXV3X2NOYnE4OGgxa2JVIn0.eyJleHAiOjE2NDIwMDU2NDksImlhdCI6MTY0MjAwNTM0OSwianRpIjoiOWIyNGUwNGQtZjkwMi00ZTg2LWE2NDctYWRiNzc1OGIwNjI5IiwiaXNzIjoiaHR0cHM6Ly9idWxrLmNsdXN0ZXIxLWJsdWUtMjUwYmFiYmJlNGMzMDAwZTE1NTA4Y2QwN2MxZDI4MmItMDAwMC51cy1lYXN0LmNvbnRhaW5lcnMuYXBwZG9tYWluLmNsb3VkL2F1dGgvcmVhbG1zL3Rlc3QiLCJhdWQiOiJodHRwczovL2J1bGsuY2x1c3RlcjEtYmx1ZS0yNTBiYWJiYmU0YzMwMDBlMTU1MDhjZDA3YzFkMjgyYi0wMDAwLnVzLWVhc3QuY29udGFpbmVycy5hcHBkb21haW4uY2xvdWQvZmhpci1zZXJ2ZXIvYXBpL3Y0Iiwic3ViIjoiOWYxYjJiNmYtMjg5NS00YmM1LWI5OGYtMGNlMGNmZmZmZmY5IiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiaW5mZXJub0J1bGsiLCJhY3IiOiIxIiwic2NvcGUiOiJsYXVuY2gvZXhwb3J0IHN5c3RlbS8qLnJlYWQiLCJjbGllbnRIb3N0IjoiMTcyLjE3LjE1My4xOTMiLCJjbGllbnRJZCI6ImluZmVybm9CdWxrIiwiY2xpZW50QWRkcmVzcyI6IjE3Mi4xNy4xNTMuMTkzIiwiZ3JvdXAiOlsiZmhpclVzZXIiXX0.cL8B6ov9Qd7i5em5owP6DSvLn_kOw7GT6cYqrVIEJl2_ed5T3sFvtKb6FnaG57-FLEX2CPbXqgz0qhByx3Pi7_h5IjMsWZ_a1EfAJpV2YbRJTxLhoagOrA9If3RWzy-Tk4Sroy_GXOvq3Ho6i2IjJ6VNQagyKKjB91Jn2_hn_9YLeahIHma0mujZnQxmeBCh13tUy7wF1tHH2WScSCKVsFFQ7ZmuzdWD-NsV93LDCql9Eu1zIk5Xnh0oQGY8LtEDJHYRrAdOdaa6r1zB3hedkoP-EZnB8rkJ9GkSL0M776jCr5f23dpZzuqMC61wfTA7A-3WFT6EWjAT-eY_A1K80A',
  'accept-encoding': 'gzip, deflate, br'
}
response: 401 Unauthorized
response headers: {
  date: 'Wed, 12 Jan 2022 16:35:49 GMT',
  'content-length': '0',
  connection: 'close',
  'www-authenticate': 'Bearer realm="MP-JWT", error="invalid_token"',
  'content-language': 'en-US',
  'strict-transport-security': 'max-age=15724800; includeSubDomains'
}

Dan Gottlieb (Jan 12 2022 at 16:45):

@Bin Mao maybe the lower case bearer header bug Paul raised yesterday in https://chat.fhir.org/#narrow/stream/179250-bulk-data/topic/January.202022.20Connectathon/near/267621541 ?

Bin Mao (Jan 12 2022 at 16:48):

Dan Gottlieb said:

Bin Mao maybe the lower case bearer header bug Paul raised yesterday in https://chat.fhir.org/#narrow/stream/179250-bulk-data/topic/January.202022.20Connectathon/near/267621541 ?

Got it :+1:

Bin Mao (Jan 12 2022 at 17:11):

The token seems working fine. The bulk export has completed. The error happens when file download starts. Here is the log. Any thought?

Kick-off started
Got new access token
Kick-off completed
Bulk Data export started
Bulk Data export completed in 32 seconds

Downloading exported files: ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0%
          Downloaded Files: 0 of 14
            FHIR Resources: 0
               Attachments: 0
           Downloaded Size: 0.0 B

Received an error from the server: Response code 403 (Forbidden)

Paul Bastide (Jan 12 2022 at 17:38):

I haven't fully debugged that scenario - we use presigned urls, and I think it's ignoring the presigned url and trying to setup auth

Dan Gottlieb (Jan 12 2022 at 21:12):

For folks participating in the Connectathon track, please add a sentence or two to describing what you worked on and learned to our Track Report Out document in the "notable achievements", "discovered issues", and "now what" sections:

Dan Gottlieb (Jan 12 2022 at 21:12):

https://docs.google.com/document/d/1vodcjzdS-T3nqlKKuW1GtzpUtmHjIJ7I/edit#heading=h.sekyth4wzi9t

Dan Gottlieb (Jan 12 2022 at 21:12):

This will help people who attended other tracks but want to keep on top of what's happening with Bulk Data!

Paul Bastide (Jan 14 2022 at 14:00):

FYI - I opened PR https://github.com/smart-on-fhir/bulk-data-client/pull/4/files it addresses User-Agent, AccessToken and Bearer issues.

Last updated: Apr 12 2022 at 19:14 UTC