FHIR Chat · Bulk Export · ibm

Stream: ibm

Topic: Bulk Export

Mahesh Dabi (Sep 21 2021 at 14:03):

Below are a couple of errors we are getting while testing a section on Bulk in Inferno:

1 -
Request: GET https://fhir.secureit.co.in:9443/fhir-server/api/v4/$bulkdata-status?job=oVWUczflBh_chJhcTfY2Hw

Response:

{
"transactionTime": "2021-09-21T10:53:53.879Z",
"request": "https://fhir.secureit.co.in:9443/fhir-server/api/v4/Group/17b5e38b010-74858f83-7570-45cb-979b-8d00b694a293/$export",
"requiresAccessToken": true,
"output": [
{
"type": "Provenance",
"url": "KTwhDhHknT5k7lLtqB8eUQM2e0LuvIgX7NQSLqbH5_A/Provenance_1.ndjson",
"count": 1
},
{
"type": "AllergyIntolerance",
"url": "KTwhDhHknT5k7lLtqB8eUQM2e0LuvIgX7NQSLqbH5_A/AllergyIntolerance_1.ndjson",
"count": 1
}
],
"error": []
}

Can we get a complete URL in the highlighted section above? The reason is that Inferno further uses the URL above to access the respective resources. In the current situation, when it tries to access the same, it fails because the URL is incomplete.

2 -
After the bulk export was completed, we could not find below resources
Device
Organization
Practitioner

 Is there a way to configure the export of the above resources or do you need to add support for this in IBM FHIR?

Lee Surprenant (Sep 21 2021 at 14:51):

it sounds like you're using file-based "storageProvider" in your fhir-server-config.json
for the file-based export, we don't support proxying the exported files from the server and so we cannot give you a full URL (or if we did, it would be a file: url).
are you able to use aws-s3 / ibm-cos? if you need to keep the data local, you could look at using MinIO which is what our CI pipeline tests

Mahesh Dabi (Sep 22 2021 at 13:46):

Hi Lee

Will you please help us with second point?
2 -
After the bulk export was completed, we could not find below resources
Device
Organization
Practitioner

 Is there a way to configure the export of the above resources or do you need to add support for this in IBM FHIR?

Paul Bastide (Sep 22 2021 at 13:50):

HI Mahesh, a few questions... how many resources are stored in the Device / Organization and Practitioner resource types? a simple FHIR Search can confirm. What was the $export request was it on Patient or a System Level export?

Mahesh Dabi (Sep 23 2021 at 12:59):

1 - $export is on group level:
Request GET https://fhir.secureit.co.in:9443/fhir-server/api/v4/Group/17b5e38b010-74858f83-7570-45cb-979b-8d00b694a293/$export

  Response 202 Accepted

 After the export is completed below is the list of resources exported. If you see here, device, organization, and practitioner resources are missing.

image.png

2 - Device resource details
Request GET https://fhir.secureit.co.in:9443/fhir-server/api/v4/Device

 Response

{
"resourceType": "Bundle",
"id": "cc4382af-9409-4d04-9141-f4a37ec80b91",
"type": "searchset",
"total": 3,
"link": [
{
"relation": "self",
"url": "https://fhir.secureit.co.in:9443/fhir-server/api/v4/Device?_count=10&_page=1"
}
],
"entry": [
{
"id": "17b587c2d01-038ded08-f14a-40b6-b0e3-6182a8cfc404",
"fullUrl": "https://fhir.secureit.co.in:9443/fhir-server/api/v4/Device/17b587c2d01-038ded08-f14a-40b6-b0e3-6182a8cfc404",
"resource": {
"resourceType": "Device",
"id": "17b587c2d01-038ded08-f14a-40b6-b0e3-6182a8cfc404",
"meta": {
"versionId": "1",
"lastUpdated": "2021-08-18T08:58:58.177514Z",
"profile": [
"http://hl7.org/fhir/us/core/StructureDefinition/us-core-implantable-device"
]
},
"udiCarrier": [
{
"deviceIdentifier": "43069338026389",
"carrierHRF": "(01)43069338026389(11)000302(17)250317(10)1134(21)842026117977"
}
],
"status": "active",
"distinctIdentifier": "43069338026389",
"manufactureDate": "2000-03-02T18:33:18-05:00",
"expirationDate": "2025-03-17T19:33:18-04:00",
"lotNumber": "1134",
"serialNumber": "842026117977",
"deviceName": [
{
"name": "Implantable defibrillator, device (physical object)",
"type": "user-friendly-name"
}
],
"type": {
"coding": [
{
"system": "http://snomed.info/sct",
"code": "72506001",
"display": "Implantable defibrillator, device (physical object)"
}
],
"text": "Implantable defibrillator, device (physical object)"
},
"patient": {
"reference": "Patient/17b57f46378-a6944791-e6ad-4f0f-8e37-683f47f24fa1"
}
},
"search": {
"mode": "match",
"score": 1
}
},
{
"id": "17b587c5238-61cb7fd9-0596-4672-b199-22cb2329c1ba",
"fullUrl": "https://fhir.secureit.co.in:9443/fhir-server/api/v4/Device/17b587c5238-61cb7fd9-0596-4672-b199-22cb2329c1ba",
"resource": {
"resourceType": "Device",
"id": "17b587c5238-61cb7fd9-0596-4672-b199-22cb2329c1ba",
"meta": {
"versionId": "1",
"lastUpdated": "2021-08-18T08:59:07.704421Z",
"profile": [
"http://hl7.org/fhir/us/core/StructureDefinition/us-core-implantable-device"
]
},
"udiCarrier": [
{
"deviceIdentifier": "43069338026389",
"carrierAIDC": "iVBORw0KGgoAAAANSUhEUgAAAioAAAB2CAIAAABpgS9BAAADHUlEQVR42u3aXY+iMBiAUbrZ//+XuxcmpunXvkABNedczahAReWZdkxbIee8bVtK6f3zS3tLaXTv6/b5nttHtuMZHaW8JXJvu8/4GNp72+NGxhN5jvPnHn8tInubn4fIfiKjHZ35+fth1bM49g4/dpaOfWrOnMk7X829449/ciOfzcjrFX/FV73f4sfdexWdv8fOvB9G17T4FW/+WkfOw8ufDQBuJz8AyA8A8gMA8gOA/ACA/AAgPwAgPwDIDwDIDwDyA4D8AID8ACA/ACA/AMgPAMgPAPIDAPIDgPwAID8AID8AyA8AyA8A8gMA8gOA/ACA/AAgPwDIDwDIDwDyAwDyA4D8AID8ACA/ACA/AMgPAPIDAPIDgPwAgPwAID8AID8AyA8AyA8A8gOA/ACA/AAgPwAgPwDIDwDIDwDyAwDyA4D8ACA/ACA/AMgPAMgPAPIDAPIDgPwAgPwAID8AyA8AyA8A8gMA8gOA/ACA/AAgPwAgPwDIDwDyAwDyA4D8AID8ACA/ACA/AMgPAMgPAPIDgPw4BQDIDwDyAwDyA4D8AID8ACA/ACA/AMgPAMgPAPIDgPwAgPwAID8AID8AfJuUc3YWADD7AUB+AOACf50CrpBSqm65dJn3dbgDhyjHeWaEKS1bx141pAefAsgPT7rzWpZzboO3d5yfc/39wCHBchbfYGXDPmdIYPYDW/dv+fev3eWm93pa+bDycjy/4p9cwhoNafRzdZTJM9o1pPYo5eajExI/evdkPrX6h/zAstgEO9FtUtmeqkDdDYP7PJPJ4Bylu/mufXaLMip39Zj40bttO3nqQH54XnvZ6oZk19XtGy+FB8bcVme0k//uPHh0q3zID7/fpPlS1fK518LN29nY1UOqgn1Pm3UI+eEHtbOfdrXnqTnH1dOshV/svrQQVtu4h2++8UB7tqOrPccuu8GtRnOa6l/xB/6TtHxIVzx3MyHMfvid2ET+pp53qNpq/kWv97Jetc/gd+RGaQxuPj/62iHN5y7Bo1cns/yqoZkQd1wivL34xskT8O0svgFg9gOA2Q8AyA8Av+MfJjJq5w+06kcAAAAASUVORK5CYII="
}
],
"status": "active",
"distinctIdentifier": "43069338026389",
"manufactureDate": "2000-03-02T18:33:18-05:00",
"expirationDate": "2025-03-17T19:33:18-04:00",
"lotNumber": "1134",
"serialNumber": "842026117977",
"deviceName": [
{
"name": "Implantable defibrillator, device (physical object)",
"type": "user-friendly-name"
}
],
"type": {
"coding": [
{
"system": "http://snomed.info/sct",
"code": "72506001",
"display": "Implantable defibrillator, device (physical object)"
}
],
"text": "Implantable defibrillator, device (physical object)"
},
"patient": {
"reference": "Patient/17b57f46378-a6944791-e6ad-4f0f-8e37-683f47f24fa1"
}
},
"search": {
"mode": "match",
"score": 1
}
},
{
"id": "17b5dec7fde-dda0524d-2256-42f0-96d1-9fe252e36d4a",
"fullUrl": "https://fhir.secureit.co.in:9443/fhir-server/api/v4/Device/17b5dec7fde-dda0524d-2256-42f0-96d1-9fe252e36d4a",
"resource": {
"resourceType": "Device",
"id": "17b5dec7fde-dda0524d-2256-42f0-96d1-9fe252e36d4a",
"meta": {
"versionId": "1",
"lastUpdated": "2021-08-19T10:19:45.502943Z",
"profile": [
"http://hl7.org/fhir/us/core/StructureDefinition/us-core-implantable-device"
]
},
"udiCarrier": [
{
"deviceIdentifier": "43069338026389",
"carrierHRF": "(01)43069338026389(11)000302(17)250317(10)1134(21)842026117977"
}
],
"status": "active",
"distinctIdentifier": "43069338026389",
"manufactureDate": "2000-03-02T18:33:18-05:00",
"expirationDate": "2025-03-17T19:33:18-04:00",
"lotNumber": "1134",
"serialNumber": "842026117977",
"deviceName": [
{
"name": "Implantable defibrillator, device (physical object)",
"type": "user-friendly-name"
}
],
"type": {
"coding": [
{
"system": "http://snomed.info/sct",
"code": "72506001",
"display": "Implantable defibrillator, device (physical object)"
}
],
"text": "Implantable defibrillator, device (physical object)"
},
"patient": {
"reference": "Patient/17b5dc09184-54ce1834-2501-4e85-a3fe-e04223da69cf"
}
},
"search": {
"mode": "match",
"score": 1
}
}
]
}

   3 - Organization resource details
Request GET https://fhir.secureit.co.in:9443/fhir-server/api/v4/Organization

Response: Please see attachment

   4 -  Practitioner resource details
Request GET https://fhir.secureit.co.in:9443/fhir-server/api/v4/Practitioner

Response: Please see attachment Responses.txt

Paul Bastide (Sep 23 2021 at 14:00):

Hi Mahesh, refer to the following snippet, for the Group Export without resource types we default to the list of resources in the Patient Compartment. If you list device, organization, et cetra on the included resources it'll export it.

if (!ExportType.INVALID.equals(exportType)) {
if (ExportType.PATIENT.equals(exportType) || ExportType.GROUP.equals(exportType)) {
if (types != null && !types.isEmpty()) {
export.checkExportPatientResourceTypes(types);
} else {
types = export.addDefaultsForPatientCompartment();
}
} else if ((ExportType.SYSTEM.equals(exportType) )
&& (types == null || types.isEmpty())) {
types = export.defaultResourceTypes();
}

Lee Surprenant (Sep 23 2021 at 14:00):

Currently our Patient and Group export implementation will only export resources in the Patient compartment.
We do support "customizable compartment definitions" but there's currently no support for "chained" inclusion criteria there. See https://github.com/IBM/FHIR/issues/2371 for more info.
Feel free to add a comment there or open a separate issue with what you you'd like to see.

Paul Bastide (Sep 23 2021 at 14:00):

by default, we limit to the patient compartment

Lee Surprenant (Sep 23 2021 at 14:00):

paul just beat me... :-)

Paul Bastide (Sep 23 2021 at 14:01):

I had to check the code :)

Lee Surprenant (Sep 23 2021 at 14:02):

So my thinking on this is that we could support these other resource types by implementing 2371 and then our users could customize what is considered "in the Patient compartment" per their own wants/needs and we wouldn't need to update our export logic

Lee Surprenant (Sep 23 2021 at 14:02):

however, that is pretty low on our priority list right now (and easier said than done)

Mahesh Dabi (Sep 30 2021 at 07:42):

Lee Surprenant said:

Currently our Patient and Group export implementation will only export resources in the Patient compartment.
We do support "customizable compartment definitions" but there's currently no support for "chained" inclusion criteria there. See https://github.com/IBM/FHIR/issues/2371 for more info.
Feel free to add a comment there or open a separate issue with what you you'd like to see.

Hi Lee

We are trying different option for bulk export. We tried Amazon S3 and it worked fine but the issue is that inferno (ONC testing tool) expects security token in bulk export URL. So this option maynot work. We are trying workaround and have two questions

1. If we want to configure HTTPS export instead of file or S3 , what will be bulk export section configuration. Do you have any sample config file to demonstrate that?
2. In Liberty (FHIR Server) can we expose any folder to public so that it can be accessed over HTTP. Basically we want to expose one folder under FHIR Server itself over HTTPS (so that security can be taken care of)

Mahesh

Lee Surprenant (Sep 30 2021 at 10:26):

If we want to configure HTTPS export instead of file or S3 , what will be bulk export section configuration. Do you have any sample config file to demonstrate that?

The "https" provider only supports import at the moment.
However, note that the generated s3 download urls ARE HTTPS.

inferno (ONC testing tool) expects security token in bulk export URL

The S3 feature uses https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html to create a URL that has the "access token" embedded in the url in the form of Credentials + Signature. Basically the fhir server has read access to the private bucket/object and then temporarilly extends that access to the end user in the form of a (signed) download url with a controllable expiry.

What is it that inferno doesn't think is valid about this?! If you can be clear about it, I can help you push back on this.

Mahesh Dabi (Oct 08 2021 at 07:23):

Hi Lee

BDE-09: Bulk Data Server SHALL restrict bulk data file access with access token

Scenario 1 - When token is provided, we get response after validating the token
Request:
GET 200 https://fhir.secureit.co.in:9443/fhir-server/api/v4/$bulkdata-status?job=O5sfGj3l4ZFOO_G0TdwHvQ

OAuth Token Provided: YES

Response:
{
"transactionTime": "2021-10-08T06:58:34.668Z",
"request": "https://fhir.secureit.co.in:9443/fhir-server/api/v4/Group/17c5ea3c53d-9ae67467-60d2-4f1d-85f6-6808d680589e/$export",
"requiresAccessToken": true,
"output": [
{
"type": "AllergyIntolerance",
"url": "https://1secureit.s3.amazonaws.com/fhir/1secureit/iLUQQ4xsv0xbYoQ-auymO567ndIPa5EsRjR40AtZEz8/AllergyIntolerance_1.ndjson",
"count": 1
},
{
"type": "Group",
"url": "https://1secureit.s3.amazonaws.com/fhir/1secureit/iLUQQ4xsv0xbYoQ-auymO567ndIPa5EsRjR40AtZEz8/Group_1.ndjson",
"count": 1
},
{
"type": "Patient",
"url": "https://1secureit.s3.amazonaws.com/fhir/1secureit/iLUQQ4xsv0xbYoQ-auymO567ndIPa5EsRjR40AtZEz8/Patient_1.ndjson",
"count": 1
}
],
"error": []
}

Scenario 2 - From the above response, Inferno then picks up individual URLs and does a GET as shown below. In one such scenario, it sends no token
and it expects 400 or 401, whereas it receives a 200.

Since we are using AWS bucket, it gives us a direct URL and there is no way it will validate the OAuth token.

Before using AWS, we tried using file as storage provider, but it provided partial path ("url" : "iLUQQ4xsv0xbYoQ-auymO567ndIPa5EsRjR40AtZEz8/AllergyIntolerance_1.ndjson")
and not the complete URL. If it could provide a complete HTTP/HTTPS path, then this may solve the problem.

Could you please help us with this?
Also, is there a way where S3 URL could be routed through our oAUTH server?

Request:
GET 200 https://1secureit.s3.amazonaws.com/fhir/1secureit/iLUQQ4xsv0xbYoQ-auymO567ndIPa5EsRjR40AtZEz8/AllergyIntolerance_1.ndjson

OAuth Token Provided: NO

Response:
{
"resourceType": "AllergyIntolerance",
"id": "17c5ea10448-672d34c4-5131-4d90-a488-fae9151fb8f8",
"meta": {
"versionId": "1",
"lastUpdated": "2021-10-08T06:39:43.178Z",
"profile": [
"http://hl7.org/fhir/us/core/StructureDefinition/us-core-allergyintolerance"
]
},
"clinicalStatus": {
"coding": [
{
"system": "http://terminology.hl7.org/CodeSystem/allergyintolerance-clinical",
"code": "inactive"
}
]
},
"code": {
"coding": [
{
"system": "http://snomed.info/sct",
"code": "300916003",
"display": "Latex allergy"
}
],
"text": "Latex allergy"
},
"patient": {
"reference": "Patient/17c5e5cabb3-19b86ee6-35be-4157-b4a0-fb9b6ad28341"
}
}

Mahesh Dabi (Oct 11 2021 at 10:32):

Based on the above description and requirement from Inferno, below could be 2 options that could be possible solutions:

1 - Before using AWS bucket, we tried using file as storage provider, but it provided partial path ("URL" : "iLUQQ4xsv0xbYoQ-auymO567ndIPa5EsRjR40AtZEz8/AllergyIntolerance_1.ndjson")
and not the complete URL. If it could provide a complete HTTP/HTTPS path, then this may solve the problem.

2 - When using AWS bucket we get a complete URL ("URL": "https://1secureit.s3.amazonaws.com/fhir/1secureit/iLUQQ4xsv0xbYoQ-auymO567ndIPa5EsRjR40AtZEz8/Patient_1.ndjson"). From here on Inferno uses this URL to fetch data.
Instead of this S3 URL, the IBM FHIR server could introduce a proxy URL, and internally it could use this S3 URL to fetch data. Doing this the OAuth server will be used and the token will be validated before fetching the fhir data from AWS S3.
This behavior can be kept configurable in fhirserverconfig.

Please let us know your thought

Lee Surprenant (Oct 11 2021 at 12:15):

We purposefully keep the FHIR server out of the path for downloading these large files as, currently, those will compete with resources with the threads serving the REST API.
At some point, I'll have a look at the inferno tests, but my first question there is "why does it require token-based auth when the spec allows for either/or?"

I'm not totally opposed to providing an option to proxy the export files, but I do think its better to keep that out of the liberty-based FHIR server app. Maybe introduce a separate component with the same auth strategy?
I'd definitely encourage you to design/develop something that works for your use case and then share those learnings.

Lee Surprenant (Oct 11 2021 at 12:26):

I did like where you were going with this earlier line of thought:

is there a way where S3 URL could be routed through our oAUTH server?

S3 (and IBM Object Storage) provide a lot of options for securing the exported files. You can find an overview of those at https://aws.amazon.com/premiumsupport/knowledge-center/secure-s3-resources/.
We chose to make presigned URLs against private objects our default, but its definitely possible to disable the presigning from your fhir-server-config and use IAM access policies (which will reject unauthenticated/unauthorized access) instead.

Lee Surprenant (Oct 11 2021 at 12:30):

The challenge there will be that the next version of the bulk export spec will indicate that, when requiresAccessToken==true, the security model for the exported files should be the same as the one for requesting the export (using SMART Backend Services):
https://chat.fhir.org/#narrow/stream/179250-bulk-data/topic/Attachments.20and.20.22additional.20authentication.20credentials.22/near/246161556

Lee Surprenant (Oct 11 2021 at 12:31):

So, it probably depends what you're trying to do (just pass some inferno tests or position yourself for future compliance)

Mahesh Dabi (Oct 12 2021 at 07:29):

One of the solutions we discussed was around using FILE as a storage provider. Below is an example response after the bulk export is completed:
{
"transactionTime": "2021-09-21T10:53:53.879Z",
"request": "https://fhir.secureit.co.in:9443/fhir-server/api/v4/Group/17b5e38b010-74858f83-7570-45cb-979b-8d00b694a293/$export",
"requiresAccessToken": true,
"output": [{
"type": "Provenance",
"url": "KTwhDhHknT5k7lLtqB8eUQM2e0LuvIgX7NQSLqbH5_A/Provenance_1.ndjson",
"count": 1
}],
"error": []
}

If you see the URL here is the reference URL and not complete. The exported files are located under fhirserver on liberty. If you could provide a complete HTTP/HTTPS URL instead of a reference URL, then this URL will inherit the authentication system configured for the FHIR server which means if the SMART is enabled it will validate the supplied token and proceed accordingly.

If you think this could be a probable solution then this could involve fewer changes. The default behavior could be as is, and having a complete URL in the response could be based on a configuration in fhirserverconfig. Again you are the correct person to judge on how much effort and changes would be involved. This URL can be exposed by doing some configuration in Liberty by publishing the folder. I donot know much about how to expose one additional folder in FHIR where these files will reside.

If we are able to achieve this then we may not need to disturb the current behavior in the S3 storage provider.

As you mentioned, "when requiresAccessToken==true, the security model for the exported files should be the same as the one for requesting the export (using SMART Backend Services)". Could this be one of the approaches to get there?

Please let us know your thoughts.

Mahesh Dabi (Oct 16 2021 at 07:19):

Hi

Have you got a chance to review suggestion above? Any ideas you can suggest whether this will work or not? Will appreciate your help.

Thanks
Mahesh

Paul Bastide (Oct 18 2021 at 18:22):

Please open an enhancement for appending a URL to the FILE Export

Paul Bastide (Oct 18 2021 at 18:22):

We'll look at implementing it as a configuration option

Mahesh Dabi (Nov 03 2021 at 04:49):

we managed to use S3 and satisfy what is needed by Inferno.

Inferno has one test case related to Bulk Export with respect to a definition in metadata as mentioned below.

BDE-02: Bulk Data Server declares support for 'group-export' operation in CapabilityStatement (http://hl7.org/fhir/uv/bulkdata/OperationDefinition-group-export.html)

Below is a snapshot of IBM FHIR metadata with respect to type:group
{
"type": "Group",
"profile": "http://hl7.org/fhir/profiles/Group",
"operation": [{
"name": "export",
"definition": "http://hl7.org/fhir/uv/bulkdata/OperationDefinition/group-export|1.0.1"
}]
}

Here, the operation name should be "group-export".

Below is a snapshot of the metadata from the FHIR server that Inferno uses:
{
"type": "Group",
"operation": [
{
"extension": [
{
"url": "http://hl7.org/fhir/StructureDefinition/capabilitystatement-expectation",
"valueCode": "SHOULD"
}
],
"name": "group-export",
"definition": "http://hl7.org/fhir/uv/bulkdata/OperationDefinition/group-export"
}
]
}

Please let us know your thoughts.

Paul Bastide (Nov 03 2021 at 13:26):

HI Mahesh, you should configure a persistence data store. You are using the in memory datastore

Paul Bastide (Nov 03 2021 at 13:27):

https://github.com/IBM/FHIR/blob/main/fhir-server/liberty-config/configDropins/disabled/postgres/bulkdata.xml

Paul Bastide (Nov 03 2021 at 13:27):

Here is an example. Line 43 has the datasource for the bulkdata Server

Paul Bastide (Nov 03 2021 at 13:27):

note you also need to support max_prepared_transactions on postgres

Paul Bastide (Nov 03 2021 at 13:28):

BulkData uses JBatch which is an XA transaction across two datasources.

Lee Surprenant (Nov 03 2021 at 14:18):

I think Paul's answer was intended for the Delete Job topic.
@Paul Bastide any thoughts on the operation name we're including in our CapabilityStatement? I think I agree with Mahesh that this should say export and not group-export...

Paul Bastide (Nov 03 2021 at 14:23):

Yes. that's right on my answer.
I think that change is OK, I think we hand crafted these quite a while ago

Sagar Sarvankar (Nov 03 2021 at 14:24):

Hello Lee, IBM FHIR gives "name": "export" in metadata but it should be "name": "group-export"

Lee Surprenant (Nov 03 2021 at 14:24):

oh, on that i disagree

Lee Surprenant (Nov 03 2021 at 14:24):

the name in the CapabilityStatement should match that which it is invoked on

Lee Surprenant (Nov 03 2021 at 14:25):

i think export is right

Lee Surprenant (Nov 03 2021 at 14:25):

if thats what we have now, then i don't think we need a change

Lee Surprenant (Nov 03 2021 at 14:49):

I've raised this at https://chat.fhir.org/#narrow/stream/179166-implementers/topic/CapStmnt.2Econformance.2Erest.2Eoperation.2Ename.20v.20OpDef.2Ecode as I think its a bug in the bulkdata spec...I thought we'd already reported this by maybe not

Sagar Sarvankar (Nov 03 2021 at 15:07):

Lee, Sorry I do not see anything in that chat link that you sent.

Lee Surprenant (Nov 03 2021 at 15:17):

sorry, i fixed the title of it and that breaks the link in zulip :-(
try this: https://chat.fhir.org/#narrow/stream/179166-implementers/topic/CapStmnt.2Erest.2Eoperation.2Ename.20v.20OpDef.2Ecode

Sagar Sarvankar (Nov 05 2021 at 04:47):

The 'definition' here (IBM FHIR metadata) in the example below (http://hl7.org/fhir/uv/bulkdata/OperationDefinition/group-export|1.0.1) contains URL and version, whereas there is another field 'version' that can contain the version https://www.hl7.org/fhir/operationdefinition.html . We can have the version value in its own field instead.

IBM FHIR metadata
{
"type": "Group",
"profile": "http://hl7.org/fhir/profiles/Group",
"operation": [{
"name": "export",
"definition": "http://hl7.org/fhir/uv/bulkdata/OperationDefinition/group-export|1.0.1"
}]
}

Please let us know your thoughts.

Lee Surprenant (Nov 05 2021 at 11:20):

URL + "|" + version is correct.

Sagar Sarvankar (Jan 24 2022 at 12:07):

Hello @Lee Surprenant
I hope you are doing well.
This is regarding bulk export. When we do a bulk export all resources like patient, immunization, allergies etc are exported, except the below ones:

  Device 
  Organization 
  Practitioner

Could you please tell us how to export above resources also?

Thank you,
Sagar

Paul Bastide (Jan 24 2022 at 16:32):

HI Sagar, can you grab the joblogs for that particular job? And the messages.log? If you can share (after checking for any sensitive data), that would help greatly. If Device, Organization, Practitioner are not included, it seems very strange

Lee Surprenant (Jan 24 2022 at 16:42):

@Sagar Sarvankar which $export operation are you invoking? If its Patient/$export I think our implementation of that one will only export resources that are in a "patient compartment" as defined at https://www.hl7.org/fhir/compartmentdefinition-patient.html

Lee Surprenant (Jan 24 2022 at 16:44):

the spec is pretty loose in this regard...it says this:

For Patient- and Group-level requests requests, the Patient Compartment SHOULD be used as a point of reference for recommended resources to be returned. However, other resources outside of the patient compartment that are helpful in interpreting the patient data (such as Organization and Practitioner) may also be returned.

Sagar Sarvankar (Jan 24 2022 at 17:02):

@Lee Surprenant We are using Group export https://fhirserver.justtest.in:9443/fhir-server/api/v4/Group/17e8b79a2cf-20d0f360-e1e9-407a-9340-d3843a152897/$export

Could you please help us configure compartment to retrieve Device, Organization and Practitioner on IBM FHIR?

Paul Bastide (Jan 24 2022 at 17:56):

Hi Sagar, there is a limitation on the CompartmentDefinition which is static in this case.

Paul Bastide (Jan 24 2022 at 17:56):

public void checkExportPatientResourceTypes(List<String> resourceTypes) throws FHIROperationException {
        boolean valid = false;
        try {
            // Also Check to see if the Export is valid for the Compartment.
            List<String> allCompartmentResourceTypes = CompartmentUtil.getCompartmentResourceTypes("Patient");
            Set<String> supportedResourceTypes = getSupportedResourceTypes();
            List<String> tmp = resourceTypes.stream()
                                        .filter(item -> allCompartmentResourceTypes.contains(item) && supportedResourceTypes.contains(item))
                                        .collect(Collectors.toList());
            valid = tmp != null && !tmp.isEmpty();
        } catch (Exception e) {
            // No Operation intentionally
        }
        if (!valid) {
            throw buildOperationException("Resource Type outside of the Patient Compartment in Export", IssueType.INVALID);
        }
    }

Paul Bastide (Jan 24 2022 at 17:56):

it's checking the GroupExportOperation is only including valid resources. Since the resources are outside the Compartment, they are filtered/invalid.

Sagar Sarvankar (Jan 25 2022 at 13:50):

Ok, but is there a way in some configuration in IBM FHIR to include the missing resources in the bulk export operation?
There is one case in Inferno where it expects export of Device, Organization and Practitioner in Bulk Export Scenario.
This is the last step in the Inferno tool, and on passing this test case, we will be through with all the test scenarios in the Inferno tool.

Lee Surprenant (Jan 25 2022 at 13:51):

thats a funny test given how loose the spec is in this regard

Lee Surprenant (Jan 25 2022 at 13:52):

we don't support it today

Lee Surprenant (Jan 25 2022 at 13:52):

nor does the spec require it in my opinion

Lee Surprenant (Jan 25 2022 at 13:52):

however, it may be useful in real life, so I'd suggest opening an enhancement request for it

Lee Surprenant (Jan 25 2022 at 13:53):

of note, we recently did something similar for the $everything operation: https://github.com/IBM/FHIR/issues/2337

Lee Surprenant (Jan 25 2022 at 13:53):

so i think it would be very similar to that

Sagar Sarvankar (Jan 25 2022 at 13:54):

@Lee Surprenant , may be I do not understand it completely, but Device, Organization and Practitioner should be part of bulk export data like other resources?

Lee Surprenant (Jan 25 2022 at 13:55):

why?

Lee Surprenant (Jan 25 2022 at 13:55):

did you see this above, pulled right from the spec?

Lee Surprenant said:

the spec is pretty loose in this regard...it says this:

For Patient- and Group-level requests requests, the Patient Compartment SHOULD be used as a point of reference for recommended resources to be returned. However, other resources outside of the patient compartment that are helpful in interpreting the patient data (such as Organization and Practitioner) may also be returned.

Lee Surprenant (Jan 25 2022 at 13:55):

it says they "may" be returned

Lee Surprenant (Jan 25 2022 at 13:56):

our impl does not return them at the moment

Sagar Sarvankar (Jan 25 2022 at 15:07):

Yes, thank you @Lee Surprenant

So, since this is required in our case, is there a way to include this as part of the Bulk Data Export? Like including this in custom compartment or some other way?

I also checked in https://github.com/IBM/FHIR/issues/2337 , there is development happening around $everything. Will that be helpful here?

Please help.

Regards,
Sagar

Lee Surprenant (Jan 25 2022 at 17:37):

Will that be helpful here?

Just as an example of how to implement it (and potentially a source for some shared utilities). That why above I said "I'd suggest opening an enhancement request for it"

Sagar Sarvankar (Jan 26 2022 at 04:16):

Hello @Lee Surprenant , @Paul Bastide

The enhancement request for this requirement is created at https://github.com/IBM/FHIR/issues/3247
I hope this request will considered in the next release.

Thank you for your support.

Regards,
Sagar

Last updated: Apr 12 2022 at 19:14 UTC