Stream: ibm
Topic: Bulk import and search issue
Teva Riou (Jan 22 2021 at 19:12):
Hello! I noticed that when I import a resource via bulk import, I am unable to perform a search on this resource using search parameters. If I create this exact same resource via a simple http request, it works without issue. Is there a supplementary step I need to do in order to make it work ?
Lee Surprenant (Jan 22 2021 at 19:14):
no, resources should be indexed properly when imported via bulk import. if you're confident in what your seeing, maybe you can open a bug for it (with info like what resource types and which queries)?
Teva Riou (Jan 22 2021 at 21:09):
It's weird. For instance, take a resource Patient. I can find my resource with birthday or name but not with gender or a reference. I've opened an issue.
Lee Surprenant (Jan 22 2021 at 21:19):
Thank you for the report Teva, we'll have a look
Paul Bastide (Jan 22 2021 at 21:21):
I'm pretty sure it's a token problem - it might be the original url that is failing
Paul Bastide (Jan 22 2021 at 21:21):
thanks for the issue/bug
Paul Bastide (Jan 22 2021 at 21:42):
<property name="incomingUrl" value=""/> <--
Paul Bastide (Jan 22 2021 at 21:42):
that's the problem
Paul Bastide (Jan 22 2021 at 21:42):
small fix coming
Paul Bastide (Jan 22 2021 at 21:48):
I oversold the fix. still digging in...
Lee Surprenant (Feb 09 2021 at 02:21):
We believe that we've addressed this issue with the 4.5.5 release that we just tagged. Please let us know if you're able to confirm the fix on your end.
Mahesh Dabi (Mar 01 2022 at 18:44):
Hi Trying to import this file, it is giving some errors. Medicationstatement is giving error. Even though I deleted those tow manually it is successful but can't see data in DB. Here we are trying to import all resources from single bundle.
Mahesh
Mahesh Dabi (Mar 01 2022 at 18:44):
BTW we are using 4.10.1
Mahesh Dabi (Mar 09 2022 at 06:09):
Can anybody please help?
Mahesh
Lee Surprenant (Mar 09 2022 at 12:22):
Hi Mahesh, I'm afraid its not so clear how to help you. Are there any errors in the logs (either the server logs or the job logs from the import)?
Lee Surprenant (Mar 09 2022 at 12:24):
In general, we've been pointing folks at the batch/transaction endpoint for ingestion lately because our import is single-threaded and so its not really any faster. We have some utilities in our fhir-bucket project for converting NDJSON to bundles and uploading those in parallel, but no great docs on that yet.
Lee Surprenant (Mar 09 2022 at 12:26):
net: bulk import is definitely an interesting feature, but the current implementation (that was originally done for a connectathon a while back) has proved problematic...we're patching some of those issues for our next release but we probably need to rethink the whole feature in the coming months
Mahesh Dabi (Mar 12 2022 at 11:12):
Hi Lee
Here is log and ndjson file which we are trying to import along with job test.zip log. Log shows that import is successful but somehow I am unable to see any patient created. We are trying to import bundle with one patient summary containing all resources for that patient. Not sure whether it is supported or not. Individual resource import e.g. Allergies.ndjson, patient.ndjson works fine. So not sure whether it can import all resources of single patient in one go. test.zip
Here is the import request, please see the type
{
"resourceType": "Parameters",
"id": "30321130-5032-49fb-be54-9b8b82b2445a",
"parameter": [
{
"name": "inputSource",
"valueUri": "https://44.197.187.245:9443/fhir-server"
},
{
"name": "inputFormat",
"valueString": "application/fhir+ndjson"
},
{
"name": "input",
"part": [
{
"name": "type",
"valueString": "Package"
},
{
"name": "url",
"valueUrl": "test.ndjson"
}
]
},
{
"name": "storageDetail",
"valueString": "file"
}
]
}
Mahesh Dabi (Mar 12 2022 at 11:14):
I am not sure the resource type I am mentioning in request is correct or not.
"part": [
{
"name": "type",
"valueString": "Package"
},
Mahesh Dabi (Mar 14 2022 at 02:39):
In short I have data available in CCDA (visit note fomats) , what will be best way to import to IBM FHIR Server. I am converting CCDA to FHIR package and trying to import.
Mahesh Dabi (Mar 14 2022 at 05:04):
BTW if you look at my NDJSON file it is a package and that is imported as pachange. If I see
https://baseurl:9443/fhir-server/api/v4/Bundle I can get the whole bundle. So I think it imports and stores this as bundle but not as separate resources by splitting them. Is there any way to split them while import? Do I need to import as anything else other than Package or Bundle?
Lee Surprenant (Mar 14 2022 at 11:20):
thanks, Mahesh, that makes sense.
Is there any way to split them while import?
No, not with the current implementation at least
Do I need to import as anything else other than Package or Bundle?
Import is performed on an NDJSON file that contains the type of resource you're trying to import.
In the next release, we will more strictly enforce that the NDJSON contains only a single type and that this type matches what was passed on the call to invoke import: https://github.com/IBM/FHIR/issues/3180
Lee Surprenant (Mar 14 2022 at 11:26):
Mahesh Dabi said:
I am not sure the resource type I am mentioning in request is correct or not.
"part": [
{
"name": "type",
"valueString": "Package"
},
No, that doesn't make sense to me. The "type" should be a resource type. In main (and our next release) I think we'd reject this request as invalid because Package is not a valid resource type.
Lee Surprenant (Mar 14 2022 at 11:29):
You have a few options, but I think it might be easiest for you to convert your Bundle resources into 'transaction' bundles and post those to the base url (fhir-server/api/v4). That will have your desired behavior of splitting the bundle into its individual components, storing them individually, and making those searchable.
Lee Surprenant (Mar 14 2022 at 11:32):
The downside of this is if you are actually trying to find the CCDA as a whole, it can be tricky to put them back together. We have a fledgling implementation of the $document operation for doing just that, but be mindful that this is a bit experimental.
Last updated: Apr 12 2022 at 19:14 UTC
