Stream: ibm
Topic: bulk import with minio
Simon Vadée (Jan 05 2021 at 13:09):
Hey guys, is there an example of bulk import with minio somewhere in the codebase ? I get an error "Error while processing the $import request" (which is not very helpful). I'm running the server in docker using this bulkdata config:
"bulkdata": {
"applicationName": "fhir-bulkimportexport-webapp",
"moduleName": "fhir-bulkimportexport.war",
"validBaseUrlsDisabled": true,
"implementation_type": "cos",
"jobParameters": {
"cos.bucket.name": "fhirbulkdata",
"cos.location": "us",
"cos.endpoint.internal": "http://minio:9000",
"cos.endpoint.external": "http://minio:9000",
"cos.credential.ibm": "N",
"cos.api.key": "minio",
"cos.srvinst.id": "change-password"
}
}
And my HTTP request looks as follows:
curl -k -v -X POST -u "fhiruser:change-password" -H 'Content-Type: application/fhir+json' 'https://localhost:9443/fhir-server/api/v4/$import' -d '{
"resourceType": "Parameters",
"id": "30321130-5032-49fb-be54-9b8b82b2445a",
"parameter": [
{
"name": "inputFormat",
"valueString": "application/fhir+ndjson"
},
{
"name": "inputSource",
"valueUri": "https://localhost:9443/whatever"
},
{
"name": "input",
"part": [
{
"name": "type",
"valueString": "Patient"
},
{
"name": "url",
"valueUrl": "http://minio:9000/fhirbulkdata/test-import.ndjson"
}
]
},
{
"name": "storageDetail",
"valueString": "aws-s3"
}
]
}'
Simon Vadée (Jan 05 2021 at 13:10):
is it because minio is using http instead of https (does not have a self-signed certificate?)
Lee Surprenant (Jan 05 2021 at 13:21):
Hi Simon, I assume that error message is in the OperationOutcome from either the $import request or the $bulkdata-status request...which one?
Simon Vadée (Jan 05 2021 at 13:30):
from the $import one
Lee Surprenant (Jan 05 2021 at 13:30):
ok, then it seems like its not even able to submit the job
Lee Surprenant (Jan 05 2021 at 13:31):
can you check the server logs (messages.log) for more information?
Lee Surprenant (Jan 05 2021 at 13:33):
wrt examples, I believe our CI executes both $import and $export operations against minIO on each run. the specific tests are at fhir-server-test/src/test/java/com/ibm/fhir/server/test/bulkdata but the more interesting bit is the setup which uses docker-compose to create the minio cluster and a configured fhir server that uses it
Lee Surprenant (Jan 05 2021 at 13:36):
that test automation is at build/docker and the source for the fhir-server-config that uses it is at https://github.com/IBM/FHIR/blob/master/fhir-server/liberty-config/config/default/fhir-server-config-db2.json#L129-L140
Simon Vadée (Jan 05 2021 at 13:45):
I can't find messages.log anywhere (I run on 4.5.3)
Simon Vadée (Jan 05 2021 at 13:46):
yes that's what I'm looking at! there are very few differences with my local setup (i'll try using the exact same config)
Simon Vadée (Jan 05 2021 at 14:14):
found the log file, here's the stacktrace:
[1/5/21 14:11:57:956 UTC] 0000002d com.ibm.fhir.server.resources.FHIRResource E Error while processing the $import request
com.ibm.fhir.exception.FHIROperationException: Error while processing the $import request [probeId=ac-16-0-3-33dd54ae-3003-4db0-9277-cc41e4ad14d6]
at com.ibm.fhir.operation.bulkdata.processor.impl.CosExportImpl.importBulkData(CosExportImpl.java:196)
at com.ibm.fhir.operation.bulkdata.ImportOperation.doInvoke(ImportOperation.java:66)
at com.ibm.fhir.server.operation.spi.AbstractOperation.invoke(AbstractOperation.java:54)
at com.ibm.fhir.server.util.FHIRRestHelper.doInvoke(FHIRRestHelper.java:1038)
at com.ibm.fhir.server.resources.Operation.invoke(Operation.java:127)
at com.ibm.fhir.server.resources.Operation$Proxy$_$$_WeldClientProxy.invoke(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.ibm.ws.jaxrs20.cdi.component.JaxRsFactoryImplicitBeanCDICustomizer.serviceInvoke(JaxRsFactoryImplicitBeanCDICustomizer.java:339)
at com.ibm.ws.jaxrs20.server.LibertyJaxRsServerFactoryBean.performInvocation(LibertyJaxRsServerFactoryBean.java:641)
at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.performInvocation(LibertyJaxRsInvoker.java:160)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.invoke(LibertyJaxRsInvoker.java:273)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:210)
at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.invoke(LibertyJaxRsInvoker.java:444)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:112)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:61)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:99)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:318)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:123)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:273)
at com.ibm.ws.jaxrs20.endpoint.AbstractJaxRsWebEndpoint.invoke(AbstractJaxRsWebEndpoint.java:136)
at com.ibm.websphere.jaxrs.server.IBMRestServlet.handleRequest(IBMRestServlet.java:146)
at com.ibm.websphere.jaxrs.server.IBMRestServlet.doPost(IBMRestServlet.java:104)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
at com.ibm.websphere.jaxrs.server.IBMRestServlet.service(IBMRestServlet.java:96)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:729)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:426)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:182)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:93)
at com.ibm.fhir.server.filter.rest.FHIRRestServletFilter.doFilter(FHIRRestServletFilter.java:139)
at javax.servlet.http.HttpFilter.doFilter(HttpFilter.java:127)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)
at com.ibm.fhir.server.filter.rest.FHIRRestAuthorizationServletFilter.doFilter(FHIRRestAuthorizationServletFilter.java:86)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:1001)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1139)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1010)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:75)
at com.ibm.ws.webcontainer40.servlet.CacheServletWrapper40.handleRequest(CacheServletWrapper40.java:83)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:936)
at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:279)
at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:1134)
at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:415)
at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:374)
at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:577)
at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:359)
at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.ready(HttpInboundLink.java:326)
at com.ibm.ws.http.channel.h2internal.H2HttpInboundLinkWrap.ready(H2HttpInboundLinkWrap.java:98)
at com.ibm.ws.http.channel.h2internal.H2StreamProcessor$Http2Ready.run(H2StreamProcessor.java:1787)
at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:239)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:823)
Caused by: java.lang.NullPointerException
at org.eclipse.osgi.internal.loader.BundleLoader.findResource(BundleLoader.java:611)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.getResource(ModuleClassLoader.java:216)
at com.ibm.ws.classloading.internal.GatewayClassLoader.findResource(GatewayClassLoader.java:134)
at com.ibm.ws.classloading.internal.GatewayClassLoader.getResource(GatewayClassLoader.java:116)
at java.lang.ClassLoader.getResource(ClassLoader.java:595)
at com.ibm.fhir.operation.bulkdata.client.BulkDataClient.loadKeyStoreFile(BulkDataClient.java:130)
at com.ibm.fhir.operation.bulkdata.client.BulkDataClient.getWebTarget(BulkDataClient.java:113)
at com.ibm.fhir.operation.bulkdata.client.BulkDataClient.submitImport(BulkDataClient.java:609)
at com.ibm.fhir.operation.bulkdata.processor.impl.CosExportImpl.importBulkData(CosExportImpl.java:180)
... 59 more
Simon Vadée (Jan 05 2021 at 14:20):
ok looks like I need to setup a truststore
Last updated: Apr 12 2022 at 19:14 UTC
