Stream: ibm
Topic: NDJSON Resource Expectations
Mahesh Dabi (Jul 02 2021 at 15:06):
Hi
I need some help in bulk export. (IBM FHIR Server)
I am getting this error. I am using storageProviders type as file
{
"resourceType": "OperationOutcome",
"id": "c0-a8-0-69-5031ca80-1f6a-49b1-af4a-dfc757338946",
"issue": [
{
"severity": "fatal",
"code": "not-supported",
"details": {
"text": "storageType is disallowed, error processing request"
},
"expression": [
"<empty>"
]
}
]
}
And here is my configuration
"storageProviders": {
"default": {
"type": "file",
"_type": "ibm-cos|aws-s3|file|https",
"validBaseUrls": [],
"fileBase": "C:/Projects/Code/FHIR/IBM/liberty-runtime/wlp/usr/servers/fhir-server/output",
"bucketName": "fhir-performance",
"location": "us",
"endpointInternal": "https://s3.us-east.cloud-object-storage.appdomain.cloud",
"endpointExternal": "https://s3.us-east.cloud-object-storage.appdomain.cloud",
"auth": {
"type": "hmac",
"accessKeyId": "key",
"secretAccessKey": "secret"
},
"_iam_auth": {
"type": "iam",
"iamApiKey": "apiKey",
"iamResourceInstanceId": "resourceId"
},
Josh Mandel (Jul 02 2021 at 17:14):
Are you subscribed in #ibm ? I'll migrate this question over there.
Notification Bot (Jul 02 2021 at 17:14):
This topic was moved here from #bulk data > NDJSON Resource Expectations by Josh Mandel
Paul Bastide (Jul 05 2021 at 17:48):
Please try this...
curl -k -u "fhiruser:change-password" -H "Content-Type: application/fhir+json" \
-X GET 'https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir%2Bndjson&_type=Patient' -v
Mahesh Dabi (Jul 08 2021 at 14:06):
Hi
For enabling smart module, are there any changes apart from fhir-server-config.json?
We are testing with inferno testing tool and facing some issues.
Thanks
Mahesh
Lee Surprenant (Jul 08 2021 at 15:37):
yes, the fhir-smart module extends the ibm fhir server for this use case but its not prepackaged. its a little confusing.
Lee Surprenant (Jul 08 2021 at 15:38):
the documentation is in 5.3.3 under https://ibm.github.io/FHIR/guides/FHIRServerUsersGuide#53-openid-connect-and-oauth-20
Lee Surprenant (Jul 08 2021 at 15:39):
so there's sort of three parts:
- configuring liberty to validate the tokens. this can be skipped if the tokens are validated in front of the server
- advertising support (this is what is done through the fhir-server-config file)
- dropping the fhir-smart jar file into the server's userlib directory
Mahesh Dabi (Jul 08 2021 at 17:33):
Ok great I will try that. Also can we change the certificate? Seems that inferno testing tool doesn't like self signed s=certificate. Do you have any steps for that?
As we were testing some steps about security, we wanted to set security policy such that only few APIs are allowed e.g. Patient, Problem and Medication scopes. If we set that through OAuth then still it exposes other scopes also whereas it should not.
Thanks
Mahesh
Lee Surprenant (Jul 08 2021 at 19:48):
Yes, you can change the certificate. The docs for that are at https://ibm.github.io/FHIR/guides/FHIRServerUsersGuide#5231-configure-the-keystores but we mostly just rely on what Liberty provides for this. If you happen to be running on Kubernetes, we might be able to offer some more guidance.
Lee Surprenant (Jul 08 2021 at 19:50):
We wanted to set security policy such that only few APIs are allowed e.g. Patient, Problem and Medication scopes. If we set that through OAuth then still it exposes other scopes also whereas it should not.
the SMART auth server is what adds the scopes to the token. THe IBM FHIR Server fhir-smart module should properly enforce the granted SMART scopes.
Its not clear to me which of those 2 you're having an issue with. If you think the IBM FHIR Server (with the fhir-smart jar in the userlib) is returning resources it shouldn't, do let us know.
Lee Surprenant (Jul 08 2021 at 19:51):
seperately, its possible to restrict the IBM FHIR Server's REST api to select interactions on select resources types...if that is something you are interested in i can provide more info
Mahesh Dabi (Jul 09 2021 at 11:10):
Hi Lee thanks a lot for detailed explanation. https://inferno.healthit.gov/inferno is the tool where are are testing FHIR Apis. Also we dropped in fhir-smart-4.8.3.pom
fhir-smart-4.8.3.jar and some dependencies in userlib directory. Still getting some errors. Has anybody tested with Inferno? What changes are required to get it going with inferno? Does anybody has steps?
Thanks
Mahesh
Mahesh Dabi (Jul 09 2021 at 11:50):
I will anyway put any issues as we face it while testing
Mahesh Dabi (Jul 09 2021 at 12:21):
REQUEST: For allergy it fails
http://3.92.187.231:9443/fhir-server/api/v4/AllergyIntolerance?patient=17a818c38f8-229c1e6d-26a8-4b4a-aa8b-4bc81582bf4c
RESPONSE:
{
"resourceType": "OperationOutcome",
"issue": [
{
"severity": "fatal",
"code": "invalid",
"details": {
"text": "Fault: 'com.ibm.fhir.search.parameters.QueryParameter com.ibm.fhir.search.util.SearchUtil.buildInclusionCriteria(java.lang.String, java.util.Set, java.lang.String)'"
}
}
]
}
Caused by: java.lang.NoSuchMethodError: 'com.ibm.fhir.search.parameters.QueryParameter com.ibm.fhir.search.util.SearchUtil.buildInclusionCriteria(java.lang.String, java.util.Set, java.lang.String)'
at com.ibm.fhir.smart.AuthzPolicyEnforcementPersistenceInterceptor.beforeSearch(AuthzPolicyEnforcementPersistenceInterceptor.java:162)
at com.ibm.fhir.persistence.interceptor.impl.FHIRPersistenceInterceptorMgr.fireBeforeSearchEvent(FHIRPersistenceInterceptorMgr.java:175)
at com.ibm.fhir.server.util.FHIRRestHelper.doSearch(FHIRRestHelper.java:1058)
at com.ibm.fhir.server.util.FHIRRestHelper.doSearch(FHIRRestHelper.java:999)
at com.ibm.fhir.server.resources.Search.doSearch(Search.java:73)
at com.ibm.fhir.server.resources.Search.searchGet(Search.java:51)
at com.ibm.fhir.server.resources.Search$Proxy$_$$_WeldClientProxy.searchGet(Unknown Source)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.ibm.ws.jaxrs20.cdi.component.JaxRsFactoryImplicitBeanCDICustomizer.serviceInvoke(JaxRsFactoryImplicitBeanCDICustomizer.java:342)
at com.ibm.ws.jaxrs20.server.LibertyJaxRsServerFactoryBean.performInvocation(LibertyJaxRsServerFactoryBean.java:641)
at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.performInvocation(LibertyJaxRsInvoker.java:160)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:101)
FOR DEVICE IT WORKS
http://3.92.187.231:9443/fhir-server/api/v4/Device?patient=17a818c38f8-229c1e6d-26a8-4b4a-aa8b-4bc81582bf4c
Mahesh Dabi (Jul 09 2021 at 12:21):
Do I need to so some config changes?
Lee Surprenant (Jul 09 2021 at 13:52):
Thats kind of a weird error. Please make sure the version of fhir-smart you are using matches the version of the IBM FHIR Server that you're using.
Lee Surprenant (Jul 09 2021 at 13:53):
We've done some testing with inferno, although we found that the program edition didn't test what we were most interested in and so we've mostly used the community edition.
Mahesh Dabi (Jul 09 2021 at 14:01):
Yeah that is same 4.8.3 both way. It seems that there are some dependencies or configuration settings which are causing issues. I found some issues similar to this. Not sure whether we need to make changes in search config file.
https://github.com/IBM/FHIR/issues/300
BTW without Smart it works fine to get all the resources, but again it is wide open. Smart FHIR gives security but never returns required resources.
Mahesh Dabi (Jul 09 2021 at 14:08):
extension-search-parameters.json fhir-server-config.json
Lee Surprenant (Jul 09 2021 at 14:13):
https://github.com/IBM/FHIR/issues/300 is quite different from a NoSuchMethodError:
Lee Surprenant (Jul 09 2021 at 14:15):
one place we've used for testing our fhir server together with keycloak for smart app launch is https://github.com/Alvearie/health-patterns/tree/main/data-access
Lee Surprenant (Jul 09 2021 at 14:15):
i've been meaning to point inferno at that demo setup for a while but havn't gotten to it yet. perhaps it could be of some use as an example.
Lee Surprenant (Jul 09 2021 at 14:23):
here are the libs used in that env:
$ ls fhir/userlib
commons-codec-1.14.jar fhir-smart-4.8.3.jar jackson-core-2.11.0.jar java-jwt-3.13.0.jar
fhir-ig-us-core-4.8.3.jar jackson-annotations-2.11.0.jar jackson-databind-2.11.0.jar
(my PR to bring it to 4.8.3 just got merged)
Mahesh Dabi (Jul 09 2021 at 14:38):
oh we have java-jwt-3.13.0.pom, java-jwt-3.13.0.jar, fhir-smart-4.8.3.pom and fhir-smart-4.8.3.jar files only. Where I can find rest of jar files?
Lee Surprenant (Jul 09 2021 at 14:39):
i use maven to get them
Mahesh Dabi (Jul 09 2021 at 14:39):
/home/ubuntu/liberty-runtime/wlp/usr/servers/fhir-server/userlib this is my folder
Mahesh Dabi (Jul 09 2021 at 14:40):
oh ok let me search there and get them one by one and try out. the .pom files are needed right?
Lee Surprenant (Jul 09 2021 at 14:40):
here's the pom for this demo env: https://github.com/lmsurpre/health-patterns/blob/main/data-access/pom.xml
Lee Surprenant (Jul 09 2021 at 14:41):
with a pom like that you can just mvn clean install and it will copy the transitive dependencies
Lee Surprenant (Jul 09 2021 at 14:41):
obviously its just one way to do it though
Mahesh Dabi (Jul 09 2021 at 16:50):
Seems that following code is throwing error
// Build the Patient compartment inclusion criteria search parameter
QueryParameter inclusionCriteria = SearchUtil.buildInclusionCriteria(PATIENT, patientIdFromToken, event.getFhirResourceType());
Anything to do with compartment config?
Mahesh
Mahesh Dabi (Jul 09 2021 at 16:56):
While getting device using
We are getting following warning, see if it can give some clue about some configuration issue?
Found multiple resource-specific search parameters, 'http://hl7.org/fhir/us/core/SearchParameter/us-core-device-patient' and 'http://hl7.org/fhir/SearchParameter/Device-patient', for code 'patient' on resource type 'Device'; use search parameter filtering to disambiguate. Using 'http://hl7.org/fhir/us/core/SearchParameter/us-core-device-patient'.
Thanks
Mahesh
Mahesh Dabi (Jul 09 2021 at 17:05):
https://github.com/IBM/FHIR/blob/main/fhir-search/src/main/java/com/ibm/fhir/search/util/SearchUtil.java line 318 is generating warning mentioned above.
Lee Surprenant (Jul 09 2021 at 18:44):
wrt the warning, thats saying that you have multiple search parameters that share a common code. this is an unfortunate consequence of IGs like US Core redefining the search parameters that come with the base spec.
since they use exactly the same expression, you can probably just ignore the warning.
otherwise, if you want to make the warning go away, you can add "search parameter disambiguation" to your fhir-server-config. for example, for this specific search parameter, the following config tells the server to use the base spec's search parameter definition: https://github.com/IBM/FHIR/blob/main/demo/fhir/config/default/fhir-server-config.json#L60
Mahesh Dabi (Jul 10 2021 at 01:51):
So our situation right now is
with samrt-fhir jar files and dependencies, only patient export works, allergies and all other are not working due to error
QueryParameter inclusionCriteria = SearchUtil.buildInclusionCriteria(PATIENT, patientIdFromToken, event.getFhirResourceType());
We looked in code also but are not getting any clue. There might be only two possibilities
- Either are are missing some Jar file (We downloaded and copied all Jar file to userlib folder, just copy no install. I hope that is sufficient)
- Or we aremissing some config file (We are using same fhir config file as yours just changed the Auth URLs)
Any thoughts on what can go wrong? Without Smart-Fhir it works perfect without any restrictions.
Thanks
Mahesh
Mahesh Dabi (Jul 10 2021 at 07:12):
This is our token
{
"iss": "https://identityserverforfhir.azurewebsites.net",
"nbf": 1625901023,
"iat": 1625901023,
"exp": 1625904623,
"aud": "http://35.173.213.75:9443/fhir-server/api/v4",
"scope": "launch/patient openid fhirUser patient/Medication.read patient/AllergyIntolerance.read patient/CarePlan.read patient/CareTeam.read patient/Condition.read patient/Device.read patient/DiagnosticReport.read patient/DocumentReference.read patient/Encounter.read patient/Goal.read patient/Immunization.read patient/Location.read patient/MedicationRequest.read patient/Observation.read patient/Organization.read patient/Patient.read patient/Practitioner.read patient/Procedure.read patient/Provenance.read patient/Patient.write offline_access",
"group": [
"FHIRUsers",
"FHIRAdmins"
],
"amr": [
"pwd"
],
"client_id": "Inferno",
"sub": "818727",
"auth_time": 1625896133,
"idp": "local",
"sid": "79B57016D38830A721312876B9F8E6CD",
"patient": "17a818c38f8-229c1e6d-26a8-4b4a-aa8b-4bc81582bf4c",
"patient_id": "17a818c38f8-229c1e6d-26a8-4b4a-aa8b-4bc81582bf4c",
"upn": "818727",
"need_patient_banner": "false",
"smart_style_url": "https://identityserverforfhir.azurewebsites.net/SmartOnFHIR/v1.json",
"jti": "8E44AB5C7303464F19E7FA1687D57448"
}
And
Algorithm and token type
{
"alg": "RS256",
"kid": "987701C6B82719A6BD6DB54F24B42CC2",
"typ": "at+jwt"
}
Lee Surprenant (Jul 12 2021 at 14:25):
allergies and all other are not working due to error
sorry to hear that you're still having trouble with it. is it still the originally-reported error (Caused by: java.lang.NoSuchMethodError: 'com.ibm.fhir.search.parameters.QueryParameter com.ibm.fhir.search.util.SearchUtil.buildInclusionCriteria(java.lang.String, java.util.Set, java.lang.String)' or some other error?
Mahesh Dabi (Jul 13 2021 at 13:01):
Hi Lee
Will it be ok to give you access to my linux server if you want to look into any settings?
Another option is I can upload my server liberty folder which you can download
I have oAuth setup on Azure, my liberty is configured for that. I will provide username and password for that also.
Will this help to figure out what we are missing?
I think what we are looking at is simple working of smart-fhir based on scopes, e.g. if we are requesting patient and problems then only two should be allowed and rest all should be denied.
I think there might be something minor we might be missing config.zip
Thanks
Mahesh
Lee Surprenant (Jul 13 2021 at 13:07):
What would be most useful is if you share the error from the log. Let me know if you need help finding the log. If your using our docker container, its configured to go to console by default.
Lee Surprenant (Jul 13 2021 at 13:08):
Otherwise, the project I referenced above (https://github.com/Alvearie/health-patterns/tree/main/data-access) is intended as a complete working example and I worked to make that README as easy-to-follow as possible. Did you try it?
Mahesh Dabi (Jul 13 2021 at 13:14):
no problem. Let me try fresh, will clean everything and provide you log
Mahesh
Mahesh Dabi (Jul 13 2021 at 14:11):
Please see the log file. It is giving error while I try to get allergies.
I am sending scope to allow everything, not sure why smart is not allowing it
launch/patient openid fhirUser offline_access patient/Medication.read patient/AllergyIntolerance.read patient/CarePlan.read patient/CareTeam.read patient/Condition.read patient/Device.read patient/DiagnosticReport.read patient/DocumentReference.read patient/Encounter.read patient/Goal.read patient/Immunization.read patient/Location.read patient/MedicationRequest.read patient/Observation.read patient/Organization.read patient/Patient.read patient/Practitioner.read patient/Procedure.read patient/Provenance.read patient/Patient.write
Mahesh
Mahesh Dabi (Jul 13 2021 at 14:13):
I would suggest to review config I sent earlier along with log files.
Lee Surprenant (Jul 13 2021 at 16:09):
the logs indicate you are running IBM FHIR Server version 4.8.0:
[7/13/21, 14:06:24:538 UTC] 0000001d com.ibm.fhir.server.FHIRApplication I FHIR Server version 4.8.0 build id 'Integration_refs/tags/4.8.0_8c843c76a1ccbbfccd96ab75c3a950a5c827ba51' starting.
Lee Surprenant (Jul 13 2021 at 16:10):
Mahesh Dabi (Jul 14 2021 at 09:45):
Interesting, I downloaded latest release from https://github.com/ibm/fhir/releases which is FHIR 4.8.3, not sure why log is showing 4.8.o. Still I changed fhir-smart from 4.8.3 to 4.8.0 to match with whatever it says in log. In this case seems that smart configuration is not taking effect (restarted server also) and it keeps all objects wide open and not honoring scopes.
Mahesh
Lee Surprenant (Jul 14 2021 at 13:36):
I downloaded latest release from https://github.com/ibm/fhir/releases which is FHIR 4.8.3, not sure why log is showing 4.8.o
not sure how thats possible. i confirmed that the artifacts at https://github.com/IBM/FHIR/releases/tag/4.8.3 look right to me.
have you downloaded and installed 4.8.0 previously?
In this case seems that smart configuration is not taking effect (restarted server also) and it keeps all objects wide open and not honoring scopes.
and the fhir-smart jar (and dependencies) are in your userlib? any errors in the logs?
Mahesh Dabi (Jul 14 2021 at 14:26):
Let me do it again from start. I tried both 4.8.0 jar as well as 4.8.3 jar. 4.8.0 Jar is not showing any effect of Smart and 4.8.3 is throwing error.
I will keep you posted
Mahesh
Lee Surprenant (Jul 14 2021 at 14:27):
FYI personally, i find working with the docker image a little easier / less error-prone
Lee Surprenant (Jul 14 2021 at 16:20):
one thing you could try doing is to set a TRACE_SPEC of com.ibm.fhir.persistence.interceptor.*=fine so that you can see if the interceptor is even getting loaded. info on setting the TRACE_SPEC is at https://ibm.github.io/FHIR/guides/FHIRServerUsersGuide#312-logging-and-trace
Mahesh Dabi (Jul 16 2021 at 11:55):
Yeah I am anyway preparing fresh box. Let me test again step by step. I will update in a bit.
Mahesh Dabi (Jul 19 2021 at 09:57):
Hi
We installed fresh 4.8.3 and confirmed all the Jar versions and logs.
Now we configured OAuth, please see Server.XML and JWTRs.xml. Here we have configured users in Fhirusers group. server.xml jwtRS.xml fhir-server-config.json
We are using Outh from our Azure Server, it is sending scope along with token properly (please see bellow for token). Since basic auth is false in config json file, we are expecting Allergy API to work properly with 200 response.. Now somehow Liberty is throwing error 403 when we try to get to allergies API after successful authentication in OAuth. Are we missing any user setting in liberty server.xml or jwtRS?
{
"iss": "https://identityserverforfhir.azurewebsites.net",
"nbf": 1626686866,
"iat": 1626686866,
"exp": 1626690466,
"aud": "http://3.86.177.215:9443/fhir-server/api/v4",
"scope": "launch/patient openid fhirUser patient/Medication.read patient/AllergyIntolerance.read patient/CarePlan.read patient/CareTeam.read patient/Condition.read patient/Device.read patient/DiagnosticReport.read patient/DocumentReference.read patient/Encounter.read patient/Goal.read patient/Immunization.read patient/Location.read patient/MedicationRequest.read patient/Observation.read patient/Organization.read patient/Patient.read patient/Practitioner.read patient/Procedure.read patient/Provenance.read offline_access",
"group": [
"FHIRUsers",
"FHIRAdmins"
],
"amr": [
"pwd"
],
"client_id": "Inferno",
"sub": "fhiruser",
"auth_time": 1626673707,
"idp": "local",
"sid": "4B90892A443BEC26395EE752C2094C73",
"patient": "17ab437f80b-f1907e67-e141-467c-b3d9-c3f8ebd867be",
"patient_id": "17ab437f80b-f1907e67-e141-467c-b3d9-c3f8ebd867be",
"upn": "fhiruser",
"need_patient_banner": "false",
"smart_style_url": "https://identityserverforfhir.azurewebsites.net/SmartOnFHIR/v1.json",
"jti": "247B25C32AE454CD78168129E67EFB47"
}
Mahesh Dabi (Jul 20 2021 at 14:55):
Hi Lee
Any help here will be appreciated.
Thanks
Mahesh
Lee Surprenant (Jul 20 2021 at 16:39):
Hi Mahesh, I'm on vacation this week (and next), so I might be slow (or might not respond at all for a bit).
Now somehow Liberty is throwing error 403 when we try to get to allergies API after successful authentication in OAuth.
I had a peak at the config.zip you sent earlier and I may know the issue.
Specifically this line:
<group id="usersGroup" access-id="group:https://localhost:8443/auth/realms/test/fhirUser"/>
Please try replacing that with the following:
<group id="usersGroup" access-id="group:https://identityserverforfhir.azurewebsites.net/FHIRUsers"/>
This part is definitely a bit tricky/confusing, but what Liberty is doing is appending the group name from the access token to the issuer url to uniquely identify the group and we're mapping users in that group to our FHIRUsers security-role.
Lee Surprenant (Jul 20 2021 at 16:45):
For example, if your token had something like this insead:
"group": [
"fhirUser"
],
then the access-id would be group:https://identityserverforfhir.azurewebsites.net/FHIRUsers instead.
Mahesh Dabi (Jul 21 2021 at 10:34):
Awesome. Seems that OAuth is working properly now, SMART is still giving scope issues. It either blocks everything or opens everything. It doesn't honor scopes. Tried resources as given in https://github.com/Alvearie/health-patterns/blob/main/data-access/fhir/config/default/fhir-server-config.json
Thanks
Mahesh
Last updated: Apr 12 2022 at 19:14 UTC
