Stream: ibm
Topic: IBM FHIR Server Let's Encrypt certificate issue
Yannick Börner (Mar 08 2022 at 11:31):
Hi everyone,
we are trying to run the IBM FHIR server v4.10.2 and KeyCloak within a docker-compose. All these components are behind an nginx proxy and hosted on Digital Ocean with Let's Encrypt certificates. We are currently stuck with this error message:
ibm-fhir | [3/6/22, 20:06:46:347 UTC] 00000028 IncidentImpl I FFDC1015I: An FFDC Incident has been created: "org.apache.http.conn.HttpHostConnectException: Connect to keycloak.example.tech:443 [keycloak.example.tech/167.237.178.187] failed: Connection timed out (Connection timed out) com.ibm.ws.security.common.jwk.impl.JwKRetriever 792" at ffdc_22.03.06_20.06.46.0.log
ibm-fhir | [3/6/22, 20:06:46:356 UTC] 00000028 JwKRetriever E CWWKS6049E: A JSON Web Key (JWK) was not returned from the URL [https://keycloak.example.tech/auth/realms/Example/protocol/openid-connect/certs]. The response status was [0] and the content returned was [IOException: Connect to keycloak.example.tech:443 [keycloak.example.tech/167.237.178.187] failed: Connection timed out (Connection timed out) java.net.ConnectException: Connection timed out (Connection timed out)].
ibm-fhir | [3/6/22, 20:06:46:359 UTC] 00000028 TAIJwtUtils E CWWKS5524E: The MicroProfile JWT feature encountered an error while creating a JWT by using the [jwtConsumer] configuration and the token included in the request. CWWKS6031E: The JSON Web Token (JWT) consumer [jwtConsumer] cannot process the token string. CWWKS6029E: The JSON Web Token (JWT) cannot be validated because a signing key cannot be found. The configured signature algorithm [RS256] requires a key to validate the token.
ibm-fhir | [3/6/22, 20:06:46:360 UTC] 00000028 MicroProfileJ E CWWKS5523E: The MicroProfile JWT feature cannot authenticate the request because the token that is included in the request cannot be validated. CWWKS5524E: The MicroProfile JWT feature encountered an error while creating a JWT by using the [jwtConsumer] configuration and the token included in the request. CWWKS6031E: The JSON Web Token (JWT) consumer [jwtConsumer] cannot process the token string. CWWKS6029E: The JSON Web Token (JWT) cannot be validated because a signing key cannot be found. The configured signature algorithm [RS256] requires a key to validate the token.
ibm-fhir | [3/6/22, 20:06:46:362 UTC] 00000028 FHIRRestServl I Received request: tenantId:[default] dsId:[default] user:[<unauthenticated>] method:[GET] uri:[https://fhir.example.tech/fhir-server/api/v4/Person]
I attached our configuration files for the FHIR server and the docker-compose.yml. We suspect it has something to do with the Let's Encrypt certificate, which the IBM FHIR server seems to have an issue with. The entire setup does work with a different kind of certificate. Has anyone seen this type of error before? @Lee Surprenant Is there anything wrong in our configuration files that catches your eye?
Lee Surprenant (Mar 08 2022 at 12:43):
Hi Yannick, it definitely sounds like the FHIR Server is not able to dereference the JWKS endpoint. I see you have that set to https://keycloak.example.tech/auth/realms/Test/protocol/openid-connect/certs
I recommend to docker exec to the fhir server container and confirm you can dereference that from there (and that it has the expected cert).
Assuming that works, then perhaps you'll need to add the TLS cert for that endpoint to your server's truststore.
Lee Surprenant (Mar 08 2022 at 12:46):
In general, that shouldn't be needed for a valid cert that is signed by a trusted root CA because we set SEC_TLS_TRUSTDEFAULTCERTS to true by default : https://github.com/IBM/FHIR/blob/main/fhir-server-webapp%2Fsrc%2Fmain%2Fliberty%2Fconfig%2FconfigDropins%2Fdefaults%2FtrustDefault.xml#L3
So while you're in there you could confirm that:
- you have our
trustDefault.xmlsnippet in your configDropins/defaults; and - you havn't overridden the SEC_TLS_TRUSTDEFAULTCERTS env var to something other than true
Yannick Börner (Mar 09 2022 at 08:00):
Hi Lee, thank you for getting back to us so quickly. We are able to confirm 1. and 2. and your suspicion was correct: We could not reach the KeyCloak server at https://keycloak.example.tech from within the docker container. We'll investigate why that is the case a bit later today and report back to you :+1:
Yannick Börner (Mar 09 2022 at 19:23):
@Lee Surprenant We were able to find the issue thanks to your hints! However, the actual cause is too embarrassing for us to reveal :D
Lee Surprenant (Mar 09 2022 at 19:24):
Oh, we’ve all been there :-)
Last updated: Apr 12 2022 at 19:14 UTC
