Stream: IG creation
Topic: Getting terminology server timeouts
Jean Duteau (Dec 08 2020 at 20:50):
I'm getting terminology server timeouts when trying to do a run. These have been happening randomly all day yesterday and today. Today's are interesting because the error is different and the spot in the build is different:
Validating Conformance Resources (01:18.0360)
Publishing Content Failed: An error occurred trying to process this transaction request (02:01.0683)
(02:01.0684)
Use -? to get command line help (02:01.0685)
(02:01.0685)
Stack Dump (for debugging): (02:01.0685)
org.hl7.fhir.r5.utils.client.EFhirClientException: An error occurred trying to process this transaction request
at org.hl7.fhir.r5.utils.client.FHIRToolingClient.handleException(FHIRToolingClient.java:325)
at org.hl7.fhir.r5.utils.client.FHIRToolingClient.transaction(FHIRToolingClient.java:296)
at org.hl7.fhir.convertors.txClient.TerminologyClientR5.validateBatch(TerminologyClientR5.java:120)
at org.hl7.fhir.r5.context.BaseWorkerContext.validateCodeBatch(BaseWorkerContext.java:820)
at org.hl7.fhir.validation.instance.type.ValueSetValidator.validateValueSetInclude(ValueSetValidator.java:121)
at org.hl7.fhir.validation.instance.type.ValueSetValidator.validateValueSetCompose(ValueSetValidator.java:68)
at org.hl7.fhir.validation.instance.type.ValueSetValidator.validateValueSet(ValueSetValidator.java:58)
at org.hl7.fhir.validation.instance.InstanceValidator.checkSpecials(InstanceValidator.java:3919)
at org.hl7.fhir.validation.instance.InstanceValidator.startInner(InstanceValidator.java:3891)
at org.hl7.fhir.validation.instance.InstanceValidator.start(InstanceValidator.java:3750)
at org.hl7.fhir.validation.instance.InstanceValidator.validateResource(InstanceValidator.java:4916)
at org.hl7.fhir.validation.instance.InstanceValidator.validate(InstanceValidator.java:700)
at org.hl7.fhir.validation.instance.InstanceValidator.validate(InstanceValidator.java:675)
at org.hl7.fhir.igtools.publisher.Publisher.validate(Publisher.java:4867)
at org.hl7.fhir.igtools.publisher.Publisher.validate(Publisher.java:3843)
at org.hl7.fhir.igtools.publisher.Publisher.loadConformance(Publisher.java:3824)
at org.hl7.fhir.igtools.publisher.Publisher.createIg(Publisher.java:874)
at org.hl7.fhir.igtools.publisher.Publisher.execute(Publisher.java:729)
at org.hl7.fhir.igtools.publisher.Publisher.main(Publisher.java:8414)
Caused by: java.net.SocketTimeoutException: timeout
at okio.Okio$4.newTimeoutException(Okio.java:230)
at okio.AsyncTimeout.exit(AsyncTimeout.java:285)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:241)
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345)
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217)
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211)
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at org.hl7.fhir.r5.utils.client.network.RetryInterceptor.intercept(RetryInterceptor.java:60)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
at okhttp3.RealCall.execute(RealCall.java:69)
at org.hl7.fhir.r5.utils.client.network.FhirRequestBuilder.executeAsBatch(FhirRequestBuilder.java:214)
at org.hl7.fhir.r5.utils.client.network.Client.executeBundleRequest(Client.java:197)
at org.hl7.fhir.r5.utils.client.network.Client.postBatchRequest(Client.java:181)
at org.hl7.fhir.r5.utils.client.FHIRToolingClient.transaction(FHIRToolingClient.java:294)
... 17 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at okio.Okio$2.read(Okio.java:139)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
... 43 more
Jean Duteau (Dec 08 2020 at 21:11):
After testing a different IG of mine which ran fine to completion, I deleted my txcache and reran the publisher. I still get an error at the same place, but it is a slightly different error:
Validating Conformance Resources (01:11.0885)
Publishing Content Failed: null (01:22.0606)
(01:22.0606)
Use -? to get command line help (01:22.0607)
(01:22.0607)
Stack Dump (for debugging): (01:22.0608)
java.lang.NullPointerException
at org.hl7.fhir.r5.context.BaseWorkerContext.processValidationResult(BaseWorkerContext.java:1057)
at org.hl7.fhir.r5.context.BaseWorkerContext.validateCodeBatch(BaseWorkerContext.java:825)
at org.hl7.fhir.validation.instance.type.ValueSetValidator.validateValueSetInclude(ValueSetValidator.java:121)
at org.hl7.fhir.validation.instance.type.ValueSetValidator.validateValueSetCompose(ValueSetValidator.java:68)
at org.hl7.fhir.validation.instance.type.ValueSetValidator.validateValueSet(ValueSetValidator.java:58)
at org.hl7.fhir.validation.instance.InstanceValidator.checkSpecials(InstanceValidator.java:3919)
at org.hl7.fhir.validation.instance.InstanceValidator.startInner(InstanceValidator.java:3891)
at org.hl7.fhir.validation.instance.InstanceValidator.start(InstanceValidator.java:3750)
at org.hl7.fhir.validation.instance.InstanceValidator.validateResource(InstanceValidator.java:4916)
at org.hl7.fhir.validation.instance.InstanceValidator.validate(InstanceValidator.java:700)
at org.hl7.fhir.validation.instance.InstanceValidator.validate(InstanceValidator.java:675)
at org.hl7.fhir.igtools.publisher.Publisher.validate(Publisher.java:4867)
at org.hl7.fhir.igtools.publisher.Publisher.validate(Publisher.java:3843)
at org.hl7.fhir.igtools.publisher.Publisher.loadConformance(Publisher.java:3824)
at org.hl7.fhir.igtools.publisher.Publisher.createIg(Publisher.java:874)
at org.hl7.fhir.igtools.publisher.Publisher.execute(Publisher.java:729)
at org.hl7.fhir.igtools.publisher.Publisher.main(Publisher.java:8414)
Grahame Grieve (Dec 08 2020 at 21:22):
ok I just talked to James and Mark (tooling tech team) and we're going to put nginx in front of the terminology server to see if that makes any difference, but it won't be this week
Jean Duteau (Dec 08 2020 at 21:25):
i'm just wondering if this new error is a timeout? it's a NPE when validating a value set
Jean Duteau (Dec 08 2020 at 21:30):
looking at the code, this does actually seem not to be a timeout. it appears that it's getting an answer back from the tx server but it's not in the format that the code is expecting?
Grahame Grieve (Dec 08 2020 at 22:08):
no that's something else. will be fixed next release. I'm surprised it hasn't happened before, actually.
Bryn Rhodes (Jan 13 2021 at 22:21):
I am intermittently getting this error as well, but only on some IGs. Was this fix implemented? Is it a function of the number/size of value sets?
David deRoode (Mar 09 2021 at 17:15):
Getting Tx server timeout 'Error fetching the server's capability statement: connection timed out'. Cleared my Tx cache but no luck. @Grahame Grieve
Last updated: Apr 12 2022 at 19:14 UTC
