Stream: hapi
Topic: docker run hapi:latest fails after docker system prune?
Carl Anderson (Sep 18 2020 at 15:41):
This could very well be user error. But after doing a docker system prune I can no longer run docker run hapiproject/hapi:latest (although I can run hapi:v5.0.0).
What seems to happen is - the server starts but it's in a broken state. I see this in the logs
2020-09-18 15:34:49.034 [main] INFO c.u.f.j.p.PartitionLookupSvcImpl [PartitionLookupSvcImpl.java:83] Creating default partition definition
2020-09-18 15:34:49.134 [main] WARN c.u.f.jpa.starter.ApplicationContext [AbstractApplicationContext.java:558] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'empiSettings' defined in class path resource [ca/uhn/fhir/jpa/starter/EmpiConfig.class]: Unsatisfied dependency expressed through method 'empiSettings' parameter 0; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'ca.uhn.fhir.empi.rules.config.EmpiRuleValidator' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
2020-09-18 15:34:49.134 [main] INFO o.s.s.c.ThreadPoolTaskExecutor [ExecutorConfigurationSupport.java:208] Shutting down ExecutorService 'jobLaunchingTaskExecutor'
2020-09-18 15:34:49.143 [main] INFO c.u.f.j.c.HapiFhirLocalContainerEntityManagerFactoryBean [AbstractEntityManagerFactoryBean.java:598] Closing JPA EntityManagerFactory for persistence unit 'HAPI_PU'
2020-09-18 15:34:49.178 [main] ERROR o.s.web.context.ContextLoader [ContextLoader.java:313] Context initialization failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'empiSettings' defined in class path resource [ca/uhn/fhir/jpa/starter/EmpiConfig.class]: Unsatisfied dependency expressed through method 'empiSettings' parameter 0; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'ca.uhn.fhir.empi.rules.config.EmpiRuleValidator' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:798)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:539)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4682)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5150)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:713)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:978)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1849)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:773)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:427)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1576)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:929)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:841)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:421)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:932)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.startup.Catalina.start(Catalina.java:633)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:344)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'ca.uhn.fhir.empi.rules.config.EmpiRuleValidator' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
at org.springframework.beans.factory.support.DefaultListableBeanFactory.raiseNoMatchingBeanFound(DefaultListableBeanFactory.java:1695)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1253)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1207)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:885)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:789)
... 56 common frames omitted
18-Sep-2020 15:34:49.179 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
18-Sep-2020 15:34:49.185 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/hapi-fhir-jpaserver] startup failed due to previous errors
18-Sep-2020 15:34:49.209 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [hapi-fhir-jpaserver] registered the JDBC driver [org.h2.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
18-Sep-2020 15:34:49.211 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [hapi-fhir-jpaserver] appears to have started a thread named [hapi-fhir-jpa-scheduler-local-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.3/java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
18-Sep-2020 15:34:49.212 WARNING [main] org.apache
Carl Anderson (Sep 18 2020 at 15:42):
Again, this image was apparently working until I did a docker system prune.
And, I confirmed that I could run v5.0.0, but v5.1.0 (the latest) doesn't run.
Carl Anderson (Sep 18 2020 at 15:50):
I'm running docker 2.3.0.5 (the latest) on OSX, FWIW
Carl Anderson (Sep 18 2020 at 16:04):
@Sean McIlvenna, any ideas?
Jens Villadsen (Sep 18 2020 at 19:52):
@Carl Anderson are you running on master?
Jens Villadsen (Sep 18 2020 at 21:10):
@Carl Anderson may I suggest you check the documentation once https://github.com/hapifhir/hapi-fhir-jpaserver-starter/pull/140 is accepted
Carl Anderson (Sep 21 2020 at 14:43):
Thanks @Jens Villadsen for the link to the PR. I was not using hapi from source, I was just using the latest image in dockerhub.
Also, FWIW, after I saw the PR was merged I tried the following and still see the same error:
docker pull hapiproject/hapi:latest
docker run -p 8081:8080 -e "spring.batch.job.enabled=false" hapiproject/hapi:latest
Perhaps a new version of the docker image hasn't been released yet?
Jens Villadsen (Sep 21 2020 at 14:45):
Afaik It has not
Jens Villadsen (Sep 21 2020 at 14:46):
Building from source however only requires docker
Jens Villadsen (Sep 21 2020 at 14:46):
If you're up for it @Carl Anderson
Carl Anderson (Sep 21 2020 at 16:03):
I tried mvn install and then (after test failures) mvn -P ALLMODULES,NOPARALLEL install. I get many such failures after ~20 minutes:
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running ca.uhn.fhir.jpa.dao.r4.FhirResourceDaoR4SearchLastNAsyncIT
[ERROR] Tests run: 13, Failures: 0, Errors: 13, Skipped: 0, Time elapsed: 150.808 s <<< FAILURE! - in ca.uhn.fhir.jpa.dao.r4.FhirResourceDaoR4SearchLastNAsyncIT
[ERROR] testLastNSingleCategory Time elapsed: 0.013 s <<< ERROR!
java.lang.IllegalStateException: Failed to load ApplicationContext
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in ca.uhn.fhir.jpa.config.TestR4ConfigWithElasticsearchClient: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean]: Factory method 'entityManagerFactory' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jpaProperties' defined in ca.uhn.fhir.jpa.config.TestR4ConfigWithElasticsearchClient: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.Properties]: Factory method 'jpaProperties' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'embeddedElasticSearch' defined in ca.uhn.fhir.jpa.config.TestR4ConfigWithElasticsearchClient: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [pl.allegro.tech.embeddedelasticsearch.EmbeddedElastic]: Factory method 'embeddedElasticSearch' threw exception; nested exception is pl.allegro.tech.embeddedelasticsearch.EmbeddedElasticsearchStartupException: Failed to start elasticsearch. Check previous logs for details
Right now it's still going, taking a long time on:
[INFO] Running ca.uhn.fhir.jpa.dao.r4.PersistObservationIndexedSearchParamLastNR4IT
Do I need to install elasticsearch before building? I've been following along in the hacking guide.
Sean McIlvenna (Sep 21 2020 at 16:23):
Yah, I haven't rebuilt the latest
Sean McIlvenna (Sep 21 2020 at 16:23):
Docker hub's automatic builds don't work... they just hang half-way through building the image
Sean McIlvenna (Sep 21 2020 at 16:23):
so, I have to manually build and push the image
Sean McIlvenna (Sep 21 2020 at 16:25):
I'm building now... will push :latest assuming it builds successfully
Sean McIlvenna (Sep 21 2020 at 16:38):
I've updated latest
Jens Villadsen (Sep 21 2020 at 16:41):
@Sean McIlvenna you do realize that the previous version of the Docker file (https://github.com/hapifhir/hapi-fhir-jpaserver-starter/commit/af6bd69db027c6632283d03947a1b3ac4eeac4d7#diff-3254677a7917c6c01f55212f86c57fbf) required input while building the image which definitely would cause any automated setup to halt?
Jens Villadsen (Sep 21 2020 at 16:43):
@Carl Anderson it doesn't sound like you cloned https://github.com/hapifhir/hapi-fhir-jpaserver-starter and then ran ./build-docker-image. That should only take a max of 5-10 minutes or so an puts no requirements on your local environment
Jens Villadsen (Sep 21 2020 at 16:44):
@Sean McIlvenna if the state of master with its current dockerfile (https://github.com/hapifhir/hapi-fhir-jpaserver-starter/blob/master/Dockerfile) does not work in the automated build setup, then let me know.
Sean McIlvenna (Sep 21 2020 at 16:44):
@Jens Villadsen not sure what you mean...
Sean McIlvenna (Sep 21 2020 at 16:45):
what input does the dockerfile require?
Sean McIlvenna (Sep 21 2020 at 16:45):
I've ran these docker files without any extra input since day one
Jens Villadsen (Sep 21 2020 at 16:45):
didn't you as well ran on an unversioned ubuntu?
Sean McIlvenna (Sep 21 2020 at 16:45):
Also, DevOps automatically builds it just fine, it's only docker hub that freezes
Sean McIlvenna (Sep 21 2020 at 16:46):
no, have run it on win10...
Sean McIlvenna (Sep 21 2020 at 16:46):
and ubuntu
Jens Villadsen (Sep 21 2020 at 16:46):
your dockerr file claimed "FROM ubuntu"
Sean McIlvenna (Sep 21 2020 at 16:46):
oh I see what you mean
Sean McIlvenna (Sep 21 2020 at 16:47):
ok, yes.
Jens Villadsen (Sep 21 2020 at 16:47):
I noticed that when I ran the previous dockerfile, the installation of Java required input from the terminal
Jens Villadsen (Sep 21 2020 at 16:47):
building the image
Jens Villadsen (Sep 21 2020 at 16:47):
and that causes a halt
Jens Villadsen (Sep 21 2020 at 16:47):
the current setup is free of such halts
Jens Villadsen (Sep 21 2020 at 16:49):
so @Sean McIlvenna where does things stop right now? do you have a log I can see?
Sean McIlvenna (Sep 21 2020 at 17:01):
looking at the build history on docker hub, I see the last :latest actually succeeded
Sean McIlvenna (Sep 21 2020 at 17:01):
but all of the ones prior to that hung up and had to be canceled
Sean McIlvenna (Sep 21 2020 at 17:02):
I'll try manually triggering docker hub's build, and just leave it, and see if it ever finishes
Sean McIlvenna (Sep 21 2020 at 17:02):
part of the problem is that docker hub doesn't handle the massive log size from maven getting 1000's of the dependencies
Sean McIlvenna (Sep 21 2020 at 17:02):
and freezes the browser
Sean McIlvenna (Sep 21 2020 at 17:02):
anyways, we'll see what happens
Sean McIlvenna (Sep 21 2020 at 17:03):
it's running now
Sean McIlvenna (Sep 21 2020 at 17:10):
FYI, docker hub performs much better in chrome... doesn't perform well at all in firefox
Sean McIlvenna (Sep 21 2020 at 17:14):
hmmm... the last successful build ran in 6 minutes. we're approaching 12 now on this one
Jens Villadsen (Sep 21 2020 at 17:14):
kill it
Jens Villadsen (Sep 21 2020 at 17:15):
In this PR, I've made maven quite
Jens Villadsen (Sep 21 2020 at 17:15):
https://github.com/hapifhir/hapi-fhir-jpaserver-starter/pull/141
Sean McIlvenna (Sep 21 2020 at 17:16):
"I've made maven quite..."???
Sean McIlvenna (Sep 21 2020 at 17:17):
quite fast? quite better? ;)
Jens Villadsen (Sep 21 2020 at 17:17):
less verbose
Sean McIlvenna (Sep 21 2020 at 17:17):
what does -ntp do?
Gino Canessa (Sep 21 2020 at 17:17):
quite quieter :-)
Sean McIlvenna (Sep 21 2020 at 17:17):
"Do not display transfer progress when downloading or uploading"
Sean McIlvenna (Sep 21 2020 at 17:17):
gotcha
Sean McIlvenna (Sep 21 2020 at 17:17):
that's nice
Jens Villadsen (Sep 21 2020 at 17:19):
quiet
Jens Villadsen (Sep 21 2020 at 17:19):
argh.
Sean McIlvenna (Sep 21 2020 at 17:20):
gotcha lol
Sean McIlvenna (Sep 21 2020 at 17:20):
so... it auto-triggered build
Sean McIlvenna (Sep 21 2020 at 17:20):
will see what happens
Sean McIlvenna (Sep 21 2020 at 17:20):
according to the logs from my canceled build, it was still downloading maven dependencies
Sean McIlvenna (Sep 21 2020 at 17:20):
oddly, noticed that CI triggered TWO builds for the same commit
Sean McIlvenna (Sep 21 2020 at 17:20):
no idea why
Jens Villadsen (Sep 21 2020 at 17:22):
putting 2 boys to sleep while engaging with the open source community in an non-native tounge is a PITA
Sean McIlvenna (Sep 21 2020 at 17:22):
lol
Sean McIlvenna (Sep 21 2020 at 17:22):
I can only imagine
Sean McIlvenna (Sep 21 2020 at 17:22):
doing good :)
Sean McIlvenna (Sep 21 2020 at 17:31):
success!
Sean McIlvenna (Sep 21 2020 at 17:31):
still took 12 minutes
Sean McIlvenna (Sep 21 2020 at 17:31):
but it succeeded
Sean McIlvenna (Sep 21 2020 at 17:31):
no idea why it's double-building still
Sean McIlvenna (Sep 21 2020 at 17:32):
but, that's less of an issue
Gino Canessa (Sep 21 2020 at 17:35):
Jens Villadsen said:
putting 2 boys to sleep ... is a PITA
As a parent of 2 boys, I fixed it for you :-).
Jens Villadsen (Sep 21 2020 at 17:45):
image.png ... I have a daugther as well ;)
Jens Villadsen (Sep 21 2020 at 17:46):
but great stuff if the quite quieter maven works ;)
Last updated: Apr 12 2022 at 19:14 UTC
