Stream: committers
Topic: IG Builder running out of memory
Josh Mandel (Mar 20 2018 at 09:07):
The difference here may be that the entire container is running out of memory rather than the java process inside the container running out of memory. When the latter happens we should get an error message. The former perhaps not. We tried to prevent this by limiting the memory available to the JVM (so it remains within container limits), which maybe where things are breaking down.
Josh Mandel (Mar 20 2018 at 09:07):
@Chris Moesel
Josh Mandel (Mar 20 2018 at 09:12):
BTW do you know how much memory should be enough to run this job?
Chris Moesel (Mar 20 2018 at 14:11):
@Josh Mandel -- thanks for looking into this. I'm currently trying to see if there is a value for the docker run -m parameter (memory limit) that will allow it to work without having to disable the oom killer. I'll let you know what I find. BTW -- I've answered some of your other questions over on GH: https://github.com/hl7-fhir/auto-ig-builder/issues/3#issuecomment-374602422 (sorry for fragmenting this discussion).
Josh Mandel (Mar 20 2018 at 14:15):
Thanks. That report on timing is interesting -- 5min vs 30min just based on memory allocation
Josh Mandel (Mar 20 2018 at 14:17):
I'm inclined to just crank it way up, but I'd like to have property error detections in place, too -- which seems hard to accomplish from within the container. We may need a separate pod (in the kubernetes deployment, which is what manages our hosted environment) to mention for errors...
Chris Moesel (Mar 20 2018 at 14:18):
My last run I tried it with -m 6G and it still failed. I'm currently trying -m 8G (still running). I guess our IG must be huge compared to others (or I'm doing something _very_ wrong).
Josh Mandel (Mar 20 2018 at 14:22):
Interesting. Well, none of this is deeply tested, so maybe yours just is significantly bigger than the others. How many resources/profiles/valuesets are you defining?
Chris Moesel (Mar 20 2018 at 14:29):
The run w/ -m 8G succeeded in 19 minutes. Is it reasonable to request that you allocate 8G to the container?
This IG defines 134 logical models (!), 30 profiles, 31 extensions, 41 value sets, and 30 code systems.
Grahame Grieve (Mar 20 2018 at 18:53):
that's pretty substantial. I'll spend some time looking at optimization when I get a chance.
Chris Moesel (Mar 20 2018 at 18:59):
@Grahame Grieve -- as an update, I actually found that despite passing in -m 8G, my container was actually still only getting 2 GB allocation. That one successful run was pure luck. After reconfiguring my docker host (Docker for Mac) so I could specify a limit above 2 GB, I was able to successfully run it with an allocation of 4 GB. I ran top inside the container while it was running and found that the Java process maxed out at 2.9 GB. So... that 8 GB thing was totally wrong -- it's not as bad as we thought, but there probably still is some room for optimization!
Grahame Grieve (Mar 20 2018 at 19:12):
ok thx. all the definitions + all the resources in 2 dfferent object models + expansions for all the value sets... shouldn't really be 2.9GB... I'll have to investigate. How much html is produced?
Chris Moesel (Mar 20 2018 at 19:34):
FYI -- my final results and recommendations can be found here: https://github.com/hl7-fhir/auto-ig-builder/issues/3#issuecomment-374728387
Chris Moesel (Mar 20 2018 at 19:35):
@Grahame Grieve does this answer your question or are you looking for something else?
Chris Moesel (Mar 20 2018 at 19:35):
Checking Output HTML (00:04:13.0243sec) found 2946 files ... 2946 html files, 0 pages invalid xhtml (0%) (00:04:28.0707sec) ... 769433 links, 137 broken links (0%) (00:04:28.0707sec)
Grahame Grieve (Mar 20 2018 at 19:40):
that's it, thanks. its not that many pages....
Chris Moesel (Mar 21 2018 at 16:39):
With many, many thanks to @Josh Mandel, the auto-build is now producing my IG. Hurray! See the GitHub issue linked above if you want the gory details.
Last updated: Apr 12 2022 at 19:14 UTC
