Stream: committers/notification
Topic: ig-build speed
Josh Mandel (Nov 12 2020 at 18:14):
@Eric Haas The 3rd column has build times for the last 20 jobs, ranging from 1.5min to 12min (depending on the specific IG) -- they don't seem obviously different than expected... Can you point to a specific result where you're having trouble?
$ kubectl -n fhir get jobs --sort-by=.metadata.creationTimestamp | tail -n 20
ig-buildhxz2r 1/1 11m 5h17m
ig-buildc2h6x 1/1 3m50s 5h5m
ig-buildwb84f 1/1 5m31s 4h59m
ig-buildxzhwx 1/1 6m57s 4h12m
ig-build685h9 1/1 12m 3h41m
ig-build5cw9d 1/1 90s 3h38m
ig-buildw52pw 1/1 8m17s 3h38m
ig-buildv8vc6 1/1 88s 3h38m
ig-buildwjtlr 1/1 12m 3h13m
ig-buildbnrb2 1/1 6m53s 3h9m
ig-buildf5gjn 1/1 3m6s 172m
ig-buildrf8q7 1/1 7m3s 107m
ig-build5msbp 1/1 3m50s 90m
ig-buildskqst 1/1 3m29s 79m
ig-builds7n46 1/1 16m 77m
ig-buildv2658 1/1 9m33s 59m
ig-build7sd5m 1/1 17m 56m
ig-buildgtnjd 1/1 9m38s 44m
ig-buildhj2gn 1/1 10m 39m
ig-build76chr 0/1 10m 10m
Eric Haas (Nov 12 2020 at 19:19):
OK so 10-20 minutes is expected. It is killing my productivity when editing
and developing a spec in real time. Additionally, the build slow my
laptop down so much as to make it unavailable.
Anybody have an efficient workflow. I like to see the rendered output for
each change without having to go back 10-20 minutes later to review.
Eric M Haas, DVM, MS
Health eData Inc
35 Crescent Avenue, Sausalito, CA 94965
Grahame Grieve (Nov 12 2020 at 20:18):
which IG is this?
Eric Haas (Nov 12 2020 at 20:18):
us core
Sarah Gaunt (Nov 12 2020 at 20:57):
I can't run eCR or BFDR locally without walking away from my machine and doing something else for a while. (My solution is to never run locally).
The only "efficient" workflow I can suggest is not running on your machine. I use Trifolia to run outside of the CI build without slowing my machine down. Of course, I still have to wait, but at least I can use my machine.
I also often just use the CI build as my development environment - just commit a lot and keep working and then check changes there. Again, at least it doesn't slow my machine down.
My biggest issue is that I often forget what change I was even trying to test by the time it finished running (and I've made a bunch more changes since).
I just committed eCR to Git at 6:33 and the CI build finished at 6:47.
Both BFDR and eCR are very large - lots of profiles and even more examples, so I just assumed the length of time was par for the course. (And I get super envious when people mention that their IGs only take 2 minutes!)
Grahame Grieve (Nov 12 2020 at 21:09):
honestly, why work with a slow laptop? You have a bad io system if your laptop becomes unresponsive because you are running the IG publisher. It's single threaded, so it's not CPU
Sarah Gaunt (Nov 12 2020 at 21:10):
It's brand new. I don't know what to do to make it faster?
Grahame Grieve (Nov 12 2020 at 21:11):
what is it?
Sarah Gaunt (Nov 12 2020 at 21:11):
It's the highest spec 13" MacBook Pro.
Grahame Grieve (Nov 12 2020 at 21:12):
then it shouldn't become unusable
Grahame Grieve (Nov 12 2020 at 21:12):
I don't even notice an impact on my laptop for running an IG, except the fan will speed up
Grahame Grieve (Nov 12 2020 at 21:12):
but it's not a mac
Sarah Gaunt (Nov 12 2020 at 21:13):
I don't know enough about hardware to know how to stop it from slowing down.
Sarah Gaunt (Nov 12 2020 at 21:14):
Anyway - I'm reasonably happy with using Trifolia or the CI build, so not really any issues. It's not like it's going to be any faster running locally.
Sarah Gaunt (Nov 12 2020 at 21:14):
I was really just suggesting an alternate workflow for Eric.
Eric Haas (Nov 12 2020 at 21:32):
I use the CI build too. same as you but like you said its hard to keep track of changes
Josh Mandel (Nov 12 2020 at 22:15):
The CI build should not be slower than your local hardware; we're throwing a lot of compute resources at this task! Are there folks seeing the CI build running slower than a from-scratch local build?
Josh Mandel (Nov 12 2020 at 22:17):
(If the issue is just "running the publisher is slow" rather than "the auto-build pipeline is slow", that's important too -- but we should make sure we know what/where we're trying to optimize.)
Lloyd McKenzie (Nov 12 2020 at 22:39):
Builds run fine on my machine. Take 1-2 minutes depending on the IG and I can still work while they're running - though the fan does kick in. No noticable difference in speed on the CI build. Note that I do have 32GB of RAM, which probably helps
Sarah Gaunt (Nov 12 2020 at 23:08):
I have 32GB of RAM. 1-2 minutes? That would be amazing. Have you ever run eCR or BFDR on your machine??! How many profiles and examples do your IGs have?
Sarah Gaunt (Nov 12 2020 at 23:10):
Oh wait - you're saying your builds take the same amount of time on your machine as the CI build. eCR takes close to 15 mins on the CI build...
Eric Haas (Nov 12 2020 at 23:10):
USCore:
Autobuild; 16+ minutes ( It looks like about 10 of that is version comparison)
on my macbook air ca 2016: 10:43 minutes
Josh Mandel (Nov 12 2020 at 23:11):
It looks like about 10 of that is version comparison
Interesting, what does this mean?
Eric Haas (Nov 12 2020 at 23:11):
I'm guessing from this ... http://build.fhir.org/ig/HL7/US-Core/branches/master/build.log
Eric Haas (Nov 12 2020 at 23:14):
Validating Conformance Resources (03:57.0178)
Installing hl7.fhir.us.core#3.1.1 to the package cache
Fetching:.....
Installing: ..... done.
US Core Comparisons Finished
Installing hl7.fhir.r4.core#4.0.1 to the package cache
Fetching:....................................................................................................
Installing: .................................................................................................... done.
-tx: Connect to http://tx.fhir.org/r4
Version Comparison: compare 3.1.1 to current for http://hl7.org/fhir/us/core/CodeSystem/careplan-category
Version Comparison: compare 3.1.1 to current for urn:oid:2.16.840.1.113883.6.238
Version Comparison: compare 3.1.1 to current for http://hl7.org/fhir/us/core/CodeSystem/condition-category
Version Comparison: compare 3.1.1 to current for http://hl7.org/fhir/us/core/CodeSystem/us-core-documentreference-category
...
Version Comparison: compare 3.1.1 to current for http://hl7.org/fhir/us/core/SearchParameter/us-core-medicationrequest-status
Validating Resources (12:41.0123)
but I could be wrong and it could be the us core comparison or validation step
Sarah Gaunt (Nov 12 2020 at 23:16):
I do think that a lot of time is taken up with the US Core comparisons - i.e. for every single Observation. You'd think that the actual US Core build would be significantly faster because it doesn't have to do those checks.
Sarah Gaunt (Nov 12 2020 at 23:18):
And I lied - according to the log I looked at it took 9 minutes. But the time from hitting commit to the time it shows up in Notifications makes it seem a couple of minutes slower.
Lloyd McKenzie (Nov 12 2020 at 23:29):
eCR takes 10:45 on my system. About 2 minutes of which was the comparison process, where it seemed to hang on a couple of them
Sarah Gaunt (Nov 12 2020 at 23:33):
Ok, well that makes me feel better! :)
Eric Haas (Nov 13 2020 at 00:09):
OK .. do we think its too long? The -w option is also broken for me - that would be extremely helpful for checking rendering and links. but is there a dev-lite option? 1-2 minutes would be great!
Eric Haas (Nov 13 2020 at 00:10):
(not a great time to be mucking with the build so maybe for next yr?)
Sarah Gaunt (Nov 13 2020 at 01:04):
Well, I do think it's too long. But is it realistic to think it could be significantly faster? There is a LOT of checking that happens.
Max Masnick (Nov 19 2020 at 22:19):
I did some quick benchmarks on my machine with US Core:
- No local txcache folder: 11 minutes
- ~9 minutes of this was waiting for responses from tx.fhir.org - Warm txcache folder (run twice in a row, take 2nd time): 3.6 minutes
- ~2 minutes of this was waiting for responses from tx.fhir.org
My tx.fhir.org time benchmarking is based on proxying requests via a localhost server and recording how long it takes to get a response. So this may not be perfect and could presumably vary by the load on tx.fhir.org. If anyone's interested I can probably generate a log of which requests took the longest by this timing method.
The localhost server I used also has a "cache mode" that will immediately serve a locally cached response from tx.fhir.org if one exists. If I use this I can get "warm txcache" build time down to 1 minute 30 seconds :tada: but who knows what kind of unintended consequences "cache mode" has on the build.
But this suggests that if there were a way to set up a more aggressive cache in front of tx.fhir.org, that might yield some pretty significant performance improvements for IG builds. I have no idea if the tx.fhir.org responses are amenable to this kind of easy caching though...
Last updated: Apr 12 2022 at 19:14 UTC
