Stream: IG creation
Topic: Post your build time
Brett Marquard (Nov 18 2020 at 21:21):
Something went bonkers on my machine with build times back north of 30 mins -- curious if others are running faster so maybe i can use it as an excuse to buy a new computer.
Brett Marquard (Nov 18 2020 at 21:21):
Bryn Rhodes (Nov 18 2020 at 22:43):
Finished. Times: loading: 00:06.0862, generate: 32:48.0837, narrative generation: 02:00.0639, realm-rules: 03:30.0472, previous-version: 07:56.0152, jekyll: 01:56.0840, validation: 02:04.0937 (#151), template: 00:07.0255 (#3) (34:15.0830)
Adam Stevenson (Nov 18 2020 at 23:03):
Finished. Times: loading: 00:06.0080, generate: 14:28.0645, narrative generation: 00:34.0660, realm-rules: 02:27.0772, previous-version: 05:40.0346, jekyll: 00:20.0364, validation: 01:36.0571 (#151), template: 00:08.0992 (#3) (15:53.0880)
Grahame Grieve (Nov 18 2020 at 23:04):
are these first or second build times?
Adam Stevenson (Nov 18 2020 at 23:04):
Good point. Second here.
Bryn Rhodes (Nov 19 2020 at 00:01):
That was a first build, but here's the second build:
Finished. Times: loading: 00:10.0346, generate: 30:01.0685, narrative generation: 03:06.0551, realm-rules: 00:24.0906, previous-version: 00:50.0318, jekyll: 02:01.0409, validation: 01:14.0109 (#151), template: 00:08.0438 (#3) (32:14.0371)
Sarah Gaunt (Nov 19 2020 at 01:27):
My last build (on ToF) was 9:58.
I say - buy the new computer! Having said that, I got a new one in July and it really hasn't made much of a difference to local build times (and it still slows my PC down so much that it's annoying to use), so I just don't ever build locally.
Sean McIlvenna (Nov 19 2020 at 01:41):
@Grahame Grieve mentioned in another thread that the IG Publisher is single-threaded... Probably won't see much improved build times unless one or both of the following:
- code is performance-tuned
- code is refined to be multi-threaded
Maybe some progress has already been made on #1... shrug dunno.
Grahame Grieve (Nov 19 2020 at 01:41):
I spend some time working on performance issues.
Grahame Grieve (Nov 19 2020 at 01:42):
but making it multithreaded isn't going to happen, and would make things worse anyway, when the primary complaint is being unresponsive. As a single threaded application, that shouldn't happen
Sean McIlvenna (Nov 19 2020 at 01:42):
Making it multi-threaded for things like generating snapshots might be a great way to see some quick improvements
Sean McIlvenna (Nov 19 2020 at 01:42):
well... scratch that
Sean McIlvenna (Nov 19 2020 at 01:42):
lol
Grahame Grieve (Nov 19 2020 at 01:43):
it could reduce overall build times, at the price of (a) much more complexity and (b) hammering the local machine
Sean McIlvenna (Nov 19 2020 at 01:44):
In other Java projects we've been able to implement multi-threading pretty easily using parallel streams...
Sean McIlvenna (Nov 19 2020 at 01:44):
not sure if parallel streams could be used as easily in the ig publisher
Sean McIlvenna (Nov 19 2020 at 01:44):
but, maybe worth checking out...
Sean McIlvenna (Nov 19 2020 at 01:44):
(if you haven't already)
Sean McIlvenna (Nov 19 2020 at 01:45):
you can even use throttled parallel streams, and tell it to only use a single thread for debugging purposes, but multiple threads by default
Sean McIlvenna (Nov 19 2020 at 01:45):
pretty cool stuff
Sean McIlvenna (Nov 19 2020 at 01:45):
(imo)
Grahame Grieve (Nov 19 2020 at 01:45):
the question isn't how to use parallel streams, it's all the nested dependencies.
Max Masnick (Nov 20 2020 at 00:13):
Here are my US Core build times from today (modern Mac with a single-core Geekbench score of ≈1100):
- First build: Times: loading: 00:02.0960, generate: 10:49.0754, narrative generation: 00:49.0960, realm-rules: 00:02.0121, previous-version: 06:11.0127, jekyll: 00:07.0948, validation: 01:27.0814 (#230), template: 00:01.0990 (#3) (11:02.0285, including ≈9:00 waiting for tx.fhir.org explained here)
- Second build: Times: loading: 00:03.0042, generate: 03:26.0758, narrative generation: 00:52.0438, realm-rules: 00:01.0165, previous-version: 00:03.0495, jekyll: 00:09.0633, validation: 00:43.0393 (#230), template: 00:02.0020 (#3) (03:39.0894, including ≈1:20 waiting for tx.fhir.org)
- Third build + local cache of tx.fhir.org (explained here): Times: loading: 00:02.0952, generate: 01:18.0222, narrative generation: 00:07.0571, realm-rules: 00:01.0227, previous-version: 00:02.0892, jekyll: 00:08.0305, validation: 00:02.0645 (#230), template: 00:02.0792 (#3) (01:30.0681, ≈2 minutes saved by local cache)
I do see both the java and Publisher processes using >>100% CPU during builds, so not sure what is going since these are not multi-threaded.
Eric Haas (Nov 20 2020 at 01:34):
For US Core we removed the dependency to vsac because it was causing out of memory issues locally. My build is around 10 minutes... and get build failure when turn off tx server. the autobuild also around ten minutes. so I don't know why you times are so much faster?
Max Masnick (Nov 20 2020 at 10:55):
For another point of reference here's build times for a 2018 MacBook Air (1.6 GHz dual core i5; 735 single core/1468 multi-core on Geekbench):
- First build: Times: loading: 00:09.0370, generate: 15:59.0893, narrative generation: 01:16.0830, realm-rules: 00:08.0872, previous-version: 07:52.0633, jekyll: 00:15.0848, validation: 01:35.0705 (#229), template: 00:14.0535 (#3) (16:45.0519)
- Second build: Times: loading: 00:07.0764, generate: 05:34.0797, narrative generation: 01:03.0777, realm-rules: 00:03.0792, previous-version: 00:10.0190, jekyll: 00:17.0461, validation: 00:50.0034 (#229), template: 00:05.0385 (#3) (06:07.0965)
- Third build, + local cache: Times: loading: 00:09.0357, generate: 04:19.0890, narrative generation: 00:31.0255, realm-rules: 00:05.0667, previous-version: 00:12.0385, jekyll: 00:19.0188, validation: 00:17.0205 (#229), template: 00:05.0508 (#3) (05:02.0358 + 2 minutes saved by local cache)
So it looks like you can reduce build time by ≈30-40% by upgrading to a faster multi-core machine if you're on a 2 core, low clock speed laptop. Speeding up tx.fhir.org could save another 2 minutes per warm build.
@Eric Haas @Sarah Gaunt if you're seeing significantly worse build times than my MacBook Air, it seems like something may be wrong with your system. You can compare your computer on Geekbench with mine to get a sense of what it should look like if there's nothing artificially slowing down your machine.
Max Masnick (Nov 20 2020 at 11:08):
Eric Haas said:
For US Core we removed the dependency to vsac because it was causing out of memory issues locally. My build is around 10 minutes... and get build failure when turn off tx server. the autobuild also around ten minutes. so I don't know why you times are so much faster?
I don't think the IG will build without tx.fhir.org. The point of my experimentation with an additional caching layer (running locally) in front of tx.fhir.org was to see what an "offline mode" _could_ look like...but it's not truly offline because each IG build generates at least a few novel requests to tx.fhir.org that have to be proxied through.
Grahame Grieve (Nov 20 2020 at 13:37):
the issue with tx.fhir.org is latency, not server performance.
Gino Canessa (Nov 20 2020 at 13:37):
For another data point (first build):
Finished. Times: loading: 00:06.0887, generate: 14:00.0562, narrative generation: 00:54.0778, realm-rules: 00:05.0427, previous-version: 06:11.0677, jekyll: 01:32.0707, validation: 01:51.0282 (#229), template: 00:07.0190 (#3) (14:19.0185)
Second:
Finished. Times: loading: 00:06.0363, generate: 05:25.0548, narrative generation: 00:51.0962, realm-rules: 00:01.0985, previous-version: 00:05.0313, jekyll: 01:32.0432, validation: 00:43.0770 (#229), template: 00:03.0635 (#3) (05:49.0751)
Max Masnick (Nov 20 2020 at 17:09):
@Grahame Grieve I see some requests to tx.fhir.org waiting 30+ seconds for a response...that seems much to long to attribute to network latency.
Grahame Grieve (Nov 20 2020 at 17:11):
indeed. probably snomed expansions
Sarah Gaunt (Nov 20 2020 at 21:36):
This is for eCR (not US Core) (2020 MacBook Pro Intel Core i7-1068NG7 @ 2.30 GHz 1 processor, 4 cores, 8 threads, 31.81 GB): just tested on Geekbench: Single-core: 1065/Multi-core: 4104
- First build: Times: loading: 00:12.0874, generate: 12:50.0376, narrative generation: 00:39.0349, realm-rules: 01:41.0674, previous-version: 03:36.0062, jekyll: 01:27.0820, validation: 02:21.0332 (#154), template: 00:11.0274 (#3) (14:37.0987)
- Second build: Times: loading: 00:09.0631, generate: 05:51.0712, narrative generation: 00:28.0184, realm-rules: 00:05.0012, previous-version: 00:09.0966, jekyll: 01:33.0473, validation: 00:34.0548 (#154), template: 00:05.0146 (#3) (07:11.0854)
Conclusion: @Brett Marquard gets a new computer!
Having said that - my problem with running locally isn't is necessarily how long the build takes (though it's an issue), it's the fact that running the above (no matter whether it's first or later build) I see this (it's around 80%-100% CPU and 90%-100% Memory):
which means my pc is close to unusable for those 7-15 mins.
BUT, I get around this by never running locally...
David Pyke (Nov 21 2020 at 14:47):
Okay building US Core (fresh checkout):
First Run:
Finished. Times: loading: 00:03.0497, generate: 13:30.0864, narrative generation: 01:03.0939, realm-rules: 00:04.0355, previous-version: 07:42.0919, jekyll: 00:05.0861, validation: 01:39.0138 (#229), template: 00:05.0071 (#3) (13:48.0197)
real 13m50.052s
user 4m15.016s
Second Run:
Finished. Times: loading: 00:03.0749, generate: 03:14.0447, narrative generation: 00:46.0934, realm-rules: 00:01.0531, previous-version: 00:03.0339, jekyll: 00:05.0260, validation: 00:40.0525 (#229), template: 00:01.0831 (#3) (03:26.0472)
real 3m29.327s
user 5m17.078s
Eric Haas (Nov 22 2020 at 02:40):
the current autopublisher: 12:53.0302
Max Masnick (Nov 23 2020 at 11:00):
Sarah Gaunt said:
Having said that - my problem with running locally isn't is necessarily how long the build takes (though it's an issue), it's the fact that running the above (no matter whether it's first or later build) I see this (it's around 80%-100% CPU and 90%-100% Memory):
This seems strange...my MacBook Air is still more or less usable while doing an IG build, and it's way less powerful than your machine. Based on your screenshot it seems like you're running Windows on Mac hardware? Are you virtualizing Windows or using Boot Camp?
Sarah Gaunt (Nov 23 2020 at 20:00):
Oh yeah, forgot to mention - runnning BootCamp. It was the same problem with my Dell though (which wasn't all that much less specced out than the Mac).
Grahame Grieve (Nov 23 2020 at 20:50):
Finished. Times: loading: 00:03.0771, generate: 06:43.0486, narrative generation: 00:59.0644, realm-rules: 00:02.0931, previous-version: 00:04.0632, jekyll: 00:32.0090, validation: 00:51.0635 (#229), template: 00:06.0390 (#3) (07:19.0834)
Grahame Grieve (Nov 23 2020 at 20:50):
there's some problem with NDC. I'm investigating
Eric Haas (Nov 24 2020 at 02:47):
.Having said that - my problem with running locally isn't is necessarily how long the build takes (though it's an issue), it's the fact that running the above (no matter whether it's first or later build) I see this (it's around 80%-100% CPU and 90%-100% Memory):
my macbook air is unusable during the build too.. I don't use bootcamp or virtual machines. Sarah's machine is more powerful than mine but having said that having the latest and greatest hardware should not be a prerequisite for publishing.
Grahame Grieve (Nov 24 2020 at 03:00):
there's something weird going on here. I hardly notice when I have the publisher running in the background. It's almost exclusively a single threaded application. What it does do is hammer the IO subsystem. but that shouldn't be a problem. I wonder whether you have something else going on - anti-virus?
Sarah Gaunt (Nov 24 2020 at 04:36):
Maybe - will look into it. Looking at my screen shot I notice there is only 21% used by of whatever that top process is (I presume that was the publisher)... Hmmmm.... Will check it out at end of week.
Jean Duteau (Nov 24 2020 at 20:52):
same here - my MacBook Pro is about 10-20% at worst. When Jekyll runs, there is a slight hiccup every now and then, but that's it.
Max Masnick (Nov 25 2020 at 18:03):
@Sarah Gaunt are you running antivirus in Windows? Or are you building the IG inside a cloud sync folder (like Dropbox or OneDrive)? It sounds like your IG builds are triggering something else that is swamping your CPU.
Sarah Gaunt (Nov 25 2020 at 19:36):
Yeah, I'm thinking Dropbox might be the culprit actually, just haven't had a chance to look at it properly again... I did turn Dropbox off yesterday to try to test but my IG just crashed so I had other fish to fry and didn't get back to testing locally... Maybe next week when I don't have so many ballot commitments.
Last updated: Apr 12 2022 at 19:14 UTC
