Stream: fhir/infrastructure-wg
Topic: Working session to get newbie committers up to speed
Michael Donnelly (Sep 27 2021 at 14:15):
At the WGM last week, we talked about having a session where experienced folks would some people new to the committing process through getting the build working locally, so we'd be able to usefully volunteer to get down the list of outstanding trackers to be applied. When are we planning to do that?
Notification Bot (Sep 27 2021 at 14:15):
Michael Donnelly has marked this topic as resolved.
Notification Bot (Sep 27 2021 at 14:15):
Michael Donnelly has marked this topic as unresolved.
Lloyd McKenzie (Sep 27 2021 at 14:55):
My leaning is next week. Finalize that on today's call?
Michael Donnelly (Sep 27 2021 at 17:55):
Cool. I won't be there today, but I'll make it solid on my calendar for next week.
Michael Donnelly (Oct 04 2021 at 19:03):
How much other interest have we gotten so far?
Josh Mandel (Oct 04 2021 at 19:42):
Did we... advertise it the way we said we would?
Lloyd McKenzie (Oct 04 2021 at 20:50):
I don't remember how we said we would...
Josh Mandel (Oct 04 2021 at 21:02):
I added documentation for the Docker environment at https://github.com/FHIR/auto-ig-builder/tree/master/images/ig-publisher-base#build-the-core-fhir-spec and included a note + link to this in bolded text at https://confluence.hl7.org/display/FHIR/FHIR+Build+Process
Grahame Grieve (Oct 04 2021 at 21:04):
note that I don't understand a word of that
Josh Mandel (Oct 04 2021 at 21:12):
You might not need to at this stage; if you don't prefer to use Docker, you shouldn't. But of course, if it works and we get feedback, we can start making it more approachable for folks who aren't already seeking it out.
Josh Mandel (Oct 04 2021 at 21:15):
That said, if you do have docker installed locally (e.g. on linux, or in wsl on windows, or on an intel mac, or following instructions for platform emulation on an M1 Mac (though for the M1 environment, someone with a mac should probably produce a distinct image to take advantage of the architecture)... you should be able to literally paste the following into a bash shell and have the spec build:
git clone https://github.com/hl7/fhir
cd fhir
docker run --rm -it \
-v $(pwd):/home/publisher/ig \
hl7fhir/ig-publisher-base:latest \
./publish.sh
Grahame Grieve (Oct 04 2021 at 21:16):
Get the image locally: do I do both those things? or just choose one?
Build an IG: do I want to do that? is "$(pwd):" something I copy and paste, or do I do something with it first?
"If you want to avoid downloading depenencies like gradle and kindling on each run" - how would I know whether I want to do that or not? What's gradle and kindling?
same for "Similarly if you wan to avoid downloading FHIR cache entries repeatedly" - would I want to ? What it be good? or bad?
Grahame Grieve (Oct 04 2021 at 21:17):
I'm trying to get my tx server to run on linux and osx right now, and I'm encountering piles of documentation like this - giving me a choice I don't understand, in a syntax I don't understand - insider doco written for other insiders
Josh Mandel (Oct 04 2021 at 21:19):
Get the image locally: do I do both those things? or just choose one?
You can pick; if you just follow the examples, Docker will download the image automatically.
Josh Mandel (Oct 04 2021 at 21:19):
Build an IG: do I want to do that?
The premise if you either want to build an IG or you want to build the FHIR core spec. There is a section for each.
Grahame Grieve (Oct 04 2021 at 21:21):
I'm just picking on the documentation. Saying 'you'll want to' makes me ask myself if I do, and why I do.
Josh Mandel (Oct 04 2021 at 21:21):
Is "$(pwd):" something I copy and past
You copy and paste. It's a syntax that the shell resolves as "run the pwd command and insert its output here", which is to say: $(pwd) will resolve to the full path of the folder you're in when you run the docker command (i.e., wherever you cloned the fhir spec to).
Josh Mandel (Oct 04 2021 at 21:21):
This is 100% insider documentation.
Josh Mandel (Oct 04 2021 at 21:21):
First it needs to work. If it works, we can start documenting it in a way that others find useful.
Josh Mandel (Oct 04 2021 at 21:22):
But I'll want feedback from @Bas van den Heuvel and @Michael Donnelly and @Vassil Peytchev or others on the "insider track" to get the mechanics ironed out before we go deep on explanations. (And as I suggested: probably if you don't know Docker, you don't want to use a docker-focused build process.)
Grahame Grieve (Oct 04 2021 at 21:23):
I reckon they don't know all the answers to my questions
Vassil Peytchev (Oct 04 2021 at 21:45):
FWIW, I just finished a non-Docker build on a Linux VM with:
- Java 11 OpenJDK
- Jeckyl (latest Ruby and latest Jeckyl for Ubuntu LTS 20.04)
- 12 GB memory
- No JAVA_OPTIONS settings
and also with the caveat that I had been able to occasionally build the core spec over the last few years with only few issues.
Total time: 52m 29s
I will see if I can get a clean environment to try Docker.
Grahame Grieve (Oct 05 2021 at 01:32):
52min is very long
Lloyd McKenzie (Oct 05 2021 at 04:07):
I've updated the documentation here: https://confluence.hl7.org/display/FHIR/FHIR+Build+Process to reflect what's actually needed:
-yes, you need a JDK (not a JRE)
- no need for Ant, Jekyll or .NET or to full around with JAVA_HOME
- you do need to set JAVA_TOOL_OPTIONS
Lloyd McKenzie (Oct 05 2021 at 04:07):
On a Windows machine w/ 32GB, full build took me 23min
Vassil Peytchev (Oct 05 2021 at 14:15):
52min is very long
That was the first gradle-based build. After running clean.sh, the second build took 31 min.
What I noticed was that the validation of each artifact (resource, or example) took at least one second each. Maybe that is one place where the build can be sped up.
Josh Mandel (Oct 05 2021 at 15:47):
On my Linux desktop (64GB RAM, SSD, Ryzen 5 3600 CPU -- my 2020 work-from-home PC) the full build took ~11min. This includes downloading gradle and whatever else "publish.sh" needed to do from a clean state.
Yunwei Wang (Oct 05 2021 at 19:12):
Do we need to do docker build? I used to just run ./gradlew publish.sh as instructed by README
Lloyd McKenzie (Oct 05 2021 at 19:16):
No need for docker if you want to invoke it directly.
Bas van den Heuvel (Oct 06 2021 at 07:02):
Docker is useful if there are specific environment requirements as it allows setting them up in a clean environment. This allows for checking those requirements each time the container is run and shields the local system from this specific requirements. E.g. in the example provided by Josh, running the IG builder requires sushi, python, graphiz, jekyll, asciidoc,... In such situation, a Dockerized build as a lot of advantages. It also makes it easy to run build jobs on different computers.
In the case of the FHIR build, the only remaining environment requirement seams to be a recent Java SDK. Although it is perfectly feasible running the build in Docker. I think taking the Docker approach in this situation might be a little bit overkill.
It could still be useful if one wants to offload the build to a different machine. We can provide a script/Dockerfile for this but I think this is less critical.
Josh Mandel (Oct 06 2021 at 13:02):
100% agree. Docker is not a solution I'm trying to push on anyone or "sell" this community on. It's just something I use already in the auto build pipeline and in my own development, and for folks who want it, it's available.
Grahame Grieve (Oct 12 2021 at 19:22):
What I noticed was that the validation of each artifact (resource, or example) took at least one second each. Maybe that is one place where the build can be sped up
well, you're welcome to dive into the validation code and find speed improvements. But the low hanging fruit was picked a long time ago
Vassil Peytchev (Oct 13 2021 at 04:09):
The slowdown must have been due to either that I was running it on VMWare VM, or that I didn't give enough memory to Java.
I ran a new clone directly on a 4-core Intel laptop with 16GB memory, SSD, Linux with _JAVA_OPTIONS=-Xmx15000m (note the leading underscore) - 17 minutes
Gino Canessa (Oct 13 2021 at 15:01):
Somewhat related, if you are planning on working on the core spec, it is probably a good idea to subscribe to #committers . E.g., Grahame asked for people to pause work while he straightens out some build issues in this thread.
Somewhat related to that, if you are pushing builds you can subscribe to #committers/notification to see the results of builds in Zulip =).
Josh Mandel (Oct 13 2021 at 15:17):
(back in the day, https://github.com/HL7/fhir/wiki/Get-Started-with-FHIR-on-GitHub#get-started-with-git-github-and-the-fhir-spec documented a process for joining committers; is this documentation still correct or authoritative? I couldn't find a Confluence equivalent but may not have searched for the right things.)
John Moehrke (Oct 13 2021 at 16:52):
should be a fine place to keep that on github, right? Just might be useful to have a small article on confluence pointing to that github wiki page.
John Moehrke (Oct 13 2021 at 16:54):
that wiki does not mention vscache, and doesn't mention what to do when your branch gets behind master.
John Moehrke (Oct 13 2021 at 16:54):
likely should also have some hints about how helpful revert is.
Last updated: Apr 12 2022 at 19:14 UTC
