Stream: IG creation
Topic: Setting up an auto-build
Elliot Silver (Sep 01 2020 at 20:39):
For people doing non-HL7 IGs, I've set up a GitHub workflow to do IG auto-builds and publish to GitHub pages outside the build.fhir.org infrastructure. It leverages the ig-publisher-base docker image. There are likely a few more tweaks I need to make, but I've documented the process here.
Thanks to @Josh Mandel for helping me get started on this.
Grahame Grieve (Sep 01 2020 at 20:47):
why is this good?
Josh Mandel (Sep 01 2020 at 21:00):
It shares tooling and provides automation to give people more options for working with the fhir publication infrastructure.
Josh Mandel (Sep 01 2020 at 21:01):
for example some organizations might want to develop guides that they publish entirely internally, and it would not be fair to expect they would leverage a universally readable build.fhir.org infrastructure.
Josh Mandel (Sep 01 2020 at 21:01):
At the same time, it would be great to make sure they are able to leverage the existing tool chain and get the same results
Grahame Grieve (Sep 01 2020 at 21:03):
that's fair. What I can tell you is:
- this page https://fhir.github.io/auto-ig-builder/builds.html is incredibly useful, and it would be sad for it to be missing stuff
- there's a view amongst some (including on the HL7 side) that build.fhir.org is only for HL7 IGs and I don't want to give that weight
- I didn't follow the justification about IGs being removed from build.fhir.org after 30 days - that doesn't happen, except for branches?
Elliot Silver (Sep 01 2020 at 21:26):
The FHIR IG auto-build is great, but:
- How do I get my IG off of the server after the build? I don't want to use my local build for the source of a published version--I'd prefer a controlled environment, but I don't know how to download the entire published IG from build.fhir.org.
- I see this as the first step toward being able to automatically publish milestone releases linked to code branches (or labels). This would give a fully automated publication system without need for manual steps.
I don't know where I'd heard the 30-day deletion. I'll remove the comment if it is wrong. "Except for branches" -- does that mean builds from master are kept, but not others?
Josh Mandel (Sep 01 2020 at 21:34):
Currently we delete branches on the main fhir build but I don't think we delete anything from implementation guides based on time elapsed. Of course we do overwrite old branch builds when new content is committed to those branches.
Grahame Grieve (Sep 01 2020 at 21:37):
well, publishing is a different thing indeed. We do a local build for publishing, and you currently need to, I agree. It seems like overkill to set up a different publishing paradigm just for that
Grahame Grieve (Sep 01 2020 at 21:38):
have you looked at the publishing process we currently use? It's quite manual, but the manual stuff is very much around QA
Josh Mandel (Sep 01 2020 at 21:41):
@Elliot Silver thanks so much for writing this up. I learned something, or at least I have questions :-) Without mounting any directories from the host into the docker image, how do you ensure that the outputs generated in one step are available mounted into the container in the next step? There must be some magic under the hood?
Elliot Silver (Sep 01 2020 at 21:42):
I did look at the current publication process, but (and this ties back to an earlier discussion), it wasn't easy to figure out which steps were HL7-specific, and which were necessary for anyone publishing.
Josh Mandel (Sep 01 2020 at 21:43):
https://docs.github.com/en/actions/creating-actions/dockerfile-support-for-github-actions#workdir is the answer to my question, I think. So it's not using the expected working directory but rather one supplied by the GitHub actions framework. Cool!
Grahame Grieve (Sep 01 2020 at 21:44):
well, perhaps we could clarify that, and more generally, the documentation needs work (as always). And there's not really a set answer - there's certain things about the way the HL7 publishes that are not necessary but are an extremely good idea. So what is HL7 specific? The only truly HL7 specific stuff is the the HL7 publishing requirements
Elliot Silver (Sep 01 2020 at 21:45):
@Josh Mandel Exactly, and the whole action is being run from an ubuntu image that hosts the complete process and launches the docker images from within it. The ubuntu drive persists across steps.
Elliot Silver (Sep 01 2020 at 21:47):
I don't know which parts are HL7 specific--that was part of my problem. I suspect when I get to actually doing a milestone build, I'll have to figure that out.
Grahame Grieve (Sep 01 2020 at 21:50):
so I was thinking about this. The first problem is that you can't publish the build.fhir.org version - it's to do with the internal identification of what is generated - it's always and specifically identified internally as the ci-build output. And when you build locally, that is also labelled internally as local.
There's a special parameter to generate a build intended for publication
Josh Mandel (Sep 01 2020 at 21:56):
Perhaps a useful analogy here is a system offered by Docker Hub called a "trusted build" -- the idea is that instead of uploading your own images to Docker hub, you can ask Docker hub to watch your GitHub repository and build an image automatically following your own script anytime the content changes. This helps your users trust that the images you publish don't have random extra stuff hidden in them.
Josh Mandel (Sep 01 2020 at 21:56):
Ideally any fhir publication would be created automatically from a repository anytime certain tags were applied.
Josh Mandel (Sep 01 2020 at 21:56):
Assuming of course the various kinds of checks have passed.
Josh Mandel (Sep 01 2020 at 21:57):
That way has an author you can declaratively state that it's time for a commit to be picked up as a publication, simply by playing a tag .
Josh Mandel (Sep 01 2020 at 21:57):
I recognize that's probably a ton of complexity that will make this impractical for hl7, but at least we can start to match things in that direction...
Grahame Grieve (Sep 01 2020 at 21:58):
well, we could. The auto-ig-build could add the requisite parameters under the right conditions, and then it could provide for a zip of the output to be downloaded.
Grahame Grieve (Sep 01 2020 at 21:58):
but I don't exactly follow the tag thing, and how it inter-relates with the PR process
Grahame Grieve (Sep 01 2020 at 21:59):
but we could make that change and remove the need for a local build from our publishing process
Josh Mandel (Sep 01 2020 at 22:00):
I think it's a distinct thing from the pull request process -- a tag can be applied to any commit on a branch. Then various systems that listen for changes can look for new tags. So you can embed some semantics in these tags and have new tags like v1.2.3 trigger certain actions, like a build and test suite run and publication to npm registry...
Elliot Silver (Sep 01 2020 at 22:01):
I think Josh means that if I label a build with "2020Sep" it would recognize it as a milestone build.
Grahame Grieve (Sep 01 2020 at 22:01):
so if a commit makes it into 2 different PRs, which PR is the actual right one?
Grahame Grieve (Sep 01 2020 at 22:02):
or if you build off master, because a tag is present, when does it stop being present?
Elliot Silver (Sep 01 2020 at 22:02):
The PR is the update of the content. The labelling is the trigger for a publication build.
Elliot Silver (Sep 01 2020 at 22:03):
I suspect you'd want to branch off main for each milestone so that you can do later technical corrections.
Grahame Grieve (Sep 01 2020 at 22:05):
so a branch for the publication, and the branch carries the tag? That's something I can understand
Josh Mandel (Sep 01 2020 at 22:06):
Yes that is correct. The branch might eventually advance past the tag; the tag is immutable.
Josh Mandel (Sep 01 2020 at 22:06):
That's why I say that technically the tag is attached to a commit rather than to a branch.
Josh Mandel (Sep 01 2020 at 22:08):
Fundamentally in git, tags and branches are both just ways to label a specific commit; but tags are best understood as immutable, whereas branches are typically advanced along to the newest commit in a particular line of work.
Grahame Grieve (Sep 01 2020 at 22:10):
clearly my understanding of git is deficient. If you tag master, how is that evident to someone who is attached to master?
Grahame Grieve (Sep 01 2020 at 22:11):
because I don't see how you 'tag a commit'.
Elliot Silver (Sep 01 2020 at 22:12):
They are listed on the same pulldown as the branches.
Screen-Shot-2020-09-01-at-3.11.40-PM.png
Grahame Grieve (Sep 01 2020 at 22:12):
Right. I had thought they were pretty much the same as branches.
Elliot Silver (Sep 01 2020 at 22:13):
And at the top right of that image, obscured by some floating text, is how you create a tag.
Grahame Grieve (Sep 01 2020 at 22:15):
so you actually tag a particular version... Josh, does that prompt a build through the ci-build?
Elliot Silver (Sep 01 2020 at 22:25):
Definitely maybe.
I think the standard way of setting up the auto-build webhook is to only kick it off on a push, but there is an option to set the webhook to kick off on a label action too. The payload that is sent with the webhook probably has enough content to be able to identify what the trigger was (and what the label is).
Josh Mandel (Sep 01 2020 at 22:26):
Currently the CI build infrastructure pays no attention to tags. But of course it could, if we want to adopt some conventions around publication. Right now the CI infrastructure is all about preview
Grahame Grieve (Sep 01 2020 at 22:28):
if it would build on a tag, and provide the tag to the IG publisher (-tag v3.1.1), then I could do things differently and generate as if it's a real publication. And then, if the ci-build provides a way to download a .zip, I would rewrite the process to use that zip)
Josh Mandel (Sep 01 2020 at 22:29):
Cool. These seem like ideas worth pursuing.
Grahame Grieve (Sep 01 2020 at 22:30):
note, though that this brings me to a different problem - there's a set of QA requirements for publication that are not imposed for regular builds, and we typically have to do a round of editing to finish those issues off. And as I understand this workflow, tagging is not a proposal to start that workflow, but a declaration it's finished. So I'm nervous about sorting that workflow out
Grahame Grieve (Sep 01 2020 at 22:30):
at least I can probably fail the build if they're not met and the tag is passed to the IG publisher.
Elliot Silver (Sep 01 2020 at 22:35):
Definitely seems like ideas worth pursuing.
I wonder if there is a better way than a zip (or at least a way for the process to get notification of the availability of the zip), so that the content can be auto published to a website?
I've seen various QA tasks, like security analysis, included in workflows. If the workflow fails, publication fails. You could also set up publication to require the presence of a particular file or similar that flags that the review has been done.
Elliot Silver (Sep 01 2020 at 22:37):
I don't know what happens if you label a commit as "R6" but then decide that the commit has errors. I suspect you want to do all the touch-up etc. and validate before doing the label.
Grahame Grieve (Sep 01 2020 at 22:42):
so that the content can be auto published to a website
Let's get to that later. It's a big step for me given how much work we have to do around getting things right for publication
Grahame Grieve (Sep 01 2020 at 22:43):
very few editors at all understand package-list.json, for instance. but that's critical to the actual publishing process
Grahame Grieve (Sep 08 2020 at 20:39):
so @Josh Mandel and @Elliot Silver I just talked to @Mark Iantorno about this, who runs the build pipelines for all the java code and will eventually take over all the build processes for the FHIR stuff.
We don't think that a tag based process works for this; the problem is that fundamentally you're taking an action ("check and publish this version"), and the tag is the outcome ("we just published this version"). So we think that the right way to set this up is to create a pipeline in (e.g.) azure that runs the checks and executes the publishing, and that tags the release as part of the process.
Elliot Silver (Sep 08 2020 at 20:56):
OK. I don't know enough of the details of doing automated deployments to know how others handle this. It seems like it should be possible, but I'll defer to you and Mark.
Josh Mandel (Sep 08 2020 at 23:27):
Sure, either of these is good, and if you get your tag semantics right they're not exclusive. For example, you might have a tag meaning "this is a candidate for 1.2 publication" and the pipeline can pick it up, run checks, publish, and apply a tag meaning "this is memorialized as version 1.2"
Elliot Silver (Sep 09 2020 at 03:11):
@Josh Mandel I'm still running the webhook in parallel with my action. My action is triggered by a commit on the master branch, and ends up committing the generated IG on the gh-pages branch. The webhook build seems to watch for a commit on any branch, and is thus doing a build on master and a build on gh-pages for every commit. The build on gh-pages will obviously fail, since the content isn't IG source. Is it possible to get the webhook auto-build to abort if, for example, there is a "no-autobuild" file in the branch root directory?
Josh Mandel (Sep 09 2020 at 03:13):
I assume your desired behavior is to only build the master branch, and not other random branches that would overwrite the master branch content?
Josh Mandel (Sep 09 2020 at 03:14):
https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#onpushpull_requestbranchestags shows how you can specify one or more branches explicitly; just leave the GitHub pages Branch out of this list
Elliot Silver (Sep 09 2020 at 03:15):
Well, "only build the master branch" is equivalent in my project to "not build the gh-pages branch" since I only have two branches. I'm not so worried about overwriting the content, as I am ...
Elliot Silver (Sep 09 2020 at 03:15):
... about triggering an auto-build with known content that will fail.
Josh Mandel (Sep 09 2020 at 03:15):
Fair enough. You can have a policy of defaulting to "always build" and skipping the pages branch or default to "never build" and only building if it's master
Elliot Silver (Sep 09 2020 at 03:16):
That link is for a workflow actions I'm authoring. I can deal with that.
Elliot Silver (Sep 09 2020 at 03:17):
It's the webhook that calls https://us-central1-fhir-org-starter-project.cloudfunctions.net/ig-commit-trigger that I have no control over.
Josh Mandel (Sep 09 2020 at 03:17):
Ohh, I understand what you're talking about now. You don't want to have the centralized Auto builder try to build your gh-pages branch.
Elliot Silver (Sep 09 2020 at 03:18):
Exactly.
Josh Mandel (Sep 09 2020 at 03:18):
Hopefully this is not causing you any actual problems, but I agree it would be nice to provide some way to opt out.
Grahame Grieve (Sep 09 2020 at 03:19):
in general you wouldn't to build the gh-pages branch but I think people starting out would find simply being ignored because their configuration is wrong deeply disorientating
Elliot Silver (Sep 09 2020 at 03:19):
No, not causing problems. Just an extra failed notification in committers/notification.
Elliot Silver (Sep 09 2020 at 03:20):
@Grahame Grieve agree, which is why I suggested the explicit act of adding a file to the root directory a branch you didn't want built.
Josh Mandel (Sep 09 2020 at 03:21):
Cool. If you're feeling adventurous, the place to add logic is at https://github.com/FHIR/auto-ig-builder/blob/master/images/ig-build/builder/builder.py#L39 -- otherwise if you don't mind creating an issue on the repository to track this, that would be lovely.
Elliot Silver (Sep 09 2020 at 03:22):
Thanks. Let me take a look.
Josh Mandel (Sep 09 2020 at 03:22):
I might just do an explicit filter to test for the GitHub pages Branch rather than a special "no build" file, to start.
Brian Postlethwaite (Sep 15 2020 at 11:59):
I'm late picking this up too, I have 3 internal projects setup now with CI through publishing in Azure using docker images for build and some custom release actions.
In my case each official release ends up with the output in git too under a versions folder.
The release prep step runs through all builds and updates the links based on package-list.json I assume like the hl7 ones do.
Brian Postlethwaite (Sep 15 2020 at 12:00):
To get the output from the docker image, I mount a folder into it for both the package cache and output folder.
Brian Postlethwaite (Sep 15 2020 at 12:01):
I should write this up on my public blog too.
Brian Postlethwaite (Sep 15 2020 at 12:03):
This is one of the guides produced by it.
https://fhir.medicationknowledge.com.au/
Saul Kravitz (Sep 16 2020 at 14:24):
@Elliot Silver Two questions:
1) Is the docker approach faster than the alternative of installing jekyl and downloading the ig publisher jar on each invocation?
2) Does your recipe build all branches (similar to the HL7 build system), which I've found to be very useful.
Elliot Silver (Sep 17 2020 at 22:56):
@Saul Kravitz Sorry for the delay responding.
- My Docker approach uses the image Josh has prepared, which already has jekyl in it, and it downloads the IG publisher each time it is run. I think it is slower than the HL7 auto-build, which I believe stays alive between invocations.
- The action I wrote only looks for commits to the master branch, but could easily be extended for any branch, with the caveat that the output of the build is committed to the gh-pages branch, so you don't want changes to that branch to trigger build.
Josh Mandel (Sep 18 2020 at 02:50):
I think it is slower than the HL7 auto-build, which I believe stays alive between invocations.
Nope -- they're both based on docker containers that spin up just-in-time. Both download the publisher jar every time, too.
Josh Mandel (Sep 18 2020 at 02:51):
(The auto-build version is on a much beefier VM than Github Actions offers by default, so it probably does run faster...)
Josh Mandel (Sep 18 2020 at 02:51):
output of the build is committed to the gh-pages branch,
It might be a neat enhancement to make branch builds get committed to subdirectories of the gh-pages build, to preserve content.
Elliot Silver (Sep 18 2020 at 18:13):
It might be a neat enhancement to make branch builds get committed to subdirectories of the
gh-pagesbuild, to preserve content.
Agree. However the action I'm leveraging doesn't support that right now. There is an existing Github (issue)[https://github.com/peaceiris/actions-gh-pages/issues/455] for it, and I've added our use case, but I don't know what sort of development schedule they are on.
Last updated: Apr 12 2022 at 19:14 UTC
