Stream: IG creation
Topic: Primary branch name
Max Masnick (Oct 21 2020 at 11:28):
Git and GitHub are considering moving from master to main for default branch names (new repos on GitHub are already using main rather than master).
Are there technical issues that would prevent existing IGs from switching primary branch names? For example, does the build.fhir.org CI hard-code master as the primary branch, or is it deriving this from the GitHub repo somehow?
Lloyd McKenzie (Oct 21 2020 at 15:36):
@Josh Mandel @James Agnew
Josh Mandel (Oct 21 2020 at 16:12):
For the auto-build pipeline: https://github.com/FHIR/auto-ig-builder/search?q=master gives you a quick sense of what's hard-coded. Summarizing...
- For IGs, we delete all "stale" branches after 3 weeks, except
masterwhich lives indefinitely - For https://fhir.github.io/auto-ig-builder/builds.html, we only show
masterbranches on the dashboard - For https://build.fhir.org/ig/:org/:repo URLs, we display the
masterbranch by default, if no "/branches/:branch" path suffix is provided
Josh Mandel (Oct 21 2020 at 16:18):
Bottom line: If you create a new repo with a main instead of master, it should work right now; but you'll have to send people to https://build.fhir.org/ig/:org/:repo/:branches/main to see it.
Max Masnick (Oct 21 2020 at 19:34):
Thanks!
Lloyd McKenzie (Oct 21 2020 at 19:48):
Should we update our code to handle both master and main as equivalent?
Max Masnick (Oct 21 2020 at 20:47):
@Lloyd McKenzie given that new GitHub repos are already defaulting to using main as the primary branch, I think that it makes sense to have build code assume main is the primary branch if it exists, and otherwise fall back to master.
Max Masnick (Oct 21 2020 at 20:47):
(For example, I just created this repo via the github.com interface: https://github.com/masnick/test-repo. It has one branch called main.)
Josh Mandel (Oct 22 2020 at 15:09):
Should we update our code to handle both master and main as equivalent?
Yeah, I'd love it if someone else wanted to come up to speed on the auto-build infrastructure. Right now I think I'm the only person familiar with it, which isn't great because:
- It lets me write idiosyncratic, hard-to-understand stuff, and
- It means my availability places a hard limit on speed of development
So... consider this an open call for anyone who wants to co-maintain this infrastructure with me. There's a fair amount of low hanging fruit that'd make it easier to use/maintain over time (which is a polite way of saying that today it's... ad-hoc.)
Josh Mandel (Oct 23 2020 at 16:08):
OK, so tentative plan...
- When building an IG, determine if the current branch is "the default branch per github"
- If not, no special processing;
- If yes, symlink a "_ghDefaultBranch" directory on the build server to this branch
Then, the web hosting logic, the dashboard, etc, can be driven based on the "_ghDefaultBranch" symlink (which will point to main or master or default or whatever people choose as their default branch in github).
Josh Mandel (Oct 23 2020 at 17:29):
I'm taking the first step (which should have no visible effect) with this commit. Will continue with the plan next week if this steps works OK.
Josh Mandel (Oct 23 2020 at 17:41):
(Please @-mention me here if you see anything unusual.)
Vassil Peytchev (Oct 27 2020 at 19:28):
@Josh Mandel How do I get up to speed with the auto-build infrastructure?
Josh Mandel (Oct 27 2020 at 19:33):
I love this question @Vassil Peytchev ;-) What I'd propose if you're willing would be we schedule a video chat to talk through what's there -- and ideally we record it, and see if we want to use that as a starting point for writing up additional docs.
Josh Mandel (Oct 27 2020 at 20:03):
I just made https://docs.google.com/presentation/d/12JykZwSdQ1pwSuzP2fGZSXr3jYMmvEcwVgNAy3dWr_U/present?slide=id.p with a quick overview of the build infrastructure
Josh Mandel (Oct 27 2020 at 20:19):
(permissions fixed)
Vassil Peytchev (Oct 27 2020 at 20:24):
It all fits on one slide, can't be that hard...
:dragon: :dragon: :dragon_face: :dragon: :dragon:
Grahame Grieve (Oct 27 2020 at 21:35):
@Josh Mandel while you are looking at this, I have a need for a secret - an api key - to be stored in a secure location, and passed to the IG publisher as a parameter. I'll provide the key, but it can't be public. Is that doable?
Josh Mandel (Oct 27 2020 at 21:44):
It's sorta doable. It depends on your threat model -- anyone on the sushi team, or anyone able to push the ig publisher jar can obviously cause this to leak. A clever ig author might also be able to cause it to leak. (This is similar to how we protect access to the "publish" script today with a publication keypair; a dedicated attacker could probably exfiltrate it.)
Grahame Grieve (Oct 27 2020 at 21:45):
why on the sushi team?
Grahame Grieve (Oct 27 2020 at 21:45):
An IG Author won't be able to cause it to leak.
Chris Moesel (Oct 27 2020 at 22:11):
I don't know for sure, but I think @Josh Mandel probably said anyone on the SUSHI team could leak it because SUSHI gets automatically loaded from NPM and is run inside the CI environment. So anything accessible in the CI environment (via file system, environment variables, etc) is accessible to SUSHI code -- and therefore to nefarious SUSHI developers. But don't worry, you can trust us. ;-)
Josh Mandel (Oct 27 2020 at 22:15):
That's what I meant, yes. And the surface area of the IG Publisher's inputs is huge, so I'm pretty confident that a determined IG author could cause arbitrary code execution in the "IG Build" docker image.
Grahame Grieve (Oct 27 2020 at 22:17):
ok. I don't think the sushi team can get to the IG publisher memory space or parameters but agree they have access to the system. I'm not concerned about that.
IG authors cannot execute arbitrary code. Template authors can. I already have sufficient control over that.
Josh Mandel (Oct 27 2020 at 22:19):
IG authors cannot execute arbitrary code. Template authors can. I already have sufficient control over that.
It's really hard to prove something like that. Maybe what you mean is you don't intend for IG authors to execute arbitrary code :-)
Grahame Grieve (Oct 27 2020 at 22:20):
I have taken what steps I know to take, but agree it's hard to prove.
Grahame Grieve (Oct 27 2020 at 22:21):
they can't run scripts. They can't register plug-ins with Jekyll. They don't have any loaded html space to play with javascript
Grahame Grieve (Oct 27 2020 at 22:26):
although I'm not really worried about ig authors in actual fact. if they can really figure out how to get the API key, then they're welcome to it...
Josh Mandel (Oct 27 2020 at 22:29):
Do keep in mind that anyone in the world can be an IG author just by calling our webhook
Grahame Grieve (Oct 27 2020 at 22:30):
I very much keep that in mind.
Josh Mandel (Oct 27 2020 at 22:32):
Okay, so s/ig authors/anyone/ in the statement above
Grahame Grieve (Oct 27 2020 at 22:32):
anyone really technically capable, yes. And if they're really dedicated to spend that much to acquire an API Key that they can get for free and only gives them access to data that's publicly available... sure.
Josh Mandel (Oct 27 2020 at 22:33):
Anyway, we can definitely call the publisher with an API key. Where/how will that be used downstream?
Grahame Grieve (Oct 27 2020 at 22:34):
I will use it to access NLM terminology services. People using the IG publisher locally who use NLM resources will have to provide their own key.
Josh Mandel (Oct 27 2020 at 22:34):
Perfect -- more NLM silliness causing people to jump through hoops to access public data :(
Josh Mandel (Oct 27 2020 at 22:35):
Anyway that makes me feel much better about the risks of leakage
Grahame Grieve (Oct 27 2020 at 22:35):
yep that's exactly what it is. You should read the API - 3 levels of keys, including getting a new API key before every single transaction.
Grahame Grieve (Oct 27 2020 at 22:36):
this is fort knox levels of security.
Josh Mandel (Oct 27 2020 at 22:36):
I've read it and used it Alas, many years ago. So painful
Josh Mandel (Oct 27 2020 at 22:36):
Enough to make me never use it again :-)
Josh Mandel (Oct 27 2020 at 22:36):
And I did share this feedback at the time of course.
Chris Moesel (Oct 28 2020 at 14:58):
I've been told that the NLM FHIR interface should support the API Key and intentional value sets by the end of the year. Then I think you should be able to ditch using the SVS API.
Josh Mandel (Oct 28 2020 at 15:54):
Re: single-slide overview, I've updated it with links to the relevant bits in the auto-ig-builder repo (FYI @Vassil Peytchev)
Grahame Grieve (Oct 28 2020 at 19:41):
I'm not using the SVS api
Chris Moesel (Oct 28 2020 at 21:06):
3 levels of keys, including getting a new API key before every single transaction
That's CAS authentication. Their SVS API uses CAS authentication. Do they have another value set API (besides SVS) that also uses CAS? AFAIK, their FHIR API only uses Basic Auth at this time.
Grahame Grieve (Oct 28 2020 at 22:55):
that's changing
Josh Mandel (Nov 05 2020 at 17:11):
For anyone interested in details of the auto-build infrastructure, @Vassil Peytchev and I did a deep dive yesterday: https://youtu.be/VVbF1O4pgQA
Lloyd McKenzie (Dec 01 2020 at 16:45):
@Vassil Peytchev @Josh Mandel Any insight into when the CI build tool will handle 'main' the same way it handles 'master'?
Vassil Peytchev (Dec 01 2020 at 16:49):
It won't be for another probably two weeks from my side. I haven't had a chance to do any changes yet.
Last updated: Apr 12 2022 at 19:14 UTC
