Stream: committers
Topic: IG retention policy for build.fhir.org
Josh Mandel (Sep 17 2020 at 15:04):
Right now we hold on to built branches forever; we replace a built branch whenever new content is committed to that branch on github. I want to make sure we don't run out of storage; for now I'll just increase our storage so we have headroom. But in general, what requirements do folks have for retaining old builds, and what should we try to do here?
Josh Mandel (Sep 17 2020 at 15:04):
I'd like to have a policy for IG builds similar to core FHIR builds -- where we delete builds for branches that have been inactive for >XX days, where "XX" is some window we agree on (e.g., 90).
Jose Costa Teixeira (Sep 17 2020 at 15:08):
90 days for "my" branches seems good. Do you want to add a way to force stale branches to be kept?
Josh Mandel (Sep 17 2020 at 15:13):
Re: indefinite retention, I kind of don't mind having some friction here (like, maybe build.fhir.org is the wrong place if you need indefinite retention for some builds, and there are other options like Github pages, etc). Would like to hear if folks here would find this overly constraining, though.
David Pyke (Sep 17 2020 at 15:15):
I'm not sure why we would need infinite retention on Build. If you haven't STU'd it in a year, it gets archived. If you want it back up, push a change.
Jose Costa Teixeira (Sep 17 2020 at 15:30):
I can imagine that some branches are not picked up within that time, and this would be a way to say "we are stiiiiiill working on it, see lates t branch there"...
David Pyke (Sep 17 2020 at 15:32):
If you've not updated or STU'd in a year then "still working on it" is a misnomer
Jose Costa Teixeira (Sep 17 2020 at 15:34):
(i was thinking of 90 days - for me it's a fine deadline, but time not being a linear concept esp. in pandemic season...)
Josh Mandel (Sep 17 2020 at 15:34):
So... maybe a 365-day window?
David Pyke (Sep 17 2020 at 15:34):
THat's my thought
Jose Costa Teixeira (Sep 17 2020 at 15:35):
Anyway, I just mentioned the possible stale branches as an edge case. I'm happy with 90 or 365 days
Josh Mandel (Sep 17 2020 at 15:37):
It's always possible to re-trigger / reset the retention window with a trivial commit on the branch
David Pyke (Sep 17 2020 at 15:37):
Right, but inactivity for 365 gets it archived. Did we want to do it at the branch level or just for whole projects?
Lloyd McKenzie (Sep 17 2020 at 15:38):
I think people would be more comfortable with a shorter timeframe if there was an emailed warning of impending expiry. However, agree w/ David that if you want semi-permanent hosted content, build.fhir.org shouldn't be your solution.
Josh Mandel (Sep 17 2020 at 15:44):
Branch level makes sense to me in terms of storage requirements; active projects might have hundreds of branches over the course of a year, each of which takes up hundreds of megs for output storage.
David Pyke (Sep 17 2020 at 15:45):
Maybe we do branches at 90 days, projects at 365?
Josh Mandel (Sep 17 2020 at 15:45):
I kinda like that
John Moehrke (Sep 17 2020 at 22:57):
I had thought that it was 90 days.. I recall builds being flushed, was that last year? things change so fast 90 seems shortest possible, 365 does not feel too long. so anywhere in between seems right
John Moehrke (Sep 17 2020 at 22:58):
when this flush happens, a note to the committers/notifications would be 'nice'
John Moehrke (Sep 17 2020 at 22:58):
I wonder if 14 day pre-flush notification might be useful... giving people a chance to quickly grab it before it is gone
Josh Mandel (Sep 18 2020 at 01:22):
We do clean up branches for the core fhr specification, but we don't clean up the branches for implementation guide builds. (We only started building branches for implementation guides relatively recently; before that only the master branch was auto-built for IG repositories.)0
Josh Mandel (Sep 18 2020 at 01:23):
It would be nice to have a consistent policy for the core build as well as IGs, just so editors participating in both camps don't have to keep different models in their heads
Kevin Power (Oct 15 2021 at 23:03):
@Josh Mandel - Sorry if missed another update, but I happened upon this old thread. Was an automated clean up of old/stale branches for IGs ever implemented? For our IG, we have several old branches still hanging around.
Josh Mandel (Oct 15 2021 at 23:16):
No, I haven't pulled the trigger on this. (If this is causing an issue and you want me to manually delete anything just let me know.)
Kevin Power (Oct 15 2021 at 23:31):
No specific issues right now, just wanting to help eliminate possible confusion down the road (if someone mistakenly looks like the old branch without realizing it). Let me ask around on my side and see if we want to do a manual delete at some point. Thanks @Josh Mandel
Kevin Power (Oct 18 2021 at 20:35):
@Josh Mandel - If you don't mind, can you delete the following from http://build.fhir.org/ig/HL7/genomics-reporting/branches/ :
jrj-jira-pending
kp-improve-names
kp-loinc-copyright
kp-qa
kp-simple-updates
kp-task-category-changes
kpower_SomaticImplication
Josh Mandel (Oct 18 2021 at 20:51):
Done
Kevin Power (Oct 18 2021 at 20:54):
Thanks Josh
Josh Mandel (Jan 29 2022 at 18:32):
Per discussion here, I'm going to start enforcing "IG branches >90 days old will be cleared from the build server".
Josh Mandel (Jan 29 2022 at 18:32):
Any new commits will revive them...
John Moehrke (Jan 29 2022 at 20:46):
did this cleanup kill things like templates -- ihe.fhir.template -- that really don't change much?
Josh Mandel (Jan 29 2022 at 21:36):
If they're on the build server in stale branches, yes. That "shouldn't" cause problems, but if it did we can:
-
Trigger a build on any of this content via https://fhir.github.io/auto-ig-builder/builds.html (looks like you may have already done so for IHE templates
-
Figure out a process to ensure this server isn't used for dependencies in the future (Grahame I think has a way to use his package server for dependencies, instead of build.fhir.org)
John Moehrke (Jan 30 2022 at 00:00):
I rebuilt the template, then rebuilt my IG... all better now.
Elliot Silver (Jan 30 2022 at 03:31):
@Josh Mandel , is that "all branches will be cleared," or all except main/master?
Josh Mandel (Jan 31 2022 at 02:26):
All branches; the expectation is that the CI server is a place to review active work (whether "main" or otherwise) -- but of course we can revisit this.
Lloyd McKenzie (Jan 31 2022 at 03:07):
I think that the main/master branch should always remain. We link to it as the CI-build from the archive pages. We don't want that to go 'poof'. The only time we might consider removing those is if we retire the spec.
John Moehrke (Jan 31 2022 at 13:19):
I think some main/master branches are experiments, and not formal projects. Somehow we need to identify the formal ci-builds, or give experimenters a way to tell us that they want their experiment to be removed.
Josh Mandel (Jan 31 2022 at 14:10):
There's a long tail of experiments indeed.
John Moehrke (Jan 31 2022 at 14:12):
I also thought about some of my experiments were to point out an issue with IG builder or templating. Needed them to build on ci-build to be visible, but they certainly don't need to live forever. (Presumably fragments become validation examples for IG builder).
Josh Mandel (Jan 31 2022 at 14:12):
I think that the main/master branch should always remain. We link to it as the CI-build from the archive pages. We don't want that to go 'poof'. The only time we might consider removing those is if we retire the spec.
Maybe this applies to HL7 published specs? We could perhaps have a query to a package server to fetch a list of those. Even so, if a spec is not under active development, the "continuously" integration doesn't have much meaning
John Moehrke (Jan 31 2022 at 14:12):
it also applies to others (IHE for sure)
John Moehrke (Jan 31 2022 at 14:14):
maybe the auto-ig-builder dashboard just needs a "delete" button? hmm, the only danger is someone deletes a ci-build that is needed, for which that ci-build gets rebuilt by someone in-the-know.
John Moehrke (Jan 31 2022 at 14:14):
yeah, bad idea for a security co-chair to be recommending that solution.
Josh Mandel (Jan 31 2022 at 14:15):
So maybe it applies to published specs; again this takes us back to the discussion about dependencies on CI: we'd need a formal mapping from package names to GitHub projects (e.g., in the package server -- do we population repository in package.json?) ( the CI system can't "trust" values it sees in repos, since anyone could create them.. )
John Moehrke (Jan 31 2022 at 14:16):
so the simplifier registry should have that
Josh Mandel (Jan 31 2022 at 14:17):
I'd like to discuss whether a well crafted 404 page on build.fhir.org could help us avoid perceived requirements here. We csn take this up in FMG this week perhaps?
John Moehrke (Jan 31 2022 at 14:21):
drat, there is no linkage. I thought that the repo ended up in the published IG json.
John Moehrke (Jan 31 2022 at 14:22):
it is somewhat indirectly recorded in the IHE IGs, as we use the somewhat undocumented input/data/features.yaml to enable github based issue submission.
John Moehrke (Jan 31 2022 at 14:40):
I would expect there would be some github repo homes that we could itemize like HL7 and IHE? I would suspect in those there is no throwaway examples of important?
Last updated: Apr 12 2022 at 19:14 UTC
