Stream: fhir/infrastructure-wg
Topic: Minimizing backlog going forward
Lloyd McKenzie (Feb 24 2022 at 20:37):
FHIR-I currently has the biggest backlog of unapplied issues of any of the work groups. I've been thinking about how we might avoid that in the future. I think the main reason for the backlog is that no-one is on the hook to 'apply' a change once it's agreed - and we don't even necessarily think about the 'size' of the work involved in applying a change. I'm wondering if it might make sense for us to establish a rule that we don't agree to make a change without identifying someone who agrees to make the change - and assigning it to them as soon as the resolution is stored. We can then check the backlog each call, or even each month, of "assigned but not yet committed" issues that are older than, say, 4 weeks and figure out whether we need to re-jig who's responsible.
Does that seem workable? Obviously it doesn't make the current backlog disappear, but it might help with ensuring it doesn't get so big in the future.
Josh Mandel (Feb 24 2022 at 23:10):
I like this line of thought. In my ideal world, change proposals would come with PRs that we'd review and accept. Or they'd lead to PRs that we reviewed before accepting. I understand there are lots of reasons why that's challenging, but I think it'd be a good goal in many cases.
Also a corollary: for us to be productive, we need agreement about the "right" answer and someone who cares enough to do the work. If we don't have both, it might be worth closing issues or moving them into an "insufficient interest" state. Again, I understand there are lots of reasons why that's challenging, but we need to be realistic about what we can get done.
Jean Duteau (Feb 24 2022 at 23:23):
I agree with this and think we might even extend it to all workgroup FHIR issues. I've been getting @Marc Duteau to do the M&M tickets and there are a number where he comes back to me to say "the group obviously knew what it meant when it wrote this resolution, but I don't". I then look at it and don't know either. Having a definite PR or at least a well thought out disposition along with assigning it to someone to get it done would help across the organization.
Marc Duteau (Feb 24 2022 at 23:32):
Yeah, I’d agree that having a definite PR would be the ideal but without that at the very least having much more descriptive dispositions and having someone on the hook for them is essential.
Lloyd McKenzie (Feb 25 2022 at 00:04):
I expect most people who submit issues won't have the skill level to create PRs. And I'm not sure we have to have a PR to resolve an issue. But we should definitely not resolve an issue without the agreed wording being present or a description of the technical changes that's sufficient for someone to just go and make the change. And agree that if we don't have someone willing to make the change, that's a sign that the change isn't super important (or that the juice isn't worth the squeeze). Also, if there's an expectation that someone is signed up to apply the change on the day it's approved - and they'll get nagged by the group if it's not done in 4 weeks, the context of the change should still be fresh enough in everyone's mind that if there's an issue and the tracker needs to come back for discussion, we won't be starting at ground zero.
Corey Spears (Feb 25 2022 at 01:16):
Is there any room for volunteers to help out?
Lloyd McKenzie (Feb 25 2022 at 03:14):
Absolutely
Vassil Peytchev (Feb 25 2022 at 20:01):
Currently for the ballot reconciliation of the IPA IG , we are following a process where no issue can be voted/resolved if a change is not applied. Is it possible to have a new flavor of "assigned" that doesn't result in a "Feedback requested" status?
Lloyd McKenzie (Feb 25 2022 at 20:03):
All things are 'possible', but that one's not easy. When you say "change is not applied", do you mean pull request is created or what? If you've assigned the issue for someone to create the pull request, that'd still technically be "waiting for input" - it's just 'technical' input rather than 'business' input.
Gino Canessa (Feb 25 2022 at 20:58):
I am wondering if there also needs to be a process change to avoid spending committee time on tickets that nobody will work on. It is not a problem for tickets that get 'nominated' for review manually, since the person asking is a logical assignee, but I have doubts about the general 'oldest open ticket' view. If the process is 'find ticket, read, understand, ask if someone is going to work on this', it is not much less time than the current voting process.
For reference, my thinking was a possible process of defaulting to the 'mover' being on the hook for resolution (don't move things you aren't willing to do! =). But I realized that generally people would just... not make motions. In that scenario, it would be fewer steps and less time wasted to mark everything 'not interesting' and only take up issues that people move out of that state.
After more reflection, I think an issue is that a lot of tickets do not meet the bar of 'interesting enough to volunteer doing' - similar to how it is easier to find volunteers to hand out refreshments than to pick up garbage. Would that be something HL7 could fund (e.g., assign to 'someone getting paid to fix this type of thing because nobody else wants to')?
I think there is also a related issue that people ( :raised_hand: =) do not have the bandwidth to consistently troll the open ticket lists and try to sort/find tickets - work I would put under the Project Manager umbrella.. I guess it leads to the same sort of question in my mind - if the process is failing today because people are not volunteering to do it, does it need to be a non-volunteer role?
(note: been sitting on this a while before sending because I don't love the direction/conclusions I have... but posting anyway in case it helps crystalize anyone else's thoughts or that someone's response can get my thoughts unstuck =)
Jean Duteau (Feb 25 2022 at 21:13):
My company has funded a resource to go and implement tickets for MnM and Vocabulary. I think that HL7 has funded resources in the past for other tickets. So funding does help things go.
John Moehrke (Feb 25 2022 at 21:16):
@Gino Canessa that is kind of how THO works
John Moehrke (Feb 25 2022 at 21:17):
many times people put in a CR becasue they don't know how to solve a problem. Might have multiple solutions, and want the committee to choose the one the committee woudl recommend
Gino Canessa (Feb 25 2022 at 21:19):
Yep. I know that I also file tickets that I just generally notice when arbitrarily looking something up. For those tickets, I am not particularly interested in fixing them myself, just trying to be a good citizen and let someone know there was an issue. If I was on the hook for fixing everything I submitted, I would think twice about submitting issues (and three times for some WGs).
John Moehrke (Feb 25 2022 at 21:22):
I think the presumption that we will get backlog to zero every ballot is a fallacy. but I think we should have a state for a ticket that is close to rejected, but just shy of rejected. It stays in that state accumulating votes (or linkages to dups).
John Moehrke (Feb 25 2022 at 21:23):
the watching flag that jira has I use to pile-on to an issue I think is important. We could make this concept of voting up/down much more easy and visible.
Lloyd McKenzie (Feb 25 2022 at 21:36):
I'm not sure it's true that most of our tickets don't meet the bar of 'interesting enough to volunteer doing'. We just haven't had a previous process of seeking people to do them at the time of resolving them. Thus they all just sat there on the presumption that 'someone' (likely Grahame) would get to them eventually.
The lack of triaging tickets is likewise a process issue. If it's not identified as a) a prioritization, and b) an expectation for certain people to do the job, then "triaging on calls" becomes the default.
I think we have enough interested & technically capable people on the FHIR-I calls that we could keep up - if we have the right processes in place.
My proposal would be something like this:
- Have an agreement amongst the co-chairs to perform triage of FHIR-I items on a regular basis (every 2 weeks? every 4 weeks?). Triage would look for issues that can have resolutions pre-defined, make sure that change request vs. technical correction is properly set, distinguish tooling from non-tooling and prioritize based on broken/enhancement/normative-impacting/whatever and (if determinable), identify the amount of effort/skill type needed to apply the proposed change.
- Our review filter would ensure we only look at items that are triaged, based on priority.
- We review all issues, however before we can vote to approve something we would need the resolution completely clear in terms of the wording or resource change (no hand-wavy stuff) and have a volunteer to make the change.
- If we have agreement on appropriateness of a change, we can tag those issues as "needs-volunteer" and let them sit for a month or two. If there are no takers in that period, we can resolve as not-persuasive
- We review the list of items that have been assigned and no updates in 4 weeks as part of our call to see if something needs to be re-assigned or re-opened and marked as "needs-volunteer".
The biggest issue is making sure we continue to stay on top of things. If we stay on top, triaging shouldn't take more than an hour a month and, amongst the 10 or so regular participants we have that know how to apply changes, probably not more than 1-2 hours outside calls per month to apply changes. It's possible that there'll be the odd major change that involves a mass change of resources and/or significant tooling work that will take some time to get to, but at least then those will be known 'BIG' items that can be scheduled by the volunteer and that we can monitor to see if they're happening when scheduled or not.
I'm certainly not suggesting that funding doesn't help. It may well be that some "volunteer-needed" issues will get assigned because the 'volunteer' who picks them up ends up being paid to do so by someone who has the incentive to make it happen.
Lloyd McKenzie (Feb 25 2022 at 21:38):
I'm not super-fond of the notion of voting an issue up or down. Say some organization puts 20 up-votes on an item - who cares? If they're not willing to fund/do the work and no one else could be bothered, it's not going to happen.
And we're not talking about getting backlog to zero. But I would like to get "volume of FHIR-I assigned issues that haven't been updated in 2+ months" to zero. Everything that comes in should end up either applied or rejected or punted to 'future use' in no more than 6 months. I think that is fully achievable with the right processes and discipline.
Gino Canessa (Feb 25 2022 at 21:51):
Fair enough, though I think we also need to note that a better phrasing of 'interesting enough to volunteer doing' would be 'interesting enough to volunteer to setup the build environment locally, spend some time finding the correct source file, make a change that takes a few seconds, wait for a build that takes up to half an hour, submit a PR, wait for checks that can take another half hour, and then merge the PR' ;-).
A bit tongue-in-cheek, but I think it is relevant that even 'simple' tickets (e.g., fixing wording) still take a significant amount of time to apply. The people that have all of that working and good processes/intuitions about batching changes are often the same people who are already swamped trying to work through items.
Vassil Peytchev (Feb 25 2022 at 22:04):
I never realized that when a ticket is in a a proposed resolution status you can assign it, and it goes into a "waiting for input" state. Is it possible to distinguish when "waiting for input" is because of assignment to apply changes vs. when a clarification about the issue is needed?
As to what does it mean to have the change pre-applied, I think a PR needs to be available, so we can look at the branch output and agree that the proposed resolutions is being met.
Lloyd McKenzie (Feb 25 2022 at 22:09):
The "set up the build environment" is, in theory, a one-time activity, though agree there can be occasional maintenance. You don't actually have to run a full local build before submitting a PR if you're confident in your fix. If you've messed with spreadsheets, you only need about the first 5 minutes of the build to re-gen the spreadsheets, at which point you're good for a commit. You can also make a PR with commits for a bunch of separate changes. I'm not saying the build process is 'pleasant', but it's manageable.
@Vassil Peytchev - expectation is that when something's assigned, there's a comment that explains what is being asked of the assignee. We could define a standard Grouping though if there was a desire.
I agree that in some cases requiring a PR before voting to approve a change makes sense. But I wouldn't want to put that in place as a general expectation.
Elliot Silver (Feb 25 2022 at 22:10):
Isn't this discussion broader than FHIR-I? I've got issues that have spent months or a year in triaged; or months or years in "resolved - change required" in various WGs. Perhaps this should be a FMG-level discussion, not just FHIR-I.
Elliot Silver (Feb 25 2022 at 22:12):
Gino Canessa said:
'Interesting enough to volunteer to setup the build environment locally, spend some time finding the correct source file, make a change that takes a few seconds, wait for a build that takes up to half an hour, submit a PR, wait for checks that can take another half hour, and then merge the PR' ;-).
A bit tongue-in-cheek, but I think it is relevant that even 'simple' tickets (e.g., fixing wording) still take a significant amount of time to apply. The people that have all of that working and good processes/intuitions about batching changes are often the same people who are already swamped trying to work through items.
Don't forget step 0: request a system from IT beefy enough to do a build on, and justify it by saying you need it for volunteer work.
Lloyd McKenzie (Feb 25 2022 at 22:18):
My leaning is to try out a system with FHIR-I (seeing as we appear to have the biggest problem). If it works for us, the TSC could look at encouraging it to be used elsewhere. However, different WGs have variable capacity. With our WG, we have the benefit of 10-ish folks who can reasonably be asked to help apply changes. For some work groups, that's only 2 or even just one. (i.e. have a big enough machine and enough comfort with Git, the build process, messing with the technical artifacts, etc.) So "is there a volunteer" defacto becomes "Does Mary have time/interest?" That might still be the right question, but it's a more problematic one.
Eric Haas (Feb 28 2022 at 19:45):
I like Josh's solution shift the burden to commenter for CRs by forcing a PR. It would put the commenter's skin the game and dramatically reduce the number of trackers. - problem solved !
Lloyd McKenzie (Feb 28 2022 at 19:50):
I strongly object to that notion. We want everyone in the world to identify things that need to be fixed or could be better. We want the barrier to identifying issues to be as low as possible. (Some would argue that needing to sign up for Jira and fill in the mandatory elements is already too high.) There is no way we can expect most of the community who have issues to:
- understand how to submit Git PRs
- understand how we organize our source
- set up a build environment
- go through the steps of making the change and submitting a PR.
If someone wants to submit a PR, that's great - though given that the agreed resolution after discussion is often different anyhow, having the PR doesn't necessarily save much time.
Creating the PR is something that should fall on the work group - they're the ones who have the obligation to make the change and who have the understanding of how we maintain our source.
Yunwei Wang (Feb 28 2022 at 19:55):
I would like to set a cutoff date. Any tickets that are not applied before cutoff date will be automatically move to next release.
Yunwei Wang (Feb 28 2022 at 19:58):
Not applied ticket will be drop as "not needed" after certain time frame because apparently no one cares about the resolution.
Lloyd McKenzie (Feb 28 2022 at 19:59):
That's also a bit problematic. If we hit something that's badly broken and going to cause us grief in the next release, then we should fix it. What might be better is a bias toward "considered for future use" on new new stuff unless we believe it's really important to have in R5.
Lloyd McKenzie (Feb 28 2022 at 20:00):
So enhancements and clarifications would be "considered for future use" unless someone can make the case for including them in R5 (which will typically mean agreeing to undertake the PR)
Josh Mandel (Feb 28 2022 at 20:02):
Yunwei Wang: I would like to set a cutoff date. Any tickets that are not applied before cutoff date will be automatically move to next release.
I agree with the goal here.
Vassil Peytchev (Mar 01 2022 at 03:41):
Should we try a bit of micromanagement? We have the 2 hours on Mondays, so we could try something like this:
- A few people "on duty" who would prepare a local branch as we start the meeting (including the 10-15 min local build).
- During the first hour, as resolutions are approved, they are assigned to one of the participants on duty to apply to their branch. As each issue is applied, the branches are committed, built locally, and pushed (not as a PR, so no trigger for CI build).
- The second hour is used to finish applying all the decisions from the first hour, then merge all working branches into 1, and create one PR for that one branch, to trigger the CI build.
Why multiple people? If the changes are related (same resource/page), they go to the same person. If we have changes that are not related, they can be applied in parallel.
The idea is that a person would be "on duty" no more often than once a month. That way our commitment for Mondays is all that is needed to avoid stale decisions.
I think this will avoid marking issues as "resolved-change required" without an actual resolution.
Josh Mandel (Mar 01 2022 at 04:08):
Love this concept. Was discussing with @Gino Canessa earlier today a similar idea. Slight differences:
- a "working session branch" would live on GH and participants could push their commits directly to it (i.e., we would not enable branch protection rules and would not create a PR into master until the session was coming to a close)
- we might use this technique during calls or outside of calls during times when we're trying to rapidly make progress against our backlog (e.g., with a weekly async cadence for "sessions" working through backlog).
Gino Canessa (Mar 01 2022 at 16:31):
I am exploring some tooling to try and help here too - I think the majority of changes fall into a few buckets that could be made 'on rails' and would then not be able to break a build (e.g., changing page text needs HTML validation and verification of any links added, but doesn't affect anything else, changing the description of an element is the same, etc.).
The goal is to have something to lower the bar for 'simple' changes*, then someone who has the full build tooling and knowledge to fix any build issues can validate everything before it gets merged in.
Vassil Peytchev (Mar 01 2022 at 16:37):
I just think having one person per "session branch" is easier, because you avoid any possibility of merging until each branch is done, then you can merge once locally, build and then create the PR.
Scott Fradkin (Mar 02 2022 at 04:50):
Maybe this has been brought up in the past, but the connectathons can be used as informal hack days to work through making changes. Or, schedule once a month sessions?
Last updated: Apr 12 2022 at 19:14 UTC
