Stream: fmg
Topic: Agenda item for 17/03 Meeting
Melva Peters (Mar 15 2021 at 21:39):
Can we please add an agenda topic about quantitative results for connectathon testing requirement for FHIR IG balloting? Need to define the requirements so that it is "easier" to determine if IG has meet testing requirements. Also, what is the requirement for using non HL7 Connectathon testing as part of meeting this requirement?
Lloyd McKenzie (Mar 15 2021 at 22:09):
Current guidance is "at least 3 independently developed systems successfully interoperating and covering most (i.e. 80%+) of the solution space defined by the IG." Requirements for non-HL7 connectathon is that the connectathon is publicly announced in advance with sufficient notice (typically 30 days) to anyone who wants to participate and that anyone who wants to participate can on a non-discriminatory basis.
Lloyd McKenzie (Mar 15 2021 at 22:10):
Do we need more than that?
Melva Peters (Mar 15 2021 at 22:37):
Who decides if the testing covers 80%? So for non-HL7 connectathons, it just has to be advertised? No requirements for number of systems testing and covering 80%?
Lloyd McKenzie (Mar 16 2021 at 06:25):
Same requirements whether HL7 or not. We generally take the word of the participants at face value, though if someone wants to challenge whether an FMM level is appropriate, they could make their concerns known to the FMG and we could dig further.
John Moehrke (Mar 16 2021 at 12:19):
There should also be a requirement that Jira tickets against the IG are all resolved.
John Moehrke (Mar 16 2021 at 12:20):
Should there be a requirement to have a published test plan? Not necessary to have test tools and procedure; but a written test plan explains to everyone how the IG was tested.
John Moehrke (Mar 16 2021 at 12:21):
with an IG, I am less comfortable with the 80% measure. It would seem that an IG should be measured against 100%.
Lloyd McKenzie (Mar 16 2021 at 13:20):
No chance of us requiring that Jira tickets are all resolved - Jira tickets come in continuously, so an IG under active use will never get to 'done'. It's completely reasonable for a work group to declare a cutoff and, after that, only worry about 'critical' issues that come in until they're published.
If there was a requirement for a test plan, less than 5% of our IGs would have been published. The requirement that a connectathon has tested 80% of functionality is to allow you to get FMM 2. The objective of level 2 is "This IG has been tested in code and more or less works - i.e. it's not just a 'paper' spec anymore". Requiring that you test every single data element, extension, code, etc. isn't reasonable (or necessary) to get an evaluation that a 'reasonable' degree of implementation has occurred and the IG more or less works
John Moehrke (Mar 16 2021 at 14:09):
I think that is unfortunate position by the chair. Is that the decision of FMG? I don't recall this being discussed yet
Lloyd McKenzie (Mar 16 2021 at 14:44):
Which? We've never talked about requiring all tickets to be resolved before you can publish - because there's never been a hope of doing that. I don't think we've ever had fewer than 200 outstanding tracker items when we've put out a release of FHIR core - and that number has been trending significantly upward.
We've certainly talked about the notion of FMM 2 being tied to testing 'most' of the functionality, never 'all' of the functionality.
Lloyd McKenzie (Mar 16 2021 at 14:45):
Note that there's a base TSC/ANSI requirement to have resolutions for all ballot tracker items before we publish. That's not in question.
John Moehrke (Mar 16 2021 at 14:48):
then I lost track of the target we were discussing. sorry, so this is a definition of criteria to be allowed "into a ballot", vs "publish an STU", vs "normative"...
Lloyd McKenzie (Mar 16 2021 at 14:52):
Right. And it's not a 'hard' requirement for into ballot. You can ask for a waiver.
However, even for publishing normative or STU, we'd never require more than all ballot Jiras to be resolved. We'd never require all Jiras to be resolved.
John Moehrke (Mar 16 2021 at 15:05):
I think that needs to be considered. but that is a bit into our future
Last updated: Apr 12 2022 at 19:14 UTC
