Stream: cds hooks/github
Topic: docs / Issue #195 What's the process for creating new hooks?
Github Notifications (Apr 11 2018 at 15:47):
isaacvetter opened Issue #195
In an offline discussion with @brynrhodes, @isaacvetter, and @brettmarquard, we discussed the need for a better process for suggesting, reviewing, testing and publishing new hooks.
A few goals:
- Hooks should be able to be created, reviewed, published outside of the more rigid publishing calendar of the actual, core specification.
- Hooks should be tested before being published at FHIR Connectathons. The FHIR Maturity Model would be a good model to follow.
- Hooks should be reviewed with an overall, holistic perspective.
- Requestors should document the proposed hook on a new wiki page, link to the new wiki page from Proposed Hooks and create a new issue.
Github Notifications (Apr 11 2018 at 15:48):
isaacvetter edited Issue #195
In an offline discussion with @brynrhodes, @isaacvetter, and @brettmarquard, we discussed the need for a better process for suggesting, reviewing, testing and publishing new hooks.
A few goals:
- Hooks should be able to be created, reviewed, published outside of the more rigid publishing calendar of the actual, core specification.
- Hooks should be tested before being published at FHIR Connectathons. The FHIR Maturity Model would be a good model to follow.
- Hooks should be reviewed with an overall, holistic perspective to avoid redundancy and ensure new hooks follow best practices.
- Requestors should document the proposed hook on a new wiki page, link to the new wiki page from Proposed Hooks and create a new issue.
Github Notifications (Apr 11 2018 at 15:49):
isaacvetter edited Issue #195
In an offline discussion with @brynrhodes, @isaacvetter, and @brettmarquard, we discussed the need for a better process for suggesting, reviewing, testing and publishing new hooks.
A few goals:
- Hooks should be able to be created, reviewed, published outside of the more rigid publishing calendar of the actual, core specification.
- Hooks should be tested before being published at FHIR Connectathons. The FHIR Maturity Model would be a good model to follow.
- Hooks should be reviewed with an overall, holistic perspective to avoid redundancy and ensure new hooks follow best practices.
- Requestors should document the proposed hook on a new wiki page (following the template), link to the new wiki page from Proposed Hooks and create a new issue.
Github Notifications (Apr 11 2018 at 15:49):
isaacvetter labeled Issue #195
Github Notifications (Apr 11 2018 at 15:50):
kpshek commented on Issue #195
Some ideas on factors to consider when evaluating the maturity of the hook:
- How many EHRs have implemented it at Connectathons?
- How many CDS Service Providers have implemented it at Connectathons?
- Is the hook implemented in the CDS Hooks Sandbox?
- How many Connectathons has the hooks been tested in?
Github Notifications (Jun 05 2018 at 08:04):
kpshek commented on Issue #195
Note: this relates to #336
Github Notifications (Jun 13 2018 at 15:55):
kpshek milestoned Issue #195
Github Notifications (Jun 13 2018 at 15:55):
kpshek unlabeled Issue #195
Github Notifications (Jun 13 2018 at 15:58):
isaacvetter assigned Issue #195
Github Notifications (Aug 06 2018 at 22:06):
isaacvetter commented on Issue #195
Hey Everybody,
I wanted to get some feedback on this draft hook's maturity model before putting it into a PR:
I considered:
- http://wiki.hl7.org/index.php?title=FHIR_Maturity_Model
- http://wiki.hl7.org/index.php?title=FHIR_Conformance_QA_Criteria
- https://github.com/cds-hooks/docs/blob/master/GOVERNANCE.md
- https://github.com/cds-hooks/docs/issues/336Isaac
## Hook Maturity Mode
The Hook maturity levels use the term CDS client to generically refer to the clinical workflow system in which a CDS services returned cards are displayed. Interim minor versions may be used to reflect changes in the hook's change log.
Hook Maturity Hook Version Requirements Draft 0.10 Hook is defined according to the hook definition format. Submitted 0.2.0 _The above, and …_ Hook definition is written up as a github pull request using the Hook template and community feedback is solicited on the zulip CDS Hooks stream. Tested 0.40 _The above, and …_ The hook has been tested and successfully supports interoperability among at least one CDS client and two independent CDS services using semi-realistic data and scenarios (e.g. at a FHIR Connectathon). The github pull request defining the hook is approved and published. Considered 0.60 _The above, and …_ At least 3 distinct organizations recorded ten distinct implementer comments, including at least two CDS clients and three independent CDS services. The Hook has been tested at two FHIR Connectathons. Cleaned-up 0.80 _The above, and …_ The author agrees that the artifact is sufficiently stable to require implementer consultation for subsequent non-backward compatible changes. The hook is implemented in the standard CDS Hooks sandbox and multiple prototype projects. The Hook specification SHALL: <ul><ol>Identify a broad set of example contexts in which the hook may be used with a minimum of three, but as many as 8-10.</ol><ol>Clearly differentiate the hook from similar hooks or other standards to help an implementer determine if the hook is correct for their scenario.</ol><ol>Explicitly document example scenarios when the hook should not be used.</ol></ul> Mature 1.0 _The above, and ..._ The hook has been implemented in production in at least two CDS clients and three independent CDS services in more than one country. An HL7 working group ballots the hook and the hook has passed HL7 STU ballot. The FHIR Management Group has approved the hook.
Github Notifications (Aug 07 2018 at 15:00):
kpshek commented on Issue #195
Thanks for writing this up @isaacvetter! I agree that soliciting feedback on this here first is better than a PR. Some thoughts:
Hook Maturity
I like how FHIR uses a monotonically increasing number (FMM0, FMM1, FMM2, etc) to describe maturity. This makes it easy to know where something falls onto the maturity scale. Contrast this with the proposal here where, without the additional detail in this table, I could not tell you which is better: "Cleaned up" or "Tested".I propose we use something like Level 0, Level 1, Level 2, Level 4 instead of the current narrative terms for the aforementioned benefits.
Hook Version
Is the "Hook Version" column expressing an example of possible version or is it prescribing the version of the hook at a particular level? I believe it is the latter given this statement: "Interim minor versions may be used to reflect changes in the hook's change log."While the hook version addresses some of my concerns with the hook maturity name above, I think it would be best to not try and align the hook version with its maturity. Additionally, the hook version model proposed here does not follow semantic versioning (SemVer). In SemVer, the minor number is the second number (major.minor.revision).
Thus, in the current model, changes to a "Submitted" hook would version like: 0.2, 0.2.1, 0.2.2, etc (changes to the revision number). However, changes to a "Mature" hook would version like: 1.0, 1.1, 1.1.1, 1.2, etc (changed to either the minor or revision number).
I propose we leave the notion of version from the maturity model and let hook authors use SemVer. It is interesting to consider whether hook authors must only version their hook at 1.0 once it reaches the final level of maturity...I'm not sure yet where I sit on that.
Submitted
What is the repository of the PR? Who manages this repository?Tested
Who merges the PR?Cleaned-up
The hook is implemented in the standard CDS Hooks sandbox and multiple prototype projects
I think requiring the hook to be implemented in the CDS Hooks Sandbox is really powerful. :thumbs_up: It will require some changes to our Sandbox architecture to allow new hooks to be added in a modular fashion, but this is something we need to do.
*Mature
The hook has been implemented in production in at least two CDS clients and three independent CDS services in more than one country.
This is a really high bar (especially the bit about "more than one country"). For FHIR I think it is less of a high bar since there is only one role being measured -- the FHIR server. However, with CDS Hooks, we have the CDS client + the CDS service. While CDS clients are often deployed to more than one country, I think this is far less likely for CDS services (at least, today).
An HL7 working group ballots the hook and the hook has passed HL7 STU ballot. The FHIR Management Group has approved the hook.
Where does the FMG come in here? I think with FHIR the value proposition is that the FMG oversees all resources for consistency. But, they have not yet been involved yet with CDS Hooks. Is the intention here to have some group that ensures consistency across hooks?
Github Notifications (Aug 08 2018 at 15:38):
kpshek commented on Issue #195
@lmckenzi raise a good point on Zulip regarding editing hooks. Specifically, we need to identify a hook owner(s) so that when people have questions, want to propose modifying the hook, etc they know who to reach out to in order to collaborate on any proposed changes.
Github Notifications (Aug 08 2018 at 17:18):
lmckenzi commented on Issue #195
In terms of maturity, it's probably best to clarify what's meant by "FHIR Connectathon" and "implemented". Are we talking HL7-International hosted connectathons or is an ad-hoc connectathon amongst a small community sufficient. Do reference implementations count, or do the implementers need to be real clinical/administrative systems?
What is the pull request executed against? A markdown page of hook candidates? It might be best to have a separate page per hook. That way proposed changes to a particular hook can more easily be managed.
The notion of approval being determined by the "author" of the hook is a tad problematic. Who's the "author" of something that has been revised by a bunch of people and where the original proponent might no longer be an active part of the discussion? This should really be a community decision. Could this somehow be managed as a joint discussion with a vote on a call or community thumbs up/thumbs down in an eVote?
Github Notifications (Aug 08 2018 at 18:02):
kpshek commented on Issue #195
Are we talking HL7-International hosted connectathons or is an ad-hoc connectathon amongst a small community sufficient
Personally, I'd like to see us focus on the spirit of what we're trying to achieve in these guidelines/rules. To mean, that means _open_ developer forums/events with a diverse set of participants rather than named/sanctioned events from a particular organization (eg, HL7 Connectathons).
Do reference implementations count, or do the implementers need to be real clinical/administrative systems?
Reference implementations are great but to a point. At some point, I think you really need implementations on top of existing systems (the real systems you mentioned) as they often has constraints/challenges that reference (aka, greenfield) implementations don't have to worry about.
It might be best to have a separate page per hook. That way proposed changes to a particular hook can more easily be managed.
This is exactly our intention -- a separate page/document per hook.
Github Notifications (Aug 08 2018 at 18:41):
lmckenzi commented on Issue #195
Agree on defining the spirit, but defining the letter can be important too. My read of "spirit" of "at least two FHIR Connectathons" is "well attended connectathons with a diversity of participation held some time apart". But the letter could be satisfied by the same group of highly invested stakeholders holding virtual connectathons a couple of weeks apart that they advertise as open to other participants on chat.fhir.org without an expectation of anyone else showing up. You will want your policies to help reduce the chances of that sort of gaming the process so that you actually get what you're after - broad community engagement and consensus. In fact, simply stating up front that what you're after is broad engagement of the full community who would be expected to use/be impacted by the hook and that the criteria will be evalutated in the light of that intention would be helpful.
Github Notifications (Aug 09 2018 at 04:50):
isaacvetter commented on Issue #195
Take 2:
### Words describing maturity level
I like how FHIR uses a monotonically increasing number … easy to know where something falls onto the maturity scale
I'm not sure that this is true. FMM's levels don't inherently mean anything (they're not _semantic_). Why isn't there an FFM8? Why is FMM3 important … or is that FMM4? The FMM levels aren't naturally _bounded_ nor do they have inherent meaning. I definitely agree that the specific terms could use improvement. I think that it's worthwhile to associate meaning with maturity levels. For sure, "Cleaned up" is a poor term. How about "Documented" (the intent of this level is that the hook is being prepared for formal inclusion -- only hooks that are intended to be formally included will likely hit this level).
### Hook version
Kevin - I think that your point about SemVer is that major changes should be represented in the version number before a hook reaches maturity. That's pretty convincing, but an arbitrary, non-bounded versioning scheme is worse. I modified theMaturity Levelto be 1-10, instead of 0.1 - 1.0. The emphasis on even numbers does make less sense now.### Submitted
#### >What is the repository of the PR? Who manages this repository?
Yeah, great question. I think that's outside the scope of this issue. Do you disagree? What we've talked about is creating an additional repo in the cds-hooks github project to contain hooks. From a governance perspective, that's a bit problematic. For the purposes of this issue/PR, I'd like to leave it out of scope.### Considered
#### >is an ad-hoc connectathon amongst a small community sufficient [?]
Lloyd, I understand Da Vinci's model. I intentionally required FHIR Connectathon. This places a minimum timeline of 8 months on theConsideredmaturity level. I've ignored your good advice and updated the content to explicitly say "HL7 FHIR Connectathon", such that new hooks must be standardized as part of HL7's FHIR Connectathons.
#### >You will want your policies to help reduce the chances of that sort of gaming the process
I also added an additional sentence at the top to state the goals of the process outright.### Tested
#### Who merges the PR?
The committers of the repo, of course! Seriously, though, see above.### Mature
#### The hook has been implemented in production in at least two CDS clients and three independent CDS services in more than one country.Yeah. I do want a high bar for a hook to reach a 1.0/mature status. I think that we're blessed with no region-specific content. I did wonder, while writing this, if any of our three existing hooks could currently meet these requirements. I've removed the multiple countries requirement.
#### An HL7 working group ballots the hook and the hook has passed HL7 STU ballot. The FHIR Management Group has approved the hook.
Is the intention here to have some group that ensures consistency across hooks?
I think that the PR approvers should ensure consistency across hooks as part of the
Submittedmaturity levle. For aMaturestatus, we further need a formal SDO and more rigorous balloting process. I don't know what the right group is. Maybe the HL7 CDS wg would be more appropriate than the FMG. I removed the FMG requirement.
## Hook Maturity Model
The intent of this maturity model is to attain broad community engagement and consensus, before a hook is labeled as mature, that the hook is necessary, implementable, and worthwhile to the CDS services and CDS clients that would reasonably be expected to use it. The below criteria will be evaluated with these goals in mind.The Hook maturity levels use the term CDS client to generically refer to the clinical workflow system in which a CDS services returned cards are displayed.
Maturity Level Maturity title Requirements 1 Draft Hook is defined according to the hook definition format. 2 Submitted _The above, and …_ Hook definition is written up as a github pull request using the Hook template and community feedback is solicited on the zulip CDS Hooks stream. 4 Tested _The above, and …_ The hook has been tested and successfully supports interoperability among at least one CDS client and two independent CDS services using semi-realistic data and scenarios (e.g. at a FHIR Connectathon). The github pull request defining the hook is approved and published. 6 Considered _The above, and …_ At least 3 distinct organizations recorded ten distinct implementer comments, including at least two CDS clients and three independent CDS services. The hook has been tested at two HL7 FHIR Connectathons. 8 Documented _The above, and …_ The author agrees that the artifact is sufficiently stable to require implementer consultation for subsequent non-backward compatible changes. The hook is implemented in the standard CDS Hooks sandbox and multiple prototype projects. The Hook specification SHALL: <ul><ol>Identify a broad set of example contexts in which the hook may be used with a minimum of three, but as many as 8-10.</ol><ol>Clearly differentiate the hook from similar hooks or other standards to help an implementer determine if the hook is correct for their scenario.</ol><ol>Explicitly document example scenarios when the hook should not be used.</ol></ul> 10 Mature _The above, and ..._ The hook has been implemented in production in at least two CDS clients and three independent CDS services. An HL7 working group ballots the hook and the hook has passed HL7 STU ballot.
Github Notifications (Aug 09 2018 at 05:02):
lmckenzi commented on Issue #195
"FHIR Connectathon" doesn't necessarily imply HL7 hosted connectathon. And even if that were the requirement, would doing the May WGM and June DevDays count?
I think there's a degree of benefit to aligning our numbering convention with HL7's. The communities have huge overlap and diverging on numbering is going to be confusing. I'm not opposed to naming the levels, but the names aren't that critical. If you know what number things start at, what number things finish at and the criteria to get to each level, that's all the semantic meaning needed.
Github Notifications (Aug 13 2018 at 20:39):
isaacvetter commented on Issue #195
Alright. I changed the maturity numbers to more closely match FMM and dropped the HL7 FHIR Connectathon-specific requirement with some additional explanatory text.
Without additional feedback in the next day or two, I'll write this up as a PR on the hooks/index.md page.
Take 3
## Hook Maturity Model
The intent of this maturity model is to attain broad community engagement and consensus, before a hook is labeled as mature, that the hook is necessary, implementable, and worthwhile to the CDS services and CDS clients that would reasonably be expected to use it. Implementer feedback should drive the maturity of new hooks. Diverse participation in open developer forums and events, such as HL7 FHIR Connectathons, is necessary to achieve significant implementer feedback. The below criteria will be evaluated with these goals in mind.The Hook maturity levels use the term CDS client to generically refer to the clinical workflow system in which a CDS services returned cards are displayed.
Maturity Level Maturity title Requirements 0 Draft Hook is defined according to the hook definition format. 1 Submitted _The above, and …_ Hook definition is written up as a github pull request using the Hook template and community feedback is solicited on the zulip CDS Hooks stream. 2 Tested _The above, and …_ The hook has been tested and successfully supports interoperability among at least one CDS client and two independent CDS services using semi-realistic data and scenarios (e.g. at a FHIR Connectathon). The github pull request defining the hook is approved and published. 3 Considered _The above, and …_ At least 3 distinct organizations recorded ten distinct implementer comments, including at least two CDS clients and three independent CDS services. The hook has been tested at two connectathons. 4 Documented _The above, and …_ The author agrees that the artifact is sufficiently stable to require implementer consultation for subsequent non-backward compatible changes. The hook is implemented in the standard CDS Hooks sandbox and multiple prototype projects. The Hook specification SHALL: <ul><ol>Identify a broad set of example contexts in which the hook may be used with a minimum of three, but as many as 8-10.</ol><ol>Clearly differentiate the hook from similar hooks or other standards to help an implementer determine if the hook is correct for their scenario.</ol><ol>Explicitly document example scenarios when the hook should not be used.</ol></ul> 5 Mature _The above, and ..._ The hook has been implemented in production in at least two CDS clients and three independent CDS services. An HL7 working group ballots the hook and the hook has passed HL7 STU ballot.
Github Notifications (Oct 10 2018 at 15:55):
isaacvetter closed Issue #195
Last updated: Apr 12 2022 at 19:14 UTC
