FHIR Chat · Supporting Automated Testing of CDS Hooks via FHIR Testing F · cds hooks

Stream: cds hooks

Topic: Supporting Automated Testing of CDS Hooks via FHIR Testing F

Richard Ettema (Aug 14 2018 at 21:37):

Hello cdshooks community.

I'm reaching out to solicit feedback, suggestions and recommendations concerning the support of automated conformance testing of cdshooks implementations by leveraging and enhancing the FHIR Testing Framework/Test Engine, http://build.fhir.org/testing.html, and the FHIR TestScript resource, http://build.fhir.org/testscript.html.

The driving force behind this request is the Da Vinci project where the implementation of one of the initial use cases, Coverage Requirements Discovery (CRD), is via cdshooks. Implementation and Conformance testing of the Da Vinci use cases are to leverage the FHIR Testing Framework and TestScript resources. So, I am currently researching how best to support the testing of cdshooks services within this framework.

Richard Ettema (Aug 14 2018 at 21:40):

Here is an initial list of items where I am seeking feedback:

• CDS Hooks is a separate specfication from the FHIR specification. As such, CDS Hooks service request and response bodies/payloads cannot be validated using the current FHIR Validation Engine.
  -- Should/Can the FHIR Validation Engine be enhanced to support validation of cdshooks messages?
  -- If the FHIR Validation Engine is enhanced, how will the engine handle cdshook service definitions; i.e. are cdshooks service definitions available in a structured format that can be parsed by the engine?

• What TestScript resource definition enhancements are needed? Some initial areas:
  -- New operation code(s) are needed for the cdshooks services. Should a new ValueSet be created or should additional code(s) be added to the existing operation codes?
  -- If the FHIR Validation Engine is not enhanced to support cdshooks messages, then the current 'assert.validateProfileId' element would need to change to allow access to the contained FHIR resource instance(s). This would look something like:

 <validateProfile>
   <validateProfileId value="profileId"/>
   <validateProfilePath value="[path expression to contained FHIR resource]"/>
 </validateProfile>
 *Note: FHIR R4 ballot is currently frozen so this may require the use of extensions.

• How would we test errors, and exceptions within a cdshooks workflow?

• The cdshooks messages can, and most likely will, contain FHIR resources. I have not found any mechanism to state what FHIR version of the resource(s) are contained in the cdshooks message. This needs to be explicit and clear in order for test engines to construct expected outcomes in testing.
  -- There is a Zulip thread message that asks this question, https://chat.fhir.org/#narrow/stream/17-cds-hooks/topic/prefetch.20fhir.20version, but there has not been any reply.
  -- Also, the cdshooks specification, https://cds-hooks.org/specification/1.0, contains examples where it appears the FHIR version of the contained resources are outdated; i.e. they are not conformant to the official FHIR DSTU2 (v1.0.2) or STU3 (v3.0.1) specification. It would be helpful to have these updated.

Thank you in advance for your feedback and any additional suggestions or recommendations.

Kevin Shekleton (Aug 15 2018 at 13:19):

Should/Can the FHIR Validation Engine be enhanced to support validation of cdshooks messages?

I've expressed this opinion before and it may not be a popular opinion here, but I don't see a need to define a standard grammar/language for expressing the notion of an abstract test. I am fully behind testing (one of the major reasons why we re-wrote our Sandbox was to write it from the ground up with automated tests) but think the community would get far more value by simply writing automated tests in an existing framework (rspec, junit, unittest in python, etc) than the FHIR testing resources.

Grahame Grieve (Aug 15 2018 at 13:20):

one problem with that approach is: which one?

Kevin Shekleton (Aug 15 2018 at 13:20):

I don't care, pick one and open source it.

Jens Villadsen (Aug 15 2018 at 13:21):

kevin for president

Jens Villadsen (Aug 15 2018 at 13:21):

:D

Kevin Shekleton (Aug 15 2018 at 13:21):

are cdshooks service definitions available in a structured format that can be parsed by the engine?

A CDS Service's discovery endpoint (available at {baseUrl}\cds-services) will return the service definitions in JSON

Kevin Shekleton (Aug 15 2018 at 13:22):

I'm glad I have one supporter on this topic @Jens Villadsen :smile:

Grahame Grieve (Aug 15 2018 at 13:23):

is there an apparent preferred candidate? I'm not against picking something if it means we can share test cases and the community will coalesce to it

Kevin Shekleton (Aug 15 2018 at 13:24):

My language preference (in order) would be: Ruby, JavaScript, or Python. Java or C# if you really had to but I'd take one of the first 3 choices any day.

Grahame Grieve (Aug 15 2018 at 13:26):

I'd think about platform/library support before language....

Kevin Shekleton (Aug 15 2018 at 13:26):

The mental hurdles one has to go through to learn FHIR's custom testing language (those resources/tooling) and then execute them under one of the few existing implementations has always seemed odd to me. I'd rather just grab an open source test suite (written in one of those existing languages/testing frameworks) and add new tests (if needed) and run.

The work @Josh Mandel's team with the python Sync4Science tests was great

Kevin Shekleton (Aug 15 2018 at 13:26):

All 3 of those languages have great testing library support

Kevin Shekleton (Aug 15 2018 at 13:30):

How would we test errors, and exceptions within a cdshooks workflow?

Testing these scenarios is always difficult. One great step would be to write an open source CDS Service that returns failures/execptions when called. Something like a Chaos Monkey CDS Service. This would be beneficial only to EHR implementers.

Testing these same scenarios for CDS Services will be more difficult since the failure scenarios are much less. For instance, if the EHR fails the CDS Service just isn't invoked. You could create a similar open source Chaos FHIR Service that fails so when the EHR invokes the CDS Service, the subsequent requests to the FHIR Server (by the CDS Service) fail.

Kevin Shekleton (Aug 15 2018 at 13:32):

The cdshooks messages can, and most likely will, contain FHIR resources. I have not found any mechanism to state what FHIR version of the resource(s) are contained in the cdshooks message. This needs to be explicit and clear in order for test engines to construct expected outcomes in testing.

This is an offline coordination between the EHR and CDS Service. We have punted on this until we get more implementer feedback. I don't believe this hampers testing in any way as you can simply specify the version of FHIR you expect the EHR + CDS Service to use in the course of their transactions.

Kevin Shekleton (Aug 15 2018 at 13:33):

Also, the cdshooks specification, https://cds-hooks.org/specification/1.0, contains examples where it appears the FHIR version of the contained resources are outdated; i.e. they are not conformant to the official FHIR DSTU2 (v1.0.2) or STU3 (v3.0.1) specification. It would be helpful to have these updated.

CDS Hooks is not tied to a particular version of FHIR, just like SMART. As such, our specification will contain reference and examples to dealing with multiple versions of FHIR.

Kevin Shekleton (Aug 15 2018 at 13:34):

Also, it is "CDS Hooks", not "cdshooks" (sorry, nit correction :smile: )

Grahame Grieve (Aug 15 2018 at 13:34):

it would be good for the examples to conform to at least one of the versions...

Kevin Shekleton (Aug 15 2018 at 13:35):

We actually have examples across both DSTU2 and STU3 now

Kevin Shekleton (Aug 15 2018 at 13:35):

(Isaac has an update to our examples in a PR review right now)

Josh Mandel (Aug 15 2018 at 14:00):

Overall for testing, it's not clear to me that we need to be able to share test logic across frameworks. In other words, agreeing with Kevin: the important thing is to have tests and make them easy to run. If we have different Test Suites written by different people in different languages, that's fine too -- but we should maintain one official test suite for CDS Hooks. Keep a copy hosted online so it's easy to run without installing. Make it available in Docker so it's easy for anyone to run locally. And pick a language that at least the test Suite authors are productive with, so it's easy to maintain and extend.

Lloyd McKenzie (Aug 15 2018 at 14:32):

How does multi-version work in prefetch?

Kevin Shekleton (Aug 15 2018 at 15:15):

@Lloyd McKenzie - Are you asking about a CDS Service requesting different versions of FHIR in prefetch queries?

Lloyd McKenzie (Aug 15 2018 at 15:29):

My interpretation of your earlier statement is that prefetch definitions are version agnostic. In the case where a query would execute successfully against multiple versions, how does a server know what version is in the prefetch?

Kevin Shekleton (Aug 15 2018 at 16:43):

Ah, I get your question now. The only way to support this today is for a CDS Service to flex its Discovery data based upon the EHR calling it. This is similar to how a SMART app must flex how it calls the EHR's FHIR server depending upon the version of FHIR that is supported.

Lloyd McKenzie (Aug 15 2018 at 18:36):

In the case of a SMART app, the app initiates the data flow and it can hit the CapabilityStatement. For CDS Hooks, there's what I'd presumed was a "static" Service definition that is queried by the Client/EHR to see what prefetches are asked for. Is it really the intention that the service definition is dynamic? If it is, how does the service know what version to expose? I'm hoping it's not hard-configured to who's asking...

Kevin Shekleton (Aug 15 2018 at 21:43):

All good questions and things we have discussed previously (without coming to an answer). What we have settled on is "let's see how initial implementations goes and add enhancements/complexity when deemed necessary"

Kevin Shekleton (Aug 15 2018 at 21:45):

We really value maintaining a simple API (it's one of the biggest reasons why CDS Hooks is so popular IMHO) and not trying to design/tackle every possible scenario and corner case.

Isaac Vetter (Aug 16 2018 at 04:09):

Hey @Kevin Shekleton - Lloyd and I are taking about a potentiall iss GET parameter on the discovery endpoint as a simple method to enable dynamic CDS service functionality. See https://chat.fhir.org/#narrow/stream/17-cds-hooks/topic/Search.20param.20names.20in.20prefetch

Thoughts welcome!

Kevin Shekleton (Aug 16 2018 at 04:35):

@Isaac Vetter - What about just using the JWT instead since that is already present?

Isaac Vetter (Aug 16 2018 at 04:39):

that's a great idea.

Does the CDS client authenticate to the CDS Hooks service's discovery endpoint? It's unspecified in the spec.

With that said, if the CDS client did authenticate, per the spec to the discovery endpoint, the service would then have access to the CapabilityStatement.

I like it!

@Lloyd McKenzie - what do you think?

Isaac

Kevin Shekleton (Aug 16 2018 at 04:46):

"Each time an EHR transmits a request to a CDS Service, the request MUST include an Authorization header presenting the JWT as a “Bearer” token:"

This is supposed to convey that _every_ EHR call to a CDS Service (whether that be to the Discovery endpoint or to a CDS Service endpoint) needs to have the JWT.

Later in the spec, we call out the Discovery endpoint in an example when discussing the JWT internals:

"For example, consider a CDS Service available at a base URL of https://cds.example.org. When the EHR invokes the CDS Service discovery endpoint, the aud value is either "https://cds.example.org/cds-services" or ["https://cds.example.org/cds-services"]. Similarly, when the EHR invokes a particular CDS Service (say, some-service), the aud value is either "https://cds.example.org/cds-services/some-service" or ["https://cds.example.org/cds-services/some-service"]."

Kevin Shekleton (Aug 16 2018 at 04:47):

Oh, I see you understand that too @Isaac Vetter with your link to "the spec". :tada:

Kevin Shekleton (Aug 16 2018 at 04:48):

Well, I guess I am just posting the relevant parts of the spec that do call out that the JWT is present even on Discovery calls for people who don't want to follow the link :stuck_out_tongue_winking_eye:

Lloyd McKenzie (Aug 16 2018 at 06:13):

As I said in the other thread, I'm not thrilled with the notion of a dynamic service definition that adjusts itself to the capabilities of the EHR. I can certainly understand the appeal of that architecture from an EHR perspective, but if it's a question of "who's best able to deal with the complexity of adjusting configuration", my leaning would be that the EHRs have much more capability than a lot of the CDS services - which I think of as being not necessarily mor sophisticated than app developers.

Kevin Shekleton (Aug 16 2018 at 06:45):

SMART apps already have to dynamically adjust themselves to the capabilities of the EHRs they are interacting with. This includes differing FHIR versions + differences in supported resources, CRUD capabilities, and search/filtering capabilities on individual resources.

Lloyd McKenzie (Aug 16 2018 at 14:26):

Yes - but SMART Apps don't host dynamic websites. The CDS Services will typically have some degree of dynamicly interacting with the client's repository. The question is whether they also need to have a dynamic service declaration.

Kevin Shekleton (Aug 16 2018 at 19:35):

Yes - but SMART Apps don't host dynamic websites.

Can you elaborate on this? I'm not following.

Lloyd McKenzie (Aug 16 2018 at 19:49):

The EHR doesn't ever "query" the SMART App for configuration. You just launch it. The SMART app does all of the querying. With CDSHooks, the service is expected to have a queriable endpoint that describes what it does. @Isaac Vetter is proposing that be an 'active' endpoint the dynamicly adjusts what it displays based on who's asking. I think that's going to be too hard for a lot of services to manage.

Last updated: Apr 12 2022 at 19:14 UTC