Stream: cql
Topic: How do you test/debug CQL
John Silva (May 01 2021 at 00:12):
How do you test CQL rules and debug them when they might not work as expected?
I'm trying to use Atom with the cqf-language (2.9.2) plugin and when it doesn't work as expected there doesn't seem to be much information to help troubleshoot the (CQL rule) problem. Also, it appears to use a different version of the cql execution engine from that used by cqf-ruler (from the https://github.com/cqframework/clinical_quality_language package)? For example, a FHIRpath syntax of Patient.name.given[0] works with cqf ruler but not with Atom and the cqf-language plugin. (it returns 'string' with the plugin). How can the version of the cql execution engine used by cqf-language plugin be determined?
Thanks
Rob Reynolds (May 01 2021 at 13:53):
Hello John.
There is documentation for part of your question: https://atom.io/packages/language-cql. Specifically, in the "Using the CQL support in Atom" section, where it describes the folder structure requirements for testing.
Yes, the documentation could use some love... :) And as you go through this process, you are welcome (even encouraged) to update it!
Basically, the process for creating/running tests is:
- create test data as FHIR Resource files (as described in the section above)
- execute the CQL file
- check the resulting output for expected results/errors
More sophisticated debugging features are in the works, but, for now, the debugging process is entirely manual. I.e., write CQL that isolates the thing you're trying to test and do the above to check it (or more specifically, comment things until you figure out what's broken).
As far as the versions of the cql execution engine go, I know there is work underway to better keep those in lock-step. I'm not sure of the ETA on that though. @JP ?
Rob Reynolds (May 01 2021 at 14:04):
Also, there are working examples of functioning test implementations, if that helps. If you clone these repos you should be able to open the root folder in Atom, open a CQL file, Execute the CQL, and run the tests.
- https://github.com/cqframework/ecqm-content-r4 (looks like only CervicalCancerScreening has tests... and they both seem to be returning false for InitialPopulation atm so I'm guessing it's a work in progress, but it still demonstrates the idea)
- https://github.com/DBCG/connectathon
- open Atom at the fhir401 folder level, since this repo actually contains multiple IGs (by FHIR version)
- I specifically tried EXM125-7.3.000.cql and it returns expected results, but I think they all should (or at least most of them should).
John Silva (May 03 2021 at 12:28):
Thanks @Rob Reynolds
Yes, I've already followed the setup steps (even wrote a script to copy our test FHIR resources into the directory structure it expects). The problem is that when things fail there's not really much help as to why the CQL logic failed. For example, in one case we use a FHIRpath (like?) expression for Patient.name.given(0) -- which works with cqf-ruler (from cqfframework Github package) but doesn't with cql-language plugin in Atom. It' doesn't give an error but returns 'string' instead of the actual value. (was able to get this to work by adding ...given(0).value). It's problematic (at least for me) trying to figure out how to access properties of a selected resource(s); typically the cql-language just errors out with Java NullException errors! It's also not clear which version of the cql execution engine cql-language plugin is using -- maybe it's a version difference between what is used in the cqframework package but I can't tell.)
(yes, right now I've been using the code-test-change code-test again method. It's somewhat tedious and not very deterministic. Also, the FHIR CQL pages do not seem to have (m)any examples of data access when a rule evaluates to true; it seems like most of the examples are of rules that evaluate to true/false as the only 'return value'.)
JP (May 03 2021 at 16:20):
Hi John!
The versions of the CQL tooling being used by the Atom plugin are usually recorded in the change log when a new version is released. The cqf-ruler currently doesn't expose that information anywhere so it's difficult to compare, and it is indeed out of date.
As far as using Patient.name.given[0], what was the context of the use in the cqf-ruler? The type of Patient.name.given[0] is actually a FHIR String (as opposed to CQL String) which is a complex type which may have an id, extensions, and so on. The .value gets the simple string value contained within that more complex type. So the "string" return value you're seeing with just Patient.name.given[0] is correct in the sense that the element of that object isn't specified.
Because it's a pain to use .value everywhere, it's common to include the FHIRHelpers library which includes a few functions that automatically convert the FHIR types. It doesn't work in every context, but it does work in many of them. Here's a link that describes a bit more about what it does and why you need .value in some places:
http://build.fhir.org/ig/cqframework/cqf/#fhir-helpers
John Silva (May 03 2021 at 18:47):
@JP Thanks! That makes sense -- didn't know there's a difference between CQL String and FHIR String (new to this CQL as you can tell).
Using the .value should be what we want and that seems to work in both cqf-framework plugin and the version of cqf-ruler that we're using.
Yes, it's hard to determine (or specify) which version of the 'cql execution engine' (if there's such a thing) is being used in either of these packages and so these version differences 'bite you in the tails' ;-)
We do already include the FHIRHelpers in our CQL.
Rob Reynolds (May 04 2021 at 13:59):
If you're including FHIRHelpers in both, I'm confused why you're seeing a difference in the need to use .value between the lanaguage-cql plugin and CQFRuler. I wouldn't expect that.
@JP probably has a good reason though. :)
John Silva (May 04 2021 at 17:56):
Yeh, I don't know why but the data doesn't show the same in the plugin and cqf-ruler unless I add the .value (almost) everywhere! I'm guessing it has to do with the different versions of the CQL execution engine used by these.
JP (May 04 2021 at 18:48):
Hi John,
I actually need more detail about what you're doing to give any more insight about the difference. It's not the case that the cqf-ruler just runs the CQL in the same way the Atom plugin does. There's an additional layer of processing depending on the specific operation of the cqf-ruler you're using. So it's entirely possible for both of those tools to use the same version of the cql-engine, give different answers, and for that to be correct in both cases. For example, Measure processing is more complex than simply executing the Denominator or Numerator definitions in some CQL.
JP (May 04 2021 at 18:53):
IOW, it may be the case that the difference is due to different versions of the cql-engine. It may be the case that the difference is simply due to the specified behavior of the operations. It also may be the case that there's a bug in one or both of the tools. It's not the case that Atom and cqf-ruler will always return exactly the same thing for every definition in the CQL. For example, the Library $evaluate operation automatically casts CQL types to FHIR types (and that's required by the spec for Library $evaluate).
John Silva (May 04 2021 at 21:40):
@JP - Thanks for the follow-up. Yes, I don't know what is running 'under the covers' but there are definitely differences in behavior from cqf-ruler and the cql-language plugin. I think what you just pointed out is one of the differences -- Library $evaluate casting CQL types to FHIR types so all the places I have to add .value to get the textual values to show up in the plugin "just work" in cqf-ruler probably because of this casting!
Here's another interesting one but I'll open a separate topic about it. What/how does CQL match the Patient context for Coverage?
JP (May 04 2021 at 21:51):
@John Silva - Sorry, I've not been clear. I'm looking for the specific operations you're invoking on the ruler so I can help determine what might be happening. Are you using the $cql operation? The Library $evaluate operation? The Measure $evaluate-measure operation? Those are all specified differently so I can't help make the determination if there's an issue with the results unless I know which you are using. :smile:
John Silva (May 05 2021 at 01:31):
We're only using $evaluate with the ruler:
${HOST}/Library/${i.resource.id}/$evaluate
We use the latest cqf-ruler from contentgroup/cqf-ruler. (latest as of about 3 weeks ago)
Last updated: Apr 12 2022 at 19:14 UTC
