FHIR Chat · Test with Crucible · implementers

Stream: implementers

Topic: Test with Crucible

Adam (Jul 18 2018 at 06:02):

When I was using the Crucible to test the conformance of my FHIR implemantation, it showed a message that'Error: Conformance Statement could not be loaded crucible'. Has anyone ever met this issue? (PS:The CapabilityStatement seems to be fine). Any suggestions?

nicola (RIO/SS) (Jul 18 2018 at 08:57):

Is crucible alive?

Jason Walonoski (Jul 19 2018 at 00:30):

Yes.

Jason Walonoski (Jul 19 2018 at 00:32):

@Adam Can you provide a permalink to the result? Crucible generates those for the tests. If the errors occurs before that, please send the URL directly. Attention @Robert Scanlon

nicola (RIO/SS) (Jul 19 2018 at 00:47):

@Jason Walonoski Do you have harmonisation plans with TestScripts & TouchStone?

nicola (RIO/SS) (Jul 19 2018 at 00:48):

I see, TestScript became a part of FHIR - so i think it should be supported by more then one vendor ?!

Jason Walonoski (Jul 19 2018 at 12:16):

Crucible supports TestScript. Hell, I was the one who first proposed TestScript resource and added it to the spec in ages long past (@Richard Ettema has long maintained this resource, more credit to him). Not all of our tests are based on TestScript. In fact, a majority of Crucible tests are custom ruby code, but we do support the TestScripts built into the specification. There is currently no support for users providing their own scripts. We also don't support all of Touchstone's scripts. One reason is that we do not support TestScript.rule (https://www.hl7.org/fhir/testscript-definitions.html#TestScript.rule) because it is essentially a magical escape hatch to run your own custom (and maybe proprietary) code -- which means the interoperability of TestScript resources that use rule is zero.

Grahame Grieve (Jul 19 2018 at 12:19):

do you propose that we should limit the rules more stritly?

Jason Walonoski (Jul 19 2018 at 12:26):

I don't have a concrete proposal. I have just identified the potential problem. I have tossed out the idea of using FHIR Path or CQL as a standard language for rule... but then you need a data model for requests and responses... and there may be other problems. I haven't thought deeply enough about it yet.

Grahame Grieve (Jul 19 2018 at 12:26):

well,we do have a data model for addressing headers with fhirpath - that's already come up.

Grahame Grieve (Jul 19 2018 at 12:26):

I'd certainly be comfortable saying that the rules must be either fhirpath or queries

Jason Walonoski (Jul 19 2018 at 12:28):

I think @Richard Ettema should weigh in here, since he has used TestScript.rule much more than I have and has probably a good idea of the uses/edge-cases around it.

Grahame Grieve (Jul 19 2018 at 12:28):

indeed that would be a great idea

Richard Ettema (Jul 19 2018 at 17:00):

@Grahame Grieve, @Jason Walonoski, a couple responses:

Regarding TestScripts and FHIRPath – FHIRPath support already exists in the TestScript in the following elements:

• TestScript.variable.expression - the FHIRpath expression to evaluate against the source fixture contents that determines the value of this variable
• TestScript.setup.action.assert.compareToSourceExpression - the FHIRpath expression to evaluate against the source fixture contents that determines the comparison value
• TestScript.setup.action.assert.expression - the FHIRPath expression to be evaluated against the operation request or response fixture context either as a standalone boolean expression or path to the comparison value

TestScript Rules and Rulesets - The TestScript.rule and TestScript.ruleset elements were introduced in STU3 as a means to provide access to more complex processing logic than could be provided with the basic assert definition. For Touchstone, this allowed us to introduce a Rules Engine to provide this capability where we implemented support for Groovy, XSLT and Schematron. Our main use cases for Rules and Rulesets are supporting conditional asserts based on the system’s CapabilityStatement and conditional aspects of the FHIR specification. All of AEGIS’s TestScripts, rules, and rulesets are open-source and easily accessible from the Touchstone UI by guest users.

As Jason has explained, the rule and ruleset elements have caused an interoperability issue with Crucible. For quite some time I have been thinking about how to best resolve this and have come up with the following proposal for R4:

1. Remove TestScript.rule, TestScript.ruleset, TestScript.setup.action.assert.rule and TestScript.setup.action.assert.ruleset
2. Add new, optional Touchstone-based TestScript extensions to replace the removed TestScript rule and ruleset elements. Proposed new Touchstone extension canonical urls:

• http://touchstone.aegis.net/touchstone/fhir/StructureDefinition/testscript-rule
• http://touchstone.aegis.net/touchstone/fhir/StructureDefinition/testscript-ruleset
• http://touchstone.aegis.net/touchstone/fhir/StructureDefinition/testscript-assert-rule
• http://touchstone.aegis.net/touchstone/fhir/StructureDefinition/testscript-assert-ruleset
  As Touchstone-based extensions, other FHIR test engines could then safely ignore them if present in any TestScript instances.

Thanks

Lloyd McKenzie (Jul 19 2018 at 17:41):

Rather than have them be touchstone specific extensions, they could be HL7 standard extensions. The notion of wanting to embed Groovy or XSLT tests would presumably be generic.

Lloyd McKenzie (Jul 19 2018 at 17:41):

(Even if not part of the 80%)

Jason Walonoski (Jul 19 2018 at 18:20):

Regarding changing to extensions... I need to think about this. Are these assertions absolutely necessary for determining if a test passes or fails? If so, making them extensions... doesn't feel completely correct either. If not, then... maybe that's OK.

Jason Walonoski (Jul 19 2018 at 18:22):

Thanks for the thorough and thoughtful response @Richard Ettema

Richard Ettema (Jul 19 2018 at 18:26):

@Jason Walonoski, the (current) definition of an assert is that if any assert fails then the test fails. I don't believe that logic will change if rules and rulesets move to extensions. (Your welcome ;)

Richard Ettema (Jul 19 2018 at 18:29):

@Lloyd McKenzie, my thinking on making these extensions touchstone specific was not that they would be touchstone specific but rather part of an implementation guide for touchstone. I'm not sure if that makes a difference to whether or not they are HL7 standard extensions.

Lloyd McKenzie (Jul 19 2018 at 18:50):

If the extensions are general purpose and likely to be used by others, define them in HL7. In your IG, you can declare which ones you support/require.

Jason Walonoski (Jul 19 2018 at 19:41):

My question isn't whether or not the test would pass or fail without the extensions (because you are right only the assert matters)... but whether or not the execution of the extension should cause it to pass or fail.

So, in extension-speak, is the rule an extension or would it really be a modifierExtension? I think the answer is the latter. In which case, we're kind of back to square one...

Grahame Grieve (Jul 19 2018 at 20:38):

no it wouldn't be a modifier extension; they would be extra tests. @Richard Ettema can you provide examples of what you are using these for?

Jason Walonoski (Jul 19 2018 at 22:20):

Well, I would argue that if the rules are required for the test to pass in spirit... for example, the only meaningful and relevant assertion to a test is the rule... then it absolutely would be required, and then it probably ought to be a modifierExtension. Because without knowing that extension, the meaning of the test is different. I realize that is a theoretical argument, but without a lot of concrete examples of where Touchstone uses it, it is hard to eliminate that as a possibility.

Grahame Grieve (Jul 19 2018 at 22:38):

the definition of modifier extension is very precise now. It might be a modifier but it would also depend on the exact wording of the elements as well

Richard Ettema (Jul 19 2018 at 23:04):

A simple example would be verifying an HTTP Header was present if a system supported versioning on a specific resource type. This requires that the system's CapabilityStatement be parsed to determine their declared support for versioning of, say, the Patient resource type. Based on that evaluation, the actual test for, say, the presence of the ETag header can be done or not done (skipped). To simplify this logic we can first test if the ETag header was not present because if it is, then there is no need to test for versioning support.

So, the resulting rule logic that would perform the above "assert" looks like this:

if (response.header('ETag').empty()) // only check capStmt if ETag not present
{
    boolean supports = supportsViaFhirPath(
        "rest.where(mode.value='server').resource.where(type.value='Patient').versioning",
        "versioned,versioned-update", "in", "Patient versioning", serverCapStmt);

    assert !supports: "ETag header expected but not found when Patient versioning is supported by server capability statement";
}

Grahame Grieve (Jul 19 2018 at 23:06):

so we could do all that with FHIRPath assertions

Richard Ettema (Jul 19 2018 at 23:08):

The FHIRPath expression would need access to both the system's CapabilityStatement and the executed operation context.

Grahame Grieve (Jul 19 2018 at 23:08):

yes, but that seems a pretty minimal thing to achieve. %capabilities and %operation, for exaple

Richard Ettema (Jul 19 2018 at 23:11):

Ok. But FHIRPath today does not have those.

Grahame Grieve (Jul 19 2018 at 23:12):

those are context variables defined where FHIRPath is used. So you just have to say 'these context variables are defined' in TestScript, and then you can use them

Richard Ettema (Jul 20 2018 at 00:08):

Ok. Assuming I get the context variables %capabilities and %operation defined such that our FHIR test engine undertand how to send them to our FHIRPath engine and get our FHIRPath engine to understand them, I can see the equivalent FHIRPath expression as something like this:

(%capabilities.rest.where(mode.value='server').resource.where(type.value='Patient').versioning in 'versioned,versioned-update') and (%operation.response.header('ETag').notEmpty())

Something to keep in mind with this approach is that in order to support interoperability between FHIR test engines the definition and use of these context variables, %capabilities and %operation, will need become part of the specification which forces all FHIR test engines to support them. That make this feel like the current issue we have with rule and ruleset.

Richard Ettema (Jul 20 2018 at 00:18):

The issue is a little more problematic regarding the %capabilities context variable. The TestScript allows for multiple actors, both client and server, in the defined tests. We would need to support a %capabilities variable for each actor; e.g. %capablities1, %capabilities2, etc.

Grahame Grieve (Jul 20 2018 at 00:34):

well, more likely it's a collection you can use where with or something. but anyway, passing variables to the FHIR path engine is core FHIR Path functionality, so it should not require an update to the engine

Adam (Jul 23 2018 at 06:36):

@Jason Walonoski I've sent you the url :grin:

Last updated: Apr 12 2022 at 19:14 UTC