FHIR Chat · TestScript · implementers

Stream: implementers

Topic: TestScript

Andrew Rudenko (Sep 21 2016 at 08:08):

Hello. Can I ask a question about TestScript and Touchstone Project? As I see there is an inconsistency between spec and tests. For example, this test https://touchstone.aegis.net/touchstone/testscript?name=%2fBasic-FHIR1-6-0%2fPatient%2fClient+Assigned+Id%2fPatient-client-id-json&version=2&latestVersion=2 -> Step3-UpdatePatient -> <sourceId value="resource-update"/> . In spec (http://hl7.org/fhir/2016Sep/testing.html#ops) "The sourceId fixture cannot be statically defined because the id cannot be relied upon." for update operations. But "resource-update" is statically defined, isn't it?

I apologize if this is wrong place for such questions. Thank you

Grahame Grieve (Sep 21 2016 at 10:18):

it's not the wrong place. @Richard Ettema this is for you

Richard Ettema (Sep 21 2016 at 12:38):

Yes, this is the right place.
@Andrew Rudenko The testing.html is a new page that got created in the STU3 Ballot (2016Sep) snapshot and came from the original TestScript resource page. The information on this page along with the many How-To examples need to be updated from the DSTU2 spec and this task is on my to-do list in the next couple of weeks.
Would you please create a tracker item for this issue and make sure you reference the url link to this section of the testing.html page? Thank you.

Andrew Rudenko (Sep 21 2016 at 12:50):

@Richard Ettema this tracker http://gforge.hl7.org/gf/project/fhir/tracker/?action=TrackerItemBrowse&tracker_id=677 ?

Andrew Rudenko (Sep 21 2016 at 13:00):

Ok, let me ask a few more questions :)

1) lessThan/greaterThan operators. How should they work? Should values be coerced? By which rules?
2) in operator. Is somewhere described that it should split value by ","?
3) What does direction mean in asserts?
4) path can be "json-path" or "xpath". How can I recognise which one is used in particular place? Touchstone uses xpath for originally json resources, for example.

Andrew Rudenko (Sep 21 2016 at 15:58):

@Richard Ettema one more: 5) touchstone uses variables inside fixtures. but I can't find anything about it in spec

Richard Ettema (Sep 21 2016 at 16:14):

The link to the GForge tracker doesn't go to a new tracker item. Can you verify that link?

Answers to first four questions coming...

Richard Ettema (Sep 21 2016 at 16:15):

1) lessThan/greaterThan operators. How should they work? Should values be coerced? By which rules?

The lessThan/greaterThan operators are intended for numbers only at the moment, for example:
- Assert greaterThan 9.14. Actual value of 9.139 fails. Actual value of 9.141 passes.
- Assert lessThan 9.14. Actual value of 9.139 passes. Actual value of 9.141 fails.
Touchstone supports textual string value comparison such that 'Aaa' is less than 'aaa'.
The contains operator is intended for string values. Since spec doesn’t talk about case sensitivity, it is assumed to be case sensitive.

Richard Ettema (Sep 21 2016 at 16:16):

2) in operator. Is somewhere described that it should split value by ","?
After re-reviewing the spec, yes, this description is missing. Yes, the value list is split by comma ",".

Richard Ettema (Sep 21 2016 at 16:16):

3) What does direction mean in asserts?
direction = "request" - apply the assert to the request (client) side of the message exchange; HTTP headers, payload, etc.
direction = "response" - apply the assert to the response (server) side of the message exchange; HTTP header, payload, etc.

Richard Ettema (Sep 21 2016 at 16:16):

4) path can be "json-path" or "xpath". How can I recognise which one is used in particular place? Touchstone uses xpath for originally json resources, for example.
A FHIR test engine needs to support both xpath and jsonpath and the general assumption (probably needs to be defined) is that you would use xpath when the request and/or response payload is XML and jsonpath for JSON payload. Touchstone has implemented support for use of xpath or jsonpath against either XML or JSON. So, in the majority of the TestScript deployed in Touchstone we use xpath by default. I have run into a couple instances where I needed to resort to jsonpath for explicit JSON references.

Andrew Rudenko (Sep 21 2016 at 16:36):

3) Some asserts can't be applied to request (response, responseCode) and most of the rest explicitly defines "response" as a target in documentation (http://hl7.org/fhir/2016Sep/testing.html#assert), for example, "Asserts that compareToSourcePath against the response body..."

Andrew Rudenko (Sep 21 2016 at 16:36):

4) In https://touchstone.aegis.net/touchstone/testscript?name=%2fBasic-FHIR1-6-0%2fPatient%2fClient+Assigned+Id%2fPatient-client-id-json&version=2&latestVersion=2 it uses xpath for JSON payloads.

nicola (RIO/SS) (Sep 21 2016 at 16:46):

Are we going to switch to fluentpath sometimes?

Richard Ettema (Sep 21 2016 at 17:31):

@nicola Yes, we are planning on adding support for fluentpath (name changing back to fhirpath). We need to add a GForge tracker for that.

nicola (RIO/SS) (Sep 21 2016 at 17:34):

@Richard Ettema we have some ideas how to make tests more declarative. Are you going to Dev Days in Amsterdam?

Richard Ettema (Sep 21 2016 at 17:44):

@nicola I am hoping to get approval to go. I should know for sure in a few days.

Michel Rutten (Sep 21 2016 at 17:58):

@Richard Ettema You SHOULD join! ;p

nicola (RIO/SS) (Sep 21 2016 at 18:00):

@Richard Ettema +1 hope to see u there and have a discussion. @Andrew Rudenko will be there too!

Richard Ettema (Sep 21 2016 at 18:03):

Tracker for TestScript fhirpath support - http://gforge.hl7.org/gf/project/fhir/tracker/?action=TrackerItemEdit&tracker_item_id=12140

Richard Ettema (Sep 21 2016 at 18:20):

@Andrew Rudenko 3) Yes. The spec doesn't prevent the response, responseCode elements from being populated when the direction = "request". I have a created a new tracker for this - http://gforge.hl7.org/gf/project/fhir/tracker/?action=TrackerItemEdit&tracker_item_id=12142

Richard Ettema (Sep 21 2016 at 18:22):

4) Yes. This is a good example of where we use XPath against a JSON payload. This is convenience feature we implemented in Touchstone.

Andrew Rudenko (Sep 21 2016 at 18:39):

@Richard Ettema 4) so, how should engine choose: xpath or json path? )

Richard Ettema (Sep 21 2016 at 18:57):

@Andrew Rudenko Sorry. I meant to say that since the spec defines that the path can be either XPath or JSONPath we built Touchstone to allow the use of either regardless of the payload format. If you're building a test engine, I can see where this may be difficult as this capability took us a while to figure out. If you would like to suggest that a test engine only support XPath for XML and JSONPath for JSON, then please add a new tracker item to clarify/constrain the spec.

Richard Ettema (Sep 21 2016 at 19:41):

5) Yes, we saw this as a needed feature for the test engine and added it to Touchstone. Yes, we need a new tracker to have this defined in the spec. Thanks!

Jason Walonoski (Sep 22 2016 at 12:07):

We should probably migrate to FluentPath to be consistent with the rest of the specification.

Richard Ettema (Sep 22 2016 at 13:33):

In FHIR-I Q4 yesterday the resolution to tracker #12140 was to:
Replace the current "path" elements with *two* elements: one for an xpath, and one for a fhirpath, as we do in ElementDefinition.constraint (name them like "expression" and "xpath").

Andrew Rudenko (Sep 23 2016 at 06:39):

@Richard Ettema > we built Touchstone to allow the use of either regardless of the payload format

but how does it choose right *path?

Richard Ettema (Sep 23 2016 at 13:21):

We built a transformation from/to XPath <--> JSONPath. When the content is XML and JSONPath is present, we transform the JSONPath to XPath and vice versus.

Andrew Rudenko (Sep 23 2016 at 13:23):

@Richard Ettema i'm asking how do you determine what this is JSONPath (and should be transformed to XPath) ;)

Andrew Rudenko (Sep 23 2016 at 13:26):

by comparing first character with "$"? Maybe there should be a special flag, especially if you going to add new selectors engines?

Richard Ettema (Sep 23 2016 at 16:01):

Our logic is "If the path contains '/' and it doesn’t contain '?(@', then we consider it to be an XPath".

Richard Ettema (Sep 23 2016 at 16:02):

I actually had a discussion with Grahame this morning and we will be updating the specification for TestScript to use the fhirpath (fluentpath) expression language.

Andrew Rudenko (Sep 23 2016 at 16:05):

ok, thankyou )

Grahame Grieve (Sep 24 2016 at 01:48):

@nicola - given the timeline of STU3, please dsicuss the changes to STU3 with @Richard Ettema as soon as possible - devdays is too late

Jason Walonoski (Oct 06 2016 at 19:11):

@Grahame Grieve @Richard Ettema I mocked up the first pass of the TestReport resource coming out of our conversation in Baltimore. TestReport-FHIR-v1.7.0.pdf testreport-example.xml. These changes have not been pushed to SVN.

Grahame Grieve (Oct 06 2016 at 19:22):

score - what's that out of? (as long as the definition says)
shouldn't date be mandatory (and an 'instant')?
Origin is unexpectedly complex to me. same for destination
I'm not sure what label & description are- I would have just expected result / message / link-to-details
generally , linking to a detailed report isn't there

Richard Ettema (Oct 06 2016 at 19:52):

I'm presuming 'score' is the percentage pass for all the tests.
I agree on the origin and destination. You could probably just have the index and participant elements. The link back to the TestScript will give you the extra details.

Jason Walonoski (Oct 06 2016 at 20:32):

The origin and destination are complex, but they are in TestScript as well, this just mirrors the structure. I was trying to be consistent.

Jason Walonoski (Oct 06 2016 at 20:32):

Score would be a percentage of tests passed.

Jason Walonoski (Oct 06 2016 at 20:32):

Date should be mandatory, and instant. Yes.

Grahame Grieve (Oct 06 2016 at 20:33):

score - fine. was just checking. I don't understand the testscript origin/destination either

Jason Walonoski (Oct 06 2016 at 20:37):

Label and description was meant as a way for the TestReport to have a simple label or description of the result beyond just "error" or "skip" (or whatever). This also mirrors the TestScript structure.

Jason Walonoski (Oct 06 2016 at 20:37):

I thought about not having origin/destination, and just reference the server being tested, but I figured I'd get push back for not including all the other options (clients, form fillers, etc). I'm okay with simplifying it.

Grahame Grieve (Oct 06 2016 at 20:38):

I vote for simplifying and let pushback be towards complexity, not the other way around

Jason Walonoski (Oct 06 2016 at 20:40):

OK on simplifying. I'll revise tomorrow morning.... also, the instant should probably be 0..1 in case the report is still pending (which is one of the legal statuses)

Grahame Grieve (Oct 06 2016 at 20:43):

ok

Lloyd McKenzie (Oct 07 2016 at 04:15):

I think date shold be a dateTime, not an instant. TestReport should be limited to automated test execution, so it's possible that in some cases only a date or date + time without milliseconds could be present

Grahame Grieve (Oct 07 2016 at 06:56):

why, if it's limited to automated execution, would milliseconds not be present?

Jason Walonoski (Oct 07 2016 at 12:55):

Updates attached from discussion... using instant for the time being since that piece seems unresolved.

TestReport-FHIR-v1.7.0.pdf
testreport-example.xml

Richard Ettema (Oct 07 2016 at 14:02):

I like the simplification. It's a good starting point for implementation.

Richard Ettema (Oct 07 2016 at 14:02):

I think the operation.message and assert.message data type should be markdown. This would allow for more structure to the message content if desired.

Lloyd McKenzie (Oct 07 2016 at 16:53):

First, milliseconds are unlikely to be relevant. Second, why are you limiting it to automated execution?

Jason Walonoski (Oct 07 2016 at 17:41):

I'll make the change for markdown.

Jason Walonoski (Oct 07 2016 at 17:42):

Lloyd, it seems unlikely that a person will manually execute all the operations and assertions in a TestScript resource. Also, this is equivalent to an Observation.issued or DiagnosticReport.issued attribute, which are both instant, and which both refer to automated services.

Jason Walonoski (Oct 07 2016 at 17:43):

But, I'll make the change to dateTime... I don't think milliseconds are terribly important here.

Jason Walonoski (Oct 07 2016 at 17:48):

Updates with markdown and dateTime
TestReport-FHIR-v1.7.0.pdf
(example unchanged)

Jason Walonoski (Oct 07 2016 at 17:51):

At this point, is it OK if I add these to the current/latest in SVN? Not sure what the process is any longer.

Lloyd McKenzie (Oct 07 2016 at 19:01):

I don't agree with Observation.issued or DiagnosticReport.issued being instants either. DiagnosticReport.issued is particularly problematic because it's mandatory. The question is: If I do some manual testing, can I capture and share the results of that using TestReport or not? I think the answer should be yes. And if the answer *is* yes, then the resource design should take that into account.

Grahame Grieve (Oct 07 2016 at 19:42):

it's hard to imagine someone doing a manual test run

Grahame Grieve (Oct 07 2016 at 19:43):

Jason - looks like time to commit. But we didn;t talk about making a formal Resource proposal. Does someone want to do that?

Richard Ettema (Oct 07 2016 at 20:14):

Do we need a gforge tracker created for a formal Resource proposal?

Richard Ettema (Oct 07 2016 at 20:18):

Meaning... is there a tracker already created or would you like me to go ahead and create one?

Richard Ettema (Oct 07 2016 at 20:27):

Nevermind. I just started a "TestReport_FHIR_Resource_Proposal" page.

Richard Ettema (Oct 07 2016 at 20:37):

http://wiki.hl7.org/index.php?title=TestReport_FHIR_Resource_Proposal has been created and updated with similar content as the TestScript proposal page.

Grahame Grieve (Oct 12 2016 at 20:54):

@Jason Walonoski @Richard Ettema my server at http://fhir3.healthintersections.com.au now supports TestReport - just for testing, is it worth setting up a push from touchstone/crucible to my server with test reports

Richard Ettema (Oct 12 2016 at 21:55):

@Grahame Grieve We normally wait until an official FHIR snapshot release; i.e. Connectathon event or the official STU3, is created before adding support for that release into Touchstone. That being said, I would like to get support for TestReport into Touchstone sooner than later. I see if I have time in the next few days to implement support for the current 1.7.0 spec.

Richard Ettema (Oct 12 2016 at 21:56):

Question - what will be the FHIR version number for STU 3 when it's officially released? 2.0.0?

Grahame Grieve (Oct 12 2016 at 22:40):

yes that's the plan.

Jason Walonoski (Oct 13 2016 at 11:57):

@Grahame Grieve we have a few things to do before we'd be in a position to push TestReport to your server, but we can work toward that goal.

Grahame Grieve (Oct 14 2016 at 19:14):

ok thx

Grahame Grieve (Mar 22 2017 at 21:26):

@Richard Ettema - I don't have any idea what TestScript.profile is for, after reading the documentation

Richard Ettema (Mar 22 2017 at 21:29):

TestScript.profile defines the FHIR Profile(s) [StructureDefinition(s)] to be used in subsequent asserts where the test engine invokes a FHIR validation engine against the request or response payload; e.g.

<profile id="resource-profile">
    <reference value="http://hl7.org/fhir/StructureDefinition/Patient"/>
</profile>

Richard Ettema (Mar 22 2017 at 21:30):

Yes, the current definition is somewhat lacking. Sorry I missed updating that.

Grahame Grieve (Mar 22 2017 at 21:31):

why not just refer to the profile directly when you use it? What does an id based internal reference buy?

Grahame Grieve (Mar 22 2017 at 21:32):

I think that TestScript.setup.action.operation.type should be a URI. We define standard URIs for the standard interactions on http.html, and otherwise is the canonical URL of the operation that is being tested

Grahame Grieve (Mar 22 2017 at 21:35):

I reckon that the testscript resource should reference the testing.html page in it's introduction

Richard Ettema (Mar 22 2017 at 21:36):

re: TestScript.profile - good point. It's been defined that way from the beginning and I never went back and questioned it.

Richard Ettema (Mar 22 2017 at 21:38):

The TestScript resource page section 7.3.3.1 Background has a link to testing.html. Yes, moving it up under Introduction would make it more noticeable.

Grahame Grieve (Mar 22 2017 at 21:38):

I've read the documentation of TestScript.setup.action.operation.targetId several times, and I haev no idea what it's for

Richard Ettema (Mar 22 2017 at 21:43):

There's an example of using targetId in the How-To section of the testing.html page under http://hl7.org/fhir/STU3/testing.html#7.2.2.3 "Perform delete operation in teardown".

Richard Ettema (Mar 22 2017 at 21:47):

re: TestScript.setup.action.operation.type - I've read over the http.html page again and I'm not seeing the standard URIs for the standard interactions. How would extended operations be handled otherwise?

Richard Ettema (Mar 22 2017 at 21:53):

Sorry, you're talking about the OperationDefintion.url, right? e.g. "http://hl7.org/fhir/OperationDefinition/Resource-validate" for $validate.

Grahame Grieve (Mar 22 2017 at 21:55):

yes. (and we would have to define URLs for the standard interactions)

Richard Ettema (Mar 22 2017 at 21:56):

Yes. Agreed. That would be a better way to go.

Grahame Grieve (Mar 22 2017 at 21:59):

suggest that you add 'a target id could be used to delete a resource in a teardown operation' to the comments of the target id, and/or review it's definition, because I still don't understand "Id of fixture used for extracting the [id], [type], and [vid] for GET requests"

Richard Ettema (Mar 22 2017 at 22:01):

Ok. Will do.

Grahame Grieve (Mar 23 2017 at 02:38):

ok. there's a lot about the assert that I don't follow.

Grahame Grieve (Mar 23 2017 at 02:42):

I'd still much rather it be simple:

<assert>
   <name value=""/>
  <description value=""/>
  <source value="[what is being tested: request-header | request-body | response-header | response -body | fixture"/>
  <sourceId value="id of what is being tested - fixture id, header name"/>
  <expressionType value="fhirpath | xpath | regex | javascript"/> 
  <expression value="per above"/>
</assert>

Grahame Grieve (Mar 23 2017 at 02:43):

anyway, I sent you a draft of a test script for comment

Grahame Grieve (Mar 23 2017 at 23:36):

@Richard Ettema I don't understand the cardinality of TestScript.setup.action.assert - shouldn't that be 0..*? How can I have multiple asserts about the outcome of an operation?

Richard Ettema (Mar 23 2017 at 23:40):

The containing element "action" has the cardinality of 0..*. A test will have 1 or more actions which are combinations of 1 or more "operation"s; each "operation" can be followed by 0 or more "assert"s.

Grahame Grieve (Mar 23 2017 at 23:41):

so is there any difference between one action with an operation and an assert, and two actions, one with an operation, and one with an assert?

Richard Ettema (Mar 23 2017 at 23:42):

If you look at the invariant inv-1, an action can only contain either an operation or an assert.

Grahame Grieve (Mar 23 2017 at 23:43):

pfft. I thought I checked that. sorry

Richard Ettema (Mar 23 2017 at 23:45):

np. You can go to Touchstone and look at any of the the TestScripts via the "Test Definitions" menu item. You don't even have to be logged in.
pasted image

Grahame Grieve (Mar 23 2017 at 23:50):

while you're here... Asserts...

Grahame Grieve (Mar 23 2017 at 23:51):

I'm not sure how to use the assert to say what I want.

Richard Ettema (Mar 23 2017 at 23:52):

Ok. I've got about 10 minutes before I have to sign off. Describe what you want and I'll do my best to translate to an assert. :)

Grahame Grieve (Mar 23 2017 at 23:52):

so I want to make a fhir path assertion that compares input from one of the fixtures to the result of an operation

Richard Ettema (Mar 23 2017 at 23:55):

So you want to compare the contents of a static fixture to the response body/payload of an operation?

Grahame Grieve (Mar 23 2017 at 23:56):

yes

Richard Ettema (Mar 23 2017 at 23:58):

The minimumId assert element currently provides that functionality; for example,

<action>
    <assert>
        <description value="Confirm that read returns updated version and that update did not delete previously created fields."/>
        <direction value="response"/>
        <minimumId value="resource-update-min"/>
    </assert>
</action>

Grahame Grieve (Mar 23 2017 at 23:59):

I don't really understand direction, for a start

Richard Ettema (Mar 23 2017 at 23:59):

The minimumId points to a static fixture. The assert will have the test engine do a comparison of the static fixture to the response body of the read.

Richard Ettema (Mar 23 2017 at 23:59):

If direction="request", the comparison would be to the request body from the client.

Grahame Grieve (Mar 24 2017 at 00:00):

but that's not what i want. I have a specific FHIRPath assertion about the difference between the two resoruces

Richard Ettema (Mar 24 2017 at 00:00):

Understood. I'm just trying to give you some context.

Grahame Grieve (Mar 24 2017 at 00:00):

and that didn't bring me any clarity. Why not just identify what is being compared by the id it is assigned?

Grahame Grieve (Mar 24 2017 at 00:02):

currently, I have this:

      <assert>  
        <label value="check-elements"/>
        <description value="The snapshot elements must be the same"/>
        <direction value="response"/>
        <expression value="fixture('simple1-output').snapshot.element.all($this = fixture('patient').snapshot.element.where(path = $this.path)"/>
      </assert>

Grahame Grieve (Mar 24 2017 at 00:02):

though, as I said above, looking at assert fills me with an urgent desire to simply and calrify it

Richard Ettema (Mar 24 2017 at 00:03):

Is that FHIRPath expression currently valid? I don't recognize the fixture() method.

Grahame Grieve (Mar 24 2017 at 00:04):

it's an extension for this context (such extensions are allowed)

Grahame Grieve (Mar 24 2017 at 00:04):

but I don't know how else you would do it

Richard Ettema (Mar 24 2017 at 00:05):

Ok. Is that extension supported by the Java RI FHIRPathEngine? I ask because that's what I'm using in Touchstone.

Grahame Grieve (Mar 24 2017 at 00:06):

there's an API for regsitering extension methods

Grahame Grieve (Mar 24 2017 at 00:06):

see org.hl7.fhir.dstu3.utils.FHIRPathEngine.IEvaluationContext

Grahame Grieve (Mar 24 2017 at 00:07):

and I think that any useful use of FHIRPath will require the ability to refer to multiple fixtures

Grahame Grieve (Mar 24 2017 at 00:07):

in this context, anwyay

Richard Ettema (Mar 24 2017 at 00:08):

I was just about to ask. Thanks. Yes, very useful.
I'll have to investigate how this would work in Touchstone.

Richard Ettema (Mar 24 2017 at 00:08):

Right now I'm using FHIRPath expressions like the following:

<action>
    <assert>
        <description value="Confirm that the returned resource contains the expected start datetime value."/>
        <direction value="response"/>
        <expression value="Slot.start = @2016-10-25T09:00:00Z"/>
    </assert>
</action>

Grahame Grieve (Mar 24 2017 at 00:09):

yes that will work, but as soon as you have content that depends on other resources, then you need a fixture or a variable. I expect we'll need variable() as well

Richard Ettema (Mar 24 2017 at 00:10):

Yes, it's fairly simple at the moment but I definitely want to explore getting more complex expressions implemented.

Richard Ettema (Mar 24 2017 at 00:11):

Sorry, I have to drop now - just as things were getting interesting. :) I'd like to pick this up again later. Thanks.

Grahame Grieve (Mar 24 2017 at 00:11):

ok

Grahame Grieve (Mar 24 2017 at 01:11):

FYI:

   @Override
    public FunctionDetails resolveFunction(String functionName) {
      if ("fixture".equals(functionName))
        return new FunctionDetails("Access a fixture defined in the testing context", 0, 1);
      return null;
    }

    @Override
    public TypeDetails checkFunction(Object appContext, String functionName, List<TypeDetails> parameters) throws PathEngineException {
      if ("fixture".equals(functionName))
        return new TypeDetails(CollectionStatus.SINGLETON, TestingUtilities.context.getResourceNamesAsSet());
      return null;
    }

    @Override
    public List<Base> executeFunction(Object appContext, String functionName, List<List<Base>> parameters) {
      if ("fixture".equals(functionName)) {
        String id = fp.convertToString(parameters.get(0));
        Resource res = fetchFixture(id);
        if (res != null) {
          List<Base> list = new ArrayList<Base>();
          list.add(res);
          return list;
        }
      }
      return null;
    }

Grahame Grieve (Mar 28 2017 at 04:44):

@Richard Ettema it's not clear to me how a test script differentiates between errors and fatal errors - is there anyway to say 'just stop running the tests'?

Richard Ettema (Mar 28 2017 at 14:46):

@Grahame Grieve Let me address your two issues separately...

• how a test script differentiates between errors and fatal errors

The short answer is no. During our working group discussions during May and June 2015, the concept of a "fatal" error was not mentioned. It’s not clear what fatal would be: A prerequisite that’s not met in the test script? A fatal error within the test engine? A fatal error in the test system? The current behavior expected of a test engine is to mark the overall test script execution as Failed if the setup or any of the tests fail.

• is there anyway to say 'just stop running the tests'?

The currently defined behavior (testing.html) of the test engine's processing of the TestScript regarding failures is:
7.2.1.1 Workflow - http://hl7.org/fhir/STU3/testing.html#7.2.1.1
Setup Execution

All operations in a setup section (including assertions) must complete successfully for the subsequent tests to be executed. If an assertion operation in the setup section fails, then execution and evaluation of the tests in the TestScript should be skipped.

7.2.1.5 Operation Execution - http://hl7.org/fhir/STU3/testing.html#7.2.1.5

If an operation or assertion fails, then the test ends in failure and the test script execution proceeds to the next test.

What this means is that (currently) the only way for a test engine to not execute all the defined tests is for the setup to fail. Otherwise, each individual test will execute. An individual test execution will be halted if an operation or assert fails within the definition of that test.

A design enhancement to allow for conditional execution of a test based on a prior test execution result would be needed to allow for a test engine to stop running all the (remaining) tests in that situation.

HTH

Last updated: Apr 12 2022 at 19:14 UTC