Stream: TestScript Resource
Topic: Conditional Processing
ryan moehrke (May 18 2021 at 14:43):
As per our discussion during the connectathon, we are interested in adding conditional processing to TestScript but needed more solid use cases to drive what we need/should implement in the resource
We also wanted a implementation, mocked up or otherwise, to test these use cases against and to help us figure out which design is best from a complexity vs flexibility vs processing requirements.
ryan moehrke (May 18 2021 at 14:44):
to facilitate this I had mocked up some potential solutions to conditional processing needs:
Loop based on Test Id:
test id=test1
name+description etc.
action
operation does a polling read
action
assert
checks polling came back completed
LoopTest
Wait: ${waitUntilValueFromPoll}
When: OnFail
test id=test2
name+description etc.
action
operation does a read on completed async result
action
assert like normal
Loop Based on action.loopReference
test id=test1
name+description etc.
action
loopReference: pollingOperation
operation does a polling read
action
assert
checks polling came back completed
JumpTo:
Where: pollingOperation
When: OnFail
Wait: ${waitUntilValueFromPoll}
test id=test2
name+description etc.
action
operation does a read on completed async result
action
assert like normal
test id=test1
name+description etc.
action
operation does a POST
action
assert
type: structural (of structural|conformance|warning)
is the body correct resourceType
JumpTo:
Where: skipBodyVal
When: OnFail
action
assert
validate profileId
action
loopReference: skipBodyVal
assert
other important asserts
Conditional actions
test id=test1
name+description etc.
action
operation does a POST
action
doIf: is the body correct resourceType
assert
validate profileId
action
assert
other important asserts
ryan moehrke (May 18 2021 at 14:50):
And to start off the discussion of use-cases:
Polling requests for subscriptions and/or bulk data require an indeterminate amount of requests to the polling endpoint - especially servers that reject requests to the polling endpoint after a success has given the client the needed data as one would need the absolute perfect number of polling requests too few and too many would not process correctly. So some way to define if and how many times an operation and their asserts would need to be run/repeated is important to defining one testscript for the full polling use cases
John Moehrke (May 18 2021 at 16:40):
seems this is re-inventing scripting that has already been standardized.
John Moehrke (May 18 2021 at 16:40):
reminds me of a dilbert
Jorn Duwel (May 25 2021 at 10:18):
Our use-cases for conditional processing as mentioned in the Connectathon:
Note up front: We test if implementers can exchange FHIR resources based on dummy data that we provide, so next to testing the infrastructure we also test if they have entered our specific data correctly. minimumId has too much limitations (maybe this is a separate discussion), that is why we test this with asserts.
The FHIRpath expressions we test this with are growing pretty complex pretty quick (maybe because the asserts expressions do not recognize any 'context', so they cannot be broken down into smaller pieces). If an implementer fails the large expression, we would like to offer some extra guidance on where the error exactly is and how to solve this. This can perhaps be done looking in two directions (all element names are reflecting their exact intended functionality):
messageOnFail
<test>
<action>
<operation/>
</action>
<action>
<messageOnFail value="Guidance on how to solve this"/>
<assert>
<label value="test1"/>
<expression value="false"/>
</assert>
</action>
</test>
Execute additional asserts (three different variants of @ryan moehrke 's doIf, using assert.label:
<test>
<action>
<operation/>
</action>
<action>
<assert>
<label value="test1"/>
<expression value="..."/>
</assert>
<assert>
<doIfExpression value="${expressionIsTrue}"/>
<expression value="..."/>
</assert>
<assert>
<doIfTrue value="test1"/>
<expression value="..."/>
</assert>
<assert>
<doIfFalse value="test1"/>
<expression value="..."/>
</assert>
</action>
</test>
Another use case I mentioned is that implementers can provide a PDF as both Binary resource and inline data. At the moment we provide separate TestScripts for each use case and implementers have to select the correct one themselves. It would be more user friendly and easier to maintain if this branching could be done in 1 TestScript:
<test id="test1">
<executeWhenExpression value="${expressionIsTrue}"/>
<executeWhenTestPass value="test1"/>
<executeWhenTestFail value="test2"/>
<action>
...
</action>
</test>
<test id="test2">
<executeWhenTestPass value="test1"/>
<action>
...
</action>
</test>
<test id="test3">
<executeWhenTestFail value="test1"/>
<action>
...
</action>
</test>
There are obvious parallels between these mockups, only difference is one being on assert level and the other on test level. Maybe this can be generalized. I also see a lot of dependencies on the 'variables: how and where' discussion we had at the Connectathon
Sunil Bhaskarla (Jun 25 2021 at 15:50):
@ryan moehrke According to TestReport Scope and Usage, "The TestReport structure mirrors the TestScript concepts of having sections for setup, tests, and teardown." Have you considered the TestReport that mirrors the TestScript loop construct idea?
Last updated: Apr 12 2022 at 19:14 UTC
