Stream: implementers
Topic: TestScript - test scope
Jose Costa Teixeira (Apr 27 2020 at 13:29):
How would we say "This TestScript covers Profile XX" ? Or "This TestScript covers Transaction YY"?
Jose Costa Teixeira (Apr 27 2020 at 13:31):
"Transaction" is just an example and can be any (other) FHIR resource.
Jose Costa Teixeira (Apr 27 2020 at 13:31):
any examples of associating TestScripts to the thing they are testing?
Jose Costa Teixeira (Apr 27 2020 at 13:32):
TestScript.metadata.capability.capabilities is:
- only 1..1
- only CapabilityStatement
Lloyd McKenzie (Apr 27 2020 at 15:02):
@Richard Ettema
Jose Costa Teixeira (Apr 30 2020 at 17:58):
.. @Richard Ettema or anyone?
ryan moehrke (Apr 30 2020 at 18:35):
I believe the CapabilityStatement linked in TestScript is expected to have the relevant operations and resources defined as supported and then having TestScript.metadata.capability.validated set to true would imply what you are asking for but I'm not completely sure
Jose Costa Teixeira (Apr 30 2020 at 18:45):
that could work, but would force me to make one CapStatement for each thing i want to test (at least), right?
John Moehrke (May 01 2020 at 15:51):
isn't that what TestScript.profile is for?
John Moehrke (May 01 2020 at 16:01):
another perspective... shouldn't the IG point at the TestScripts? Not the TestScripts pointing at the IG? so, your use-case is walking through the IG for test scripts.
Jose Costa Teixeira (May 01 2020 at 16:37):
John Moehrke said:
isn't that what TestScript.profile is for?
Yes, if all we want to test is profiles
Jose Costa Teixeira (May 01 2020 at 16:37):
I created J#27007
Jose Costa Teixeira (May 01 2020 at 16:41):
My idea is to have the IG point at the test scripts for the artifacts contained in that IG
Jose Costa Teixeira (May 01 2020 at 16:41):
but I can have a couple of other scripts outside the IG
John Moehrke (May 01 2020 at 17:06):
Jose Costa Teixeira said:
John Moehrke said:
isn't that what TestScript.profile is for?
Yes, if all we want to test is profiles
.profile can point at ANY resource type, not just structureDefinition
Jose Costa Teixeira (May 01 2020 at 17:22):
Oops, true.
Jose Costa Teixeira (May 01 2020 at 17:22):
The name fooled me
Jose Costa Teixeira (May 01 2020 at 17:23):
It doesn't support version, but at least it's a start
Jose Costa Teixeira (May 01 2020 at 17:23):
thanks
Jose Costa Teixeira (May 27 2020 at 18:59):
I've created a ticket to check whether this is correct - that the TestScript.profile is "the thing that this TestScript intends to cover". The description could be improved and the name is misleading.
Jose Costa Teixeira (May 27 2020 at 19:01):
I'd prefer a different name, because we could test other things than profiles -
We need CapStatement, StructureDefs (extensions and profiles) and PlanDefinitions
Jose Costa Teixeira (May 27 2020 at 19:01):
@Richard Ettema ?
Jose Costa Teixeira (May 27 2020 at 19:02):
ryan moehrke (May 27 2020 at 19:36):
In Touchstone (and I believe as intended) TestScript.profile is used to declare an Id Alias for assert.validateProfileId not to show what the testscript is intended to cover. If you find TestScript.metadata to be lacking, sounds like that's where you should propose a change, no?
Grahame Grieve (May 28 2020 at 03:45):
what's this got to do with metadata?
Jose Costa Teixeira (May 28 2020 at 05:07):
So this is even less clear than I thought. @John Moehrke suggested looking at .profile - which I wouldn't guess by the name or by the description. Now @ryan moehrke are you suggesting TestScript.metadata? Also the name or description of are not very revealing...
Jose Costa Teixeira (May 28 2020 at 05:10):
metadata is "Capabilities that are assumed to function correctly on the FHIR server being tested" -
- I don't want assumed capabilities, I want Capabilities under test.
- I'm not intending to test only FHIR servers
Jose Costa Teixeira (May 28 2020 at 05:12):
what I'm lookin for is the system under test or capability under test
ryan moehrke (May 28 2020 at 12:39):
ryan moehrke said:
I believe the CapabilityStatement linked in TestScript is expected to have the relevant operations and resources defined as supported and then having TestScript.metadata.capability.validated set to true would imply what you are asking for but I'm not completely sure
I already suggested looking under TestScript.metadata (which is distinct from TestScript.meta and I get that's confusing I'm in no way saying it shouldn't be renamed for clarity) but It's my understanding that "metadata" is the element that should be covering your needs.
TestScript.metadata.capability.validated: Whether or not the test execution will validate the given capabilities of the server in order for this test script to execute.
It might not capture exactly how you want to use it. But this reads in the spirit of what you want. I would suggest renaming/expanding TestScript.metadata to cover your needs instead of making a new element that borders on duplication.
Jason Walonoski (May 28 2020 at 13:03):
As I have the dubious distinction of being one of the original designers of the TestScript, I can tell you that metadata was supposed to be both the required capabilities and the capabilities under test.
The required capabilities should be marked with TestScript.metadata.capability.required = true
The capabilities under test should be marked with TestScript.metadata.capability.validated = true
It is possible that a capability can be required but not validated, or required and validated, but it doesn't make a lot of sense for it to be not required or validated.
Hope that helps.
Jose Costa Teixeira (May 28 2020 at 14:28):
- what is the difference between validated and required?
Jose Costa Teixeira (May 28 2020 at 14:28):
- "metadata" - can we find a better name?
ryan moehrke (May 28 2020 at 14:33):
validated is what capabilities are under test
required is what capabilities are needed for the test to run correctly (i.e. you need to Patient Create to load test resources etc.)
Jose Costa Teixeira (May 28 2020 at 14:34):
thanks.
- "validated" - can we find a better name?
Jose Costa Teixeira (May 28 2020 at 14:34):
:)
Jose Costa Teixeira (May 28 2020 at 14:35):
- metadata.link.url: why is that not a Reference?
Jose Costa Teixeira (May 28 2020 at 14:36):
sorry, I meant metadata.capability.link.
Jose Costa Teixeira (May 28 2020 at 14:37):
- what is the difference between metadata.link and metadata.capability.url?
Jose Costa Teixeira (May 28 2020 at 14:38):
(sorry for all these questions, but reading the spec it clearly seemed that these elements were something else)
Jose Costa Teixeira (May 28 2020 at 14:39):
- in metadata and metadata.capability we read "capability/ies that is/are assumed to function correctly on the FHIR server being tested"
Can we change that description? If they are "validated" this means they are not assumed to function correctly, right?
Jason Walonoski (May 28 2020 at 21:32):
@Jose Costa Teixeira I'm not really involved in maintaining TestScript any longer and haven't been for several years. But regarding question #1, @ryan moehrke is correct. Regarding better names or descriptions, probably, but I'm not particularly vested in the outcomes. Any of the link fields are URIs and not References, because they are literally meant to be hyperlinks to sections of the FHIR specification. (e.g. this link http://hl7.org/fhir/R4/http.html#vsearch to a paragraph about variant searches).
Jose Costa Teixeira (May 29 2020 at 04:41):
I understand the reason to test the FHIR spec itself, but we should be able to test the systems and profiles.
Jose Costa Teixeira (May 29 2020 at 04:41):
In which calls is the TestScript resource discussed?
Jose Costa Teixeira (May 29 2020 at 04:42):
I would like to open the few topics aove but I would like to learn more about the current design.
Jason Walonoski (May 29 2020 at 12:28):
I'm not certain about the current design, because it evolved after I was involved. The link can be anything, which is why it is a uri -- you can point to a profile or IG. Actually, the TestScript.profile is meant to specify a profile to validate returned resources against. Use metadata.link to point to any references (and I use that term generically, as in citation) that you want, and use metadata.capability to point to capability statements that go along with IGs.
Jason Walonoski (May 29 2020 at 12:29):
TestScript is also only FMM2, so it has a way to go.
Jason Walonoski (May 29 2020 at 12:30):
Regarding "which calls" -- FHIR Infrastructure. Each resource page tells you which work group owns it.
Richard Ettema (May 30 2020 at 01:48):
@Jose Costa Teixeira,
Apologies for the lengthy response but I'm trying to catch up and cover a lot of ground. Hope this help...
I understand the reason to test the FHIR spec itself, but we should be able to test the systems and profiles.
The TestScript resource type is not designed to "test the FHIR spec itself". The purpose of TestScript is to test the implemented systems and profiles.
TestScript is "A structured set of tests against a FHIR server or client implementation to determine compliance against the FHIR specification." see http://hl7.org/fhir/testscript.html#7.3
TestScript(s) are executed by a FHIR Test Engine as defined by the FHIR spec. see http://hl7.org/fhir/testing.html#execution
TestScript.metadata
Agreed. The name is can be confused with Resource.meta and needs to be re-examined as to its usefulness.
The reality is that, based on our experience with authoring and executing TestScripts against many different testing scenarios, the TestScript.metadata is something we just don't use. We have found that the rest of the TestScript elements provide this information; fixtures (test data), profiles (StructureDefinitions from the base spec to IGs), operations (FHIR interactions that also define the Resource Types) and so on. We parse all of this information within the TestScript to provide appropriate mappings to and recording of this information against the test execution results. This information is then collected, collated and made available for analytics and reporting back to the user.
I've created a ticket to check whether this is correct - that the TestScript.profile is "the thing that this TestScript intends to cover". The description could be improved and the name is misleading.
Based on the previous point and my response, I would have to say the the TestScript.profile does indicate what the TestScript intends to cover in so much as FHIR profiles are concerned.
The TestScript.profile element, as Ryan has described, is used to declare an Id Alias for assert.validateProfileId. This profile id then refers to the canonical url (identifier) of the FHIR Profile so that it can be used in an assert that calls the FHIR validation engine to evaluate/validate the resource payload of an operation request or response against that FHIR Profile.
what I'm lookin for is the system under test or capability under test
Our implementation of the FHIR Test Engine provides an interface for pre-test setup where the destination or target System Under Test (FHIR Server) is selected. The FHIR Test Engine in this case then acts as the client constructing and sending the defined FHIR operation(s) in the TestScript(s). This follows the FHIR spec for testing as defined under the "Operation Execution" - see http://hl7.org/fhir/testing.html#op.
A TestScript can also explicitly define what actors, FHIR Client(s) and Server(s), are needed through the declaration of the TestScript.origin and TestScript.destination elements. When present, the FHIR Test Engine then needs to accept as input the actual System(s) Under Test that match these declarations.
Our testing platform provides our users the ability to register their systems, FHIR Client(s) and Server(s). These registered systems are then made available for selection against the declared TestScript actors. This also provides us with the ability to get and cache a system's CapabilityStatement. We then use the system's CapabilityStatement in our test result reporting where we can let the user filter their results to either include all results or exclude results that do not correspond to the system's declared capabilities.
Jose Costa Teixeira (May 30 2020 at 04:16):
Thanks @Richard Ettema for this comprehensive response. Just to see if you agree and if I understand correctly:
- TestScript.metadata should be renamed to be less likely to confuse with .meta.
- The place to indicate "system under test" IS TestScript.profile, at least for profiles. This should be clear
- What about CapabilityStatements, Operations, PlanDefinitions? I suggest renaming or restructuring TestScript.profile
- TestScript.profile declares an alias for assert.ProfileId. - I don't understand the implications of this, but then it may be an alias to something else, not only a profile, right?
Jose Costa Teixeira (May 30 2020 at 04:21):
I am looking at test design, more than test execution, so I am not looking at .
Concretely, what we want: with every specification that we make (a StructureDefinition, a PlanDefinition...) we want to provide a set of TestScripts to say "This is the PlanDefinition, and this is how you test it".
For this, the TestScripts should have a link to the specification that is going to be tested - not the system, because we are not fixing it to one system yet. We are just saying "whatever system you implement this on, these are the test scripts to see if it was well implemented".
Jose Costa Teixeira (May 30 2020 at 04:26):
With this, I suggest to follow that ticket I created and perhaps do something like :
- Rename .profile to .conformanceArtifact or .featureForTesting or any other name to say what we mean.
- If we want to separate between operations, plandefinitions, capabilityStatements etc, we could make this element more complex, but I believe a Reference(Any) Is good
- Fix "metadata"
- Work on documenting all of this
Jose Costa Teixeira (Jun 01 2020 at 20:50):
Can we rename TestScript.profile to TestScript.scope? I think it is clearer and I cannot think of a concise name (english-speakers: help appreciated)
Jose Costa Teixeira (Jun 01 2020 at 20:53):
As for the text, I'd use this:
"short" : "The conformance artifact(s) that this test script is meant to cover",
"definition" : "The artifacts that the TestScript is intended to cover - i.e. the features under test for this TestScript. Can be a conformance artifact like StructureDefinition (in which case the TestScript can support testing the implementation of the StructureDefinition in a given system) or a definitional artifact like PlanDefinition ( in which case the TestScripts supports testing the execution of a plan definition).",
Jose Costa Teixeira (Jun 01 2020 at 20:58):
For types of resources, I'd leave as Reference(Any). I'm looking at these resources:
- StructureDefinition
- OperationDefinition
- CapabilityStatement
- ImplementationGuide
- SearchParameter
- CompartmentDefinition
- ElementDefinition
- PlanDefinition
- MeasureDefinition
Someone else may want other resources, so I would not constrain here. We can add a valueset for resource type but I don't see the gain
Jose Costa Teixeira (Jun 01 2020 at 20:59):
(I will give my input to J#27533 with the above or any comment in this thread)
Richard Ettema (Jun 02 2020 at 13:37):
TestScript.profile is used to specifically define a FHIR Profile, StructureDefinition's canonical url, for use in asserts that call the FHIR validation engine. Changing the name and definition to something else breaks the current use.
Instead, I would be in favor of proposing TestScript.scope as a new element as described above.
Jose Costa Teixeira (Jun 02 2020 at 20:04):
Added to ticket
Jose Costa Teixeira (Jun 02 2020 at 20:06):
I'm also wondering - don't we have test types/categories?
For the same conformance artefact, one TestScript could be classified as "smoke"/"basic" the other one as "official", the other as "full"
Vassil Peytchev (Jun 03 2020 at 17:34):
Richard Ettema said:
TestScript.profile is used to specifically define a FHIR Profile, StructureDefinition's canonical url, for use in asserts that call the FHIR validation engine.
I am glad that is clarified. The Jira issue got an initial look last Monday, but those of us present did not know if indeed this was the intended use.
Richard Ettema (Jun 03 2020 at 19:45):
@Jose Costa Teixeira regarding
I'm also wondering - don't we have test types/categories?
TestScript follows the conformance related resource types with its initial set of elements. What you suggest could potentially be defined in some of those elements, like:
- TestScript.status - the current codes could be expanded
- TestScript.experimental - boolean so somewhat limited and specific
- TestScript.useContext - this is the most likely existing element that may provide this type of definition
Although, I do like the introduction of
- TestScript.type - code from potential following values: "smoke", "official", "basic", "full", etc.
Just some thoughts...
Jose Costa Teixeira (Jun 03 2020 at 19:47):
status is about the status of the test script itself, so I don't see how it could be used for "this is smoke test" vs "this is regulatory testing"
Jose Costa Teixeira (Jun 03 2020 at 19:47):
same as experimental
Jose Costa Teixeira (Jun 03 2020 at 19:49):
usageContext says "Specifies clinical/business/etc. metadata that can be used to retrieve, index and/or categorize an artifact. This metadata can either be specific to the applicable population (e.g., age category, DRG) or the specific context of care (e.g., venue, care setting, provider of care)."
could work, but in that case would be interesting to enlarge the scope a bit (not only population and context of care)
Richard Ettema (Jun 03 2020 at 19:50):
Agreed. That's why I said useContext is the most likely existing element. I think TestScript.type is the better approach.
Last updated: Apr 12 2022 at 19:14 UTC
