FHIR Chat · TestScript capabilities · implementers

Stream: implementers

Topic: TestScript capabilities

Josh Mandel (May 04 2016 at 22:32):

I'm still trying to wrap my head around what TestScript is designed to do. Can a TestScript say "run this query and ensure that all references in the returned resources are resolvable to existing resource"?

Richard Ettema (May 05 2016 at 01:48):

Yes. This is a common type of test. For example, your TestScript defines the following series of operations: create resource, search for resource. After the search operation, you define asserts; one of which is a verification that the searchset Bundle contains the just created resource.

Richard Ettema (May 05 2016 at 01:51):

There are a couple of ways you could write that assert - using the minimumId test where you reference a known external version of the resource to compare against the response, or a series of asserts that explicitly test the response for known expected values referenced via path expressions (currently xpath or jsonpath, fluentpath in the near future).

Josh Mandel (May 05 2016 at 12:07):

Thanks @Richard Ettema! Let me rephrase my question: I'm interested in TestScripts that don't/can't create new data on a server; I just want to perform consistency tests on whatever data a server exposes.

Jason Walonoski (May 05 2016 at 14:04):

So, you should be able to do a READ and then check values in the response, or compare to a known fixture (aka example)

Jason Walonoski (May 05 2016 at 14:06):

or validate against a given profile, for example

Josh Mandel (May 05 2016 at 14:06):

But for example is possible to say:

1. Get me all the MedicationOrders
2. For each, resolve the MedicationOrder.medication and make sure it exists

... with no preconception about what prescriptions and meds will actually be on the server?

Josh Mandel (May 05 2016 at 14:08):

The context here is conformance testing for projects like Argonaut or Sync for Science where the servers represent real EHR products exposing read-only interfaces on top of existing patient records. We can ask them to synthesize some specific records for testing, but ultimately we want to test systems "in the wild" to make sure references actually resolve (vs. testing that in carefully curated sample data they can resolve).

Josh Mandel (May 05 2016 at 14:09):

I think this is out of scope for TestScript, but I don't want to assume too much since I see a lot defined there and I don't understand all of it.

Jason Walonoski (May 05 2016 at 14:12):

Yeah, you can't do that with TestScript today... unless you use a ruleset

Jason Walonoski (May 05 2016 at 14:12):

Which is new and not supported outside of AEGIS

Jason Walonoski (May 05 2016 at 14:13):

All the Argonaut tests in Crucible are hand written. And I anticipate we'll do the same for S4S.

Jason Walonoski (May 05 2016 at 14:13):

We (Crucible) likely won't use TestScript for those.

Josh Mandel (May 05 2016 at 14:17):

Ah, OK. That lines up with my expectations, but I wanted to double-check. Thanks @Jason Walonoski!

Richard Ettema (May 05 2016 at 19:36):

@Josh, through the use of TestScript.variables that point to expected reference values in a retrieved resource; e.g. read or search, we can then perform a subsequent read based on the reference value to determine if the referenced resource does exist.

Richard Ettema (May 05 2016 at 19:42):

The new TestScript rule and ruleset elements are something we [AEGIS] are proposing for STU3 and we'll be trying them out at the Connectathon this weekend.

Richard Ettema (May 05 2016 at 19:44):

I'll be happy to write a TestScript that demonstrate what your example for MedicationOrder.medication defines and we can review it in Montreal.

Jason Walonoski (May 06 2016 at 11:33):

Well, he wants to do it for every single MedicationOrder, so I'm interested to see how you do that. I'd like to see that in Montreal as well.

Josh Mandel (May 06 2016 at 12:09):

Sure, it would be great to see how this works @Richard Ettema!

Richard Ettema (May 06 2016 at 14:30):

Josh, Jason is correct regarding the TestScript's current ability to not iterate through the search results. When we started working on the TestScript definition, this wasn't something we thought to initially include based on the use cases we had at the time. The proposed rule and ruleset enhancements might work but there are some security concerns I have around providing that kind of functionality in a rules engine and rule scripts. Based on your use case, I would recommend a potential enhancement to TestScript that would allow for variables to contain a collection of elements and/or values along with corresponding assert constructs to handle iterating though that collection. In any case, I will have an example TestScript ready of your use case using the current TestScript capabilities that we can review in Montreal. Looking forward to the discussion. -Thanks.

Jason Walonoski (May 07 2016 at 17:56):

Montreal Connectathon 12 Breakout recap:

• Grahame is interested in optimizing TestScript for the ImplementationGuide use-case, and making API technical conformance testing an infrastructure thing each TestScript inherits
• This would mean breaking out things like "verifying presence of headers" and very fine-grained assertions into a computable format for the base specification
• There was also discussion of replacing various assertions and rules/rulesets with FluentPath.

Did I miss anything else?

Grahame Grieve (May 07 2016 at 17:56):

I wouldn't have said this:
"Grahame is interested in confining TestScript to ImplementationGuide use-cases"

Grahame Grieve (May 07 2016 at 17:57):

I am interested in optimising it for that use case, and making API technical conformance testing an infrastructural thing you can inherit into a normal test script, and one that's maintained by only a few of us

Grahame Grieve (May 07 2016 at 17:58):

and that probably means re-organising the rules part of the spec somewhat

Jason Walonoski (May 07 2016 at 17:58):

OK, I edited my previous post to reflect some of that.

Grahame Grieve (May 07 2016 at 18:00):

I think that a summary would cover some of our scope discussions

Grahame Grieve (May 07 2016 at 18:01):

- security is out of scope

Grahame Grieve (May 07 2016 at 18:01):

- whether orchestration/choreography in scope is undecided

Grahame Grieve (May 07 2016 at 18:02):

- we're all concerned that Test Script doesn't get too complex, and most of us think that we're already there

Grahame Grieve (May 07 2016 at 18:02):

- Josh's specific need is specifically out of current scope

Grahame Grieve (May 07 2016 at 18:02):

- but we do want to able have interoperable test specifications

Jason Walonoski (May 07 2016 at 18:10):

Good adds.

Jason Walonoski (May 10 2016 at 19:12):

So, I've been thinking about the concept that was raised during the TestScript breakout on Saturday regarding pulling some assertions outside of TestScript.

What if we encoded the two tables here (http://hl7.org/fhir/2016May/http.html#summary) as structured XML/JSON -- perhaps with some more metadata in there -- as part of the build. An execution engine could then use that template for validating all client/server RESTful interactions. We'd be able to omit those from TestScripts since they would be implicitly verified, and the TestScript can focus on logic associated with the business rules of the test itself.

Thoughts?

Grahame Grieve (May 10 2016 at 19:38):

I think those are good thoughts. Do you want to make the json, or do you want me to make it?

Richard Ettema (May 10 2016 at 20:56):

So is the intent that if I defined the following operation that the 'base' asserts from these tables would already be defined and executed by the test engine?

<operation>
    <type>
            <system value="http://hl7.org/fhir/testscript-operation-codes"/>
            <code value="create"/>
    </type>
    <resource value="Patient"/>
    <accept value="xml"/>
    <contentType value="xml"/>
    <sourceId value="patient-create"/>
</operation>

And, therefore, eliminate the need for explict asserts like this?

<assert>
    <description value="Confirm that the returned HTTP status is 200(OK)."/>
    <response value="okay"/>
</assert>

Jason Walonoski (May 10 2016 at 21:34):

@Grahame Grieve I'm on vacation the next few days, but I can create it after that. @Richard Ettema Yes.

Grahame Grieve (May 11 2016 at 10:29):

yes. After all, this assert is always enforceable whether specified in the test script or not

Douglas Harley (May 13 2016 at 18:22):

This is an extremely interesting discussion, and it dovetails nicely with the work that I have been doing around ONC DAF profile testing. Basically, I am seeking to create automated test coverage for as much of the DAF profile functionality as possible, while not adding to problems of tool proliferation/duplication, and TestScripts seem the ideal solution. During my analysis of the existing coverage in both Crucible-FHIR and Aegis' Touchstone, and just looking at the FHIR TestScript resource definition itself, I have noticed some things that could be improved, and wonder what you all think: <more to come, didn't realize enter would post>

Richard Ettema (May 13 2016 at 18:24):

@Douglas Harley Yes. Any suggestions and recommendations are very welcome. I do need to re-visit the TestScripts currently deployed in Touchstone and your input would definitely help.

Douglas Harley (May 13 2016 at 18:27):

1) Coming primarily from the Java world, and a deep appreciation for the junit framework, it seems that it would be useful for purposes of aggregation that TestScript could optionally refer to children TestScripts. Additionally, it would be beneficial if there were downward-visible variable scoping, such that a child TestScript would have visibility of variables defined "above it" in the hierarchy. Aggregating TestScripts would greatly ease the burden of TestScript organization for test clients, and implementors too.

Douglas Harley (May 13 2016 at 18:30):

2) Currently, TestScript is really a great concept without a home, since it isn't referred-to by any other resources. It seems that it would be beneficial for the community to be able to define "official" TestScripts, and associated fixture data along with other profile resources. So, a profile would have an implementation guide, and conformance statement(s), and TestScripts/fixtures, which can be used to verify correct implementation of the putative conformance statement(s).

Douglas Harley (May 13 2016 at 18:32):

I had initially purposed within our team that the Conformance resource might be a good home for TestScript(s), but others on my team thought not, and I can see their point. Still, we all think it would be useful if there were some way to associate official DAF profile TestScripts and data with the DAF profile.

Douglas Harley (May 13 2016 at 18:34):

3) The current TestScript operation codes (https://www.hl7.org/fhir/valueset-testscript-operation-codes.html) while defining the r̶e̶a̶d̶ search operation, do not allow for the specification of POST HTTP method, and that is an explicit requirement of FHIR, so I don't see how to test it, and it seems that the value set needs to expanded.

Douglas Harley (May 13 2016 at 18:36):

<edit> adding: @Richard Ettema</edit> Hi Richard, thanks for your great work with the existing DAF TestScripts! They are excellent, and cover already the most important aspects of DAF functionality. :)

Richard Ettema (May 13 2016 at 18:44):

@Douglas Harley You're very welcome. I know that there have been some recent updates regarding supported search pardigms that would remove the compartment search from the Argonaut requirements. I'll be updating the DAF TestScripts at some point based on this.

Douglas Harley (May 13 2016 at 18:48):

That should be an Argonaut-only change I think...as far as I know there is no move to remove the Patient compartment search requirements from DAF Patient. However, I am relatively new to the project, so I will discuss with our team.

Lloyd McKenzie (May 13 2016 at 18:51):

Visibility of variables defined elsewhere is a bit challenging from a methodology perspective. We try hard to ensure that each resource instance can be interpreted/used on its own.

Douglas Harley (May 13 2016 at 18:52):

I think that the ability to abstract-out a lot of the lower-level validation out of TestScript is a great idea...indeed, it seems to suggest a need for a TestScript engine RI. Without a single reference implementation, it is quite possible to have a single TestScript that operates differently, and gives different results, when executed within 2 different engines.

Douglas Harley (May 13 2016 at 18:55):

@Lloyd McKenzie That makes sense...to clarify, I think that is the least-useful aspect of a potential self-referential TestScript, the aggregation and organizational abilities are more interesting.

Douglas Harley (May 13 2016 at 18:58):

all the above just my $0.02 from when I was thinking about this stuff...please feel free to shoot holes in all of it :)

Richard Ettema (May 13 2016 at 18:58):

Abstracting the lower-level validation is something we (FHIR Core, AEGIS and MITRE) discussed in Montreal during the FHIR Connectathon and some of the WGM sessions. The idea is that a FHIR TestScript test engine would always execute the base assertions for a given operation as defined on the RESTful API page (bottom two tables) http://hl7-fhir.github.io/http.html#summary

Douglas Harley (May 13 2016 at 19:04):

yes, i saw that, and it is a great idea. (unfortunately, the DAF team was not able to travel to Montreal for the connect-a-thon.)

Richard Ettema (May 13 2016 at 19:07):

No worries. This thread will be the place to collaborate and share ideas.

Richard Ettema (May 13 2016 at 19:08):

Regarding your 3rd point, each of the operations define how they are executed; i.e. read via GET, create via POST, update via PUT and so on. A TestScript test engine will know which HTTP operation to perform for each FHIR operation.

Douglas Harley (May 13 2016 at 19:10):

...but it ultimately depends upon the reach/depth/correctness of the implementation, which can be assumed to vary between engines. in an ideal world, there would be a single implementation of TestScript engine, that way the only differences between TestScript results for 2 different FHIR servers would be completely due to differences in server implementations, but that cannot be the case with multiple engine implementations.

Douglas Harley (May 13 2016 at 19:11):

argh!...i accidentally wrote "read" when i meant "search"

Douglas Harley (May 13 2016 at 19:12):

the search interaction can be either GET, or POST

Douglas Harley (May 13 2016 at 19:13):

https://www.hl7.org/fhir/http.html#search "...in addition to the get based search method above, servers that support search SHALL also support a POST based search..."

Richard Ettema (May 13 2016 at 19:35):

Currently, we control this in the the Touchstone test engine by whether the source is set in the operation where if source points to a fixture to be sent we issue a POST; if the source is empty/not defined, then a GET.

Douglas Harley (May 13 2016 at 19:40):

nice, that is a good workaround. i wasn't able to look inside Touchstone code to see how things were implemented...again though, this behavior is not specified in the TestScript standard, and other implementations (e.g., Crucible-FHIR) are not like this.

Douglas Harley (May 13 2016 at 19:41):

...at least not that i gathered, i am a relative ruby newby. ;)

Richard Ettema (May 13 2016 at 19:44):

Yes, this behavior needs to be defined and documented for TestScript. The current TestScript definition page also needs to be updated based on some recent modifications. I am also going to be authoring a new "testing" page in the spec to offload a good chunk of the TestScript page text to a new home.

Douglas Harley (May 13 2016 at 19:45):

cool...that sounds like a good idea too

Douglas Harley (May 14 2016 at 05:14):

@Richard Ettema I spoke with the DAF technical lead, and you are completely correct: DAF is following Argonaut, and we are also removing the compartment search requirements for DAF Patient.

Michel Rutten (May 14 2016 at 13:09):

@Douglas Harley interesting discussion. I've talked to Richard at the WGM about some of this, esp. how the community could provide some form of "official" or vetted Testscripts. One of the ideas that came up is to publish approved TestScript packages on the FHIR registry at simplifier.net. This could also enable links between official IG's/profiles on the registry and associated TestScripts. If the community finds this helpful, we could try and implement some basic TestScript support in the simplifier platform. Looking forward to feedback on this.

Grahame Grieve (May 14 2016 at 23:09):

So there's a daf implementation guide resource - this is where a day test script should be referenced

Douglas Harley (May 16 2016 at 02:22):

@Grahame Grieve thanks for the clarification.

Douglas Harley (May 16 2016 at 02:28):

@Michel Rutten Thanks for the suggestions, that all sounds intriguing...but I must confess that I am fairly new to the ONC DAF project, and FHIR itself, and so am not familiar with simplifier.net and its role in the FHIR ecosystem. I have passed-along this conversation to our technical lead to get his thoughts, and I will be able to respond further after we speak. cheers :)

Douglas Harley (May 16 2016 at 15:48):

@Michel Rutten DAF profile TestScripts, along with all other profile resources, are free for anyone to use, and will be made available through the usual FHIR release mechanisms. Right now though, I am just starting the process of seeing what's already out there in terms of DAF testing, what needs to get done, and ultimately where things are going to live. With regards to a "reference implementation" of a TestScript engine, I was speaking only conceptually. As a software developer who fully-embraces automated testing, I see the lack of a single common community TestScript execution engine as a weakness in the ultimate practicality and usefulness of TestScripts, and was just stating my technical opinion explicitly.

Grahame Grieve (May 16 2016 at 17:34):

but having more than one is a different kind of strength

Douglas Harley (May 17 2016 at 13:38):

Interesting.... I came up with a few potential strengths: 1) more people working on implementations means "more eyes", and so the spec can be worked-over iteratively, and innovation might occur faster; 2) potential issues discovered in one implementation might be circumvented via use of a different implementation; 3) having multiple implementations lends a kind of tacit approval of the specification, in that "the more implementers, the better the spec must be". All certainly well-and-good at this stage, but as the platform matures, uniformity of correctness, breadth-and-depth of spec coverage, and stability seems like they would become of overriding importance for a TestScript engine.

Douglas Harley (May 17 2016 at 13:53):

@Richard Ettema any thoughts on my proposal to add reference to child TestScripts to TestScript? it seems that it would be much easier for test tool implementers to just "point-to" 2 DAF-related tests (i.e., DAF query requester, and DAF query responder), and get all the TestScripts referred-to therein automatically via a resolution process. During my process I might create for example 20 different TestScripts in total, and organizing them without this proposed self-referential enhancement seems like a real headache, and a potential source of errors/ommissions

Richard Ettema (May 17 2016 at 18:13):

@Douglas Harley I do like the concept in general; however, I have to agree with Lloyd's position here where he stated earlier in this thread "We try hard to ensure that each resource instance can be interpreted/used on its own". The dependencies this would introduce into the TestScript definition goes against this philosophy. I would also be concerned of potential circular references.

Richard Ettema (May 17 2016 at 18:13):

In Touchstone we currently provide a grouping mechanism using an appropriate folder structure. This isn't part of the specification but still a useful organizational mechanism for our environment.

Douglas Harley (May 17 2016 at 18:20):

Fair enough...if it isn't an organizational problem for you guys, then it isn't a problem. :)

Jason Walonoski (May 24 2016 at 18:38):

@Grahame Grieve @Richard Ettema I've made an attempt to move the HTTP Table (http://hl7.org/fhir/2016May/http.html#summary) into JSON. You can see the proposal here:

• https://github.com/fhir-crucible/fhir_client/blob/stu3/lib/fhir_api_validation.json

Example code that uses these JSON rules to validate the structure of the HTTP request/responses is here:

• https://github.com/fhir-crucible/fhir_client/commit/31a434eb57354f241879642823da8cbbc65a1fde#diff-68edf999a120d604d2b47800ed1a3377

Let me know what you think. The JSON (or something similar) could be used to generate the HTTP Table in the specification HTML, and could be used to validate the API calls, replacing a boat-load of TestScript assertions.

Grahame Grieve (May 24 2016 at 21:31):

that's the kind of thing I was thinking of, yes. We could generate conformance statements from that as well, right?

Grahame Grieve (May 24 2016 at 21:31):

I do think we need support for 5xx as a code, and generally, 3xx codes are always allowed...

Grahame Grieve (May 24 2016 at 21:32):

could make some more detailed comment, but I think it would be good to build that into the spec itself and generate conformance statements from it,

Richard Ettema (May 24 2016 at 23:18):

@Jason Walonoski I like the initial structure. I'll have to wait until later this week to do a more detailed review. Thanks.

Jason Walonoski (May 25 2016 at 12:16):

@Grahame Grieve My thinking was that 4XX errors were valid responses... but a 5XX errors was something really bad (internal server error) so the response should not validate. As for 3XX, a redirect, the client should follow it and eventually get a different status code and server response which can be validated, therefore it didn't need to be included.

As for generating conformance statements... I'm not sure where it fits to be honest.

Do you want me to move it into the svn build, or submit a change request, or take no action?

@Richard Ettema OK, thanks.

Grahame Grieve (May 25 2016 at 21:30):

a number of the public servers have trouble separating client and server errors. I don't how to do that is very often you shouldn't have asked for that, but differentiating this is above the level of the code.

Grahame Grieve (May 25 2016 at 21:30):

I think we shouldn't require a server to get this right in conformance testing.

Grahame Grieve (May 25 2016 at 21:31):

here's an example: if a client sends me a json with duplicate properties, my json parser throws an exception.

Grahame Grieve (May 25 2016 at 21:31):

it throws the same exception if it's out of resources

Grahame Grieve (May 25 2016 at 21:32):

as for 3xx - A production client doesn't always follow that. and it's not clear that a conformance testing client will either. 304 not modified is a valid response under some circumstances.

Jason Walonoski (May 26 2016 at 12:12):

Well, I follow you in how it can be used to identify both client and server errors in conformance testing -- just not how it would be used in generating conformance statements... unless it is a statement like "your client (or server) doesn't conform to history interactions properly."

Jason Walonoski (May 26 2016 at 12:12):

At any rate, I'm going to use this JSON in the Crucible FHIR client. Let me know if you'd like me to contribute it to the SVN repo.

Richard Ettema (May 27 2016 at 15:59):

@Grahame Grieve, @Jason Walonoski Here are my current thoughts regarding the computable structured format of the http.html summary tables.

Richard Ettema (May 27 2016 at 15:59):

I agree that having this structure is good as it provides a way to generate and maintain the http.html summary tables.

Richard Ettema (May 27 2016 at 15:59):

I am with Jason regarding not quite understanding how this structure would be used to generate conformance statements. I do see it working in relation with a FHIR client's or server's conformance statement as it applies to testing that client and/or server.

Richard Ettema (May 27 2016 at 15:59):

While I do agree that this structure could be used by a test engine as a set of pre-defined test conditions, I have some questions and concerns:

Richard Ettema (May 27 2016 at 16:00):

- The http.html summary tables contain a value 'N/A' for many entries. Is the 'N/A' to be interpreted as a hard MUST not be present, or as an optional SHOULD not be present?

Richard Ettema (May 27 2016 at 16:00):

- There are other aspects of the API that are not represented in the summary tables. A few examples:

Richard Ettema (May 27 2016 at 16:00):

-- Under the Support for Versions, http://hl7-fhir.github.io/http.html#versioning, section is the statement "Server should always return the default timezone for date searches in the HTTP Response headers using the Date header." The HTTP Date header is not shown in the response summary table.

Richard Ettema (May 27 2016 at 16:00):

-- Under the Content Types and encoding, http://hl7-fhir.github.io/http.html#mime-type, section is the description of the _format parameter that every operation/interaction SHOULD support.

Richard Ettema (May 27 2016 at 16:00):

-- Under the Content Types and encoding, http://hl7-fhir.github.io/http.html#mime-type, section is the statement "Requests MAY also specify this charset parameter in the Accept header and/or use the Accept-Charset header." The HTTP Accept-Charset header is not shown in the request summary table.

Richard Ettema (May 27 2016 at 16:01):

- The structure derived from the http.html summary tables lack many features that should be explicitly defined. These features are what we as a Testing Group (FHIR Core Team, MITRE, AEGIS and a few others) worked through last summer when defining and refining the TestScript definition; i.e.:
-- Descriptions for interactions/operations and assertions. These inform the tester of what the expected behavior of a test execution should be.
-- Ability to group operations/assertions into specific tests.
-- Ability to validate against profiles.
-- Ability to check conformance of a test system for successful execution via metadata elements.
-- Sophisticated assertions based on XPath/JasonPath (FluentPath/FHIRPath in the near future).
-- Explicit checks for request/response header values.
-- Setup procedures to prepare a system under test for repeatable test execution outcomes.
-- Teardown procedures to return a system under test to its state prior to the test execution.
-- Constructs to allow client-side/peer-to-peer testing including support for multiple origins.
-- Constructs to support multiple destinations for interactions.

Richard Ettema (May 27 2016 at 16:01):

I understand the desire to simplify what I sense is the perceived complexity of the TestScript resource. However, based on my experience, implementing a robust testing platform and environment is inherently complex. I'm sure you would agree that regardless of the design and implementation details, building a testing tool is not a simple task. I'm seeing the direct use of this structure in a test engine as just a shuffling of the deck chairs (as Grahame likes to put it).

Richard Ettema (May 27 2016 at 16:01):

Getting back to the proposed structure, as I stated above I agree having this is good. Building the summary tables should be generated via a computable format. However, if we were to continue down the path of using this structure as input to a test engine, I feel that it will out of necessity need to grow and expand to support the explicit testing definitions that a test engine requires. As we add the missing features outlined above to this structure, it will increasingly look more like what we already have in the TestScript resource. The Testing Group put a lot of thought into the design of TestScript for DSTU2 and the current specification. Rather than add another format for a test engine to ingest, we could use this new computable structure format in a slightly different way. This structure could be used by a TestScript generator to produce the TestScript resources that define the approved "FHIR API <version> Test Definitions".

Richard Ettema (May 27 2016 at 16:01):

As an example of what this might look like, I have taken the read operation/interaction from the existing summary tables and built a corresponding FHIR API STU3 Test Definition (TestScript) for a Patient read on our AEGIS FHIR STU3 server (GET http://wildfhir.aegis.net/fhir/TestScript/fhir-api-stu3-patient-read-xml).

Jason Walonoski (May 27 2016 at 18:42):

@Richard Ettema the proposed JSON was not to replace TestScript at all, so things like "the ability to validate against profiles" doesn't apply here -- that will remain a function of the TestScript. The proposed JSON was just to make the basic assertions that validate the structure of the requests and responses, and eliminates the need to reassert every header and response code for each operation in every TestScript. The TestScript will have to continue to assert things like all of your bullet points above. In your example, using this "request/response schema" (using the word schema as an analogy) you can eliminate lines 32-79, 86-92, 99-120 from your example TestScript with the exact same functionality.

Grahame Grieve (May 27 2016 at 23:02):

I would have thought that at least some of the things Richard listed would be candidates for addition to the json

Mario Hyland (May 28 2016 at 18:17):

@Jason Walonoski I believe what @Richard Ettema was offering adding something new which parallels what Test Script already does may in fact become more confusing. While our objective is to make it simply, if we are not careful we could make it more confusing for the community at large. @Richard Ettema identified alternative paths to automating and simplifying the generation of Test Scripts. I think if we can support a broader range of inputs - such as IG, Profile, Resource, Model such as MDHT, and others - each providing a way to input into some mechanism which generates or outputs Test Script Resource (and does so quickly and efficiently) - we can keep the Test Script Resource as the Authoritative Approach to test FHIR Implementations. I agree with @Grahame Grieve (if I am reading him/it correctly) we can add to JSON Tests. I am keen to see what happens with the FluentPath test implementations.

Grahame Grieve (May 29 2016 at 09:57):

well, there's already a set of unit tests for fluent path. I guess I could extract them to a more structured form

Richard Ettema (May 30 2016 at 15:59):

@Jason Walonoski Yes, I know the proposed JSON is not a replacement for TestScript but is meant to be a way to simplify a TestScript. The intent as I understand it (please correct me if I mis-state) is that the proposed JSON represents pre-defined asserts that will always be used by a FHIR test engine. As you stated, this would drastically reduce the number of explict asserts following any given test operation. This sounds like a great idea at first and I agree that all of the rich features of TestScript do not need to be available for these base API asserts. Though, as I continue to think about this, I still have concerns and questions.

Richard Ettema (May 30 2016 at 15:59):

Even though the proposed JSON implicitly defines the base API asserts, the test engine will still need to explicitly implement those asserts. So, there still needs to be explicit definitions of the base API asserts for a FHIR test engine to correctly execute.

Richard Ettema (May 30 2016 at 15:59):

If a FHIR test engine were required to implement and always execute these base API asserts; i.e. make the test engine smarter, the test engine is now constrained. For instance, how does the test engine deal with testing situations where negative testing (error and exception conditions) must be verified? In other words, a test can be designed to force a system under test to respond to bad requests and data where the test will succeed when the system under test responds with an appropriate and expected error? The test engine would then need to 'know' not to apply the base API success asserts.

Richard Ettema (May 30 2016 at 16:00):

@Grahame Grieve @Jason Walonoski I believe we need to perform a full examination of all the implementation pages - http.html, search.html, operations.html, messaging.html and services.html - to produce a complete and accurate structure that represents what we are calling the "Base FHIR API". This structure would then be initially used to generate the summary tables. We can then take a better look at its potential use by and within a test engine.

Jason Walonoski (May 31 2016 at 12:36):

@Grahame Grieve @Richard Ettema I agree that clarifying the N/A makes sense (Bullet 1), and adding support to the JSON for Date Header, _format parameter, Accept, Accept-Charset (Bullet 2 and sub-bullets) makes sense. If we add the features of Bullet 3 (and sub-bullets), I suspect we'd just end up with another set of TestScripts. That is fine and where we are today, but the idea was to explore a way to make some basic things in every TestScript implicit --- unless the TestScript wanted to explicitly execute things like check response code, or check a header -- for negative testing or otherwise. I think the more complicated actions in Bullet 3 should remain in TestScript. At any rate, I think (?) that we all agree that we want a computable "Base FHIR API" structure as Richard stated, and I submit that JSON example as a beginning.

Last updated: Apr 12 2022 at 19:14 UTC