Stream: connectathon mgmt
Topic: Testing
Grahame Grieve (Feb 21 2018 at 02:40):
At least one track - terminology - is starting to get seriously into the testing phase. It's not clear how far we want to go with testing (at connectathons or otherwise) and/or how far we want to work through that with existing testing providers.
Partly this is a product question that I need to talk to the board about (and that's on my agenda). But we're interested in opinions, either about testing more widely, or about testing at a connectathon.
- Should we (as someone suggested last in the post-connectathon survey), have dedicated testers who review and approve inter-system tests (that's the IHE model)?
- Should we start trying to formally record testing outcomes?
- should we seek some kind of hand-over with IHE in this regard?
- should we start a test-the-tester track?
Lloyd McKenzie (Feb 21 2018 at 04:25):
There's also the question of what comes out of the testing. Do you get a checkmark on a wki page somewhere in the bowels of the connectathon documentation, do you get some sort of formal certification that lets you use a special mark in your marketing material, or something else?
John Moehrke (Feb 21 2018 at 13:43):
I clearly would prefer cooperation with IHE, as those that ultimately use our apps, servers, and standards are the most important. I think it is important to mature the FHIR Connectathon. I have captured details and deeper thoughts on my blog article https://healthcaresecprivacy.blogspot.com/2018/02/maturing-fhir-connectathon-without.html
Grahame Grieve (Feb 21 2018 at 19:32):
thanks. I think I agree with the thrust of that, but I'll have to read it again when I have more time
Peter Jordan (Feb 21 2018 at 21:24):
Testing - both formal and informal - is one of the major reasons that I continue to attend FHIR Connectathons. The formal component, I believe, is critical both in terms of demonstrating that my Terminology Server complies with the relevant parts of the specification, and helping to provide evidence of the maturity of the associated resources and operations.
To achieve those goals, it's necessary for that testing to be performed using a recognised Tool - using transparent and verifiable FHIR TestScript Resources, developed by the FHIR Community - with the results available for all (including product purchasers) to view. Thankfully, courtesy of David Hay's excellent app, we now have the ability to link Connectathon Management and testing tools.
However, FHIR testing is not just about Connectathons, or the large amount of work that goes into preparing for them, it requires continuous testing - test-driven development. IMHO, the fact that FHIR facilitates this - via the use of independent, on-line, tooling - is one of the strongest value propositions it offers to implementers...and, if anything, my views on this have actually strengthened since this blog post written after the Jan 2016 WGM in Orlando... https://fhirblog.com/2016/02/15/test-driven-development-with-fhir/
Kevin Shekleton (Feb 21 2018 at 22:13):
I think incorporating a stronger emphasis on testing is natural for aspects of the specification that are more mature.
You may want to consider branding this separately from the Connectathon. Today, a FHIR Connectathon has a very informal, low pressure atmosphere and culture which is something we should strive to preserve. We wouldn't want to confuse or muddle that with portion of the Connectathon that is at odds with that. Basically, just have that formal testing work be under a different name (even though it can be held at the same time/location as the Connectathon).
I think it is also important to allow tracks to not follow a more rigorous testing model (eg, immature or developing resources or projects).
As an aside, when I say "testing" I'm referring to the concept of testing -- not the FHIR TestScript or TestReport resources. This is because I think testing doesn't have to be only things done with the FHIR Test* resources. For instance, look at the great test suite the Sync4Science project created.
Abbie Watson (Mar 13 2018 at 02:07):
My major concern would be the amount of time involved in a Connectathon vs a Validationthon. In the one, you're basically calling a REST endpoint of a pre-existing server and saying HTTP.get('http://sandbox.org/fhir/STU3/Patient/12345') and you get some data, and voila! You're 'connected'. A couple of lines of code, and you're good to go hack a demo or something.
In the latter... well, you're implementing to spec and doing coding-fu on the fly. You had better have a mobile development environment, your build and deployment pipeline setup, no hiccups with the network or internet connection, API documentation for your entire stack and language on hand, etc. It's certainly doable. But there's lots of things that can go wrong at any time. And squeezing it into a weekend connectathon session? I think there needs to be really clear expectations that people will be lucky to get ONE resource under QA in that timeframe. Maybe. If they're prepared and ready to focus 12 to 18 hours of debugging.
When I went to the Chicago connectathon in April '17, there was no way I was able to get anywhere close to getting an entire resource under QA in that weekend. I learned which tools I needed. But doing the actual implementation? That was 20 to 30 hours of work that I had to do at home for the first one. Now I have it templated, and can do them fairly quickly. In only 8 or 12 hours.
That being said, I'd love to see Touchstone get more official adoption. It's the bee's knee. (Crucible too, although it's not as rigorous.)
Abbie Watson (Mar 13 2018 at 02:16):
My dream testing workflow:
- HL7 publishes a FHIR spec.
- Aegis Touchstone implements FHIR spec as a TestSuite.
- Vendor implements a FHIR endpoint, programming it against Touchstone until it passes conformance.
- Touchstone generates a TestScript results file and digital signs it and sends it back to vendor.
- Vendor feeds results file to Crucible to visualize and record in community rankings.
- Vendor submits results file to IHE, ONC, and whomever else is certifying systems.
$0.02
Last updated: Apr 12 2022 at 19:14 UTC
