Stream: inferno
Topic: Inferno vs Crucible
Ryan Harrison (Jun 12 2020 at 02:19):
@John Moehrke The difference between Crucible and Inferno also wasn't clear to me, so I took a deep-dive on the repos.
My TL;DR is use Inferno and hope that someone picks-up maintenance for Crucible before it rots too bad.
Use Aegis Touchstone if you want to use FHIR TestScripts. Inferno, and as best as I can tell Crucible, both rely on hard-coded ruby for their tests. Touchstone has all the vanilla (STU3, R4, 4.0.1) and Connectathons in FHIR TestScripts. As an indication for non-vanilla implementation guide (IG) support, I looked for CARIN CPDS and CMS Blue Button 2.0. I didn't see either. Possible non-vanilla IGs are paid. I haven't found another test service that uses FHIR TestScripts. If such as thing exists, I'd love to know about it.
Use Inferno if you want easy semi-automated testing for g(10) compliance (§170.315(g)(10) Standardized API for patient and population services). Indeed, the Inferno tests are identical to the ONC-provided test plans (e.g. https://www.healthit.gov/test-method/standardized-api-patient-and-population-services#test_procedure for (g)(10)). Inferno also uses hard-coded ruby tests, organised hierarchically into modules (_module.yml), sequences (_sequence.rb, e.g. https://github.com/onc-healthit/inferno/blob/aef309b33045cabd4a4096133068143becceb7ef/lib/modules/smart/smart_discovery_sequence.rb), etc. The good news is this organisation allows for inheritance and less duplication. The bad news is it seems like A LOT of work to add Inferno testing for a new IG. There's no plugin system, just clone and hack away. I believe ONCs hope is that Inferno will i) streamline ONC ATL (Authorized Testing Laboratories) testing and ii) encourage self-compliance even in non-certified FHIR servers.
The case for Crucible is less clear. It does not look actively maintained, which is unfortunate. The scope of Crucible is larger than Inferno;
-
Crucible includes radial conformance diagrams (love em')
20200611_Crucible_UHN_HAPIServer.png -
Tracking of conformance over time (very useful)
- Bundle leaderboard (meh)
None of these features exist in Inferno, and the latter two are definitely out of scope.
Under the hood, Crucible (Rails app) relies on repo:plan_executor to do the testing. While some FHIR TestScripts exported from Touchstone are in the repo (https://github.com/fhir-crucible/plan_executor/blob/master/lib/tests/testscripts/scripts) -- as best as I can tell -- the tests actually run by Crucible are hard-coded ruby (e.g. X010 in https://github.com/fhir-crucible/plan_executor/blob/4bb62b6cf7d515340ca085b53ec1bfa724237769/lib/tests/suites/resource_test.rb#L89).
My best guess at a history.
2020-06-11-Brief-history-of-FHIR-testing-tools.png
@Robert Scanlon Keep up the great work with Inferno and please correct if I'm in err.
John Moehrke (Jun 12 2020 at 14:20):
@Reece Adamson ?
John Snyder (Jun 16 2020 at 14:20):
@Ryan Harrison an excellent history of the testing tools. ONC is sponsoring Inferno in two "editions": Inferno Program Edition provides certification testing for the Standardized API for Patient and Population Services in ONC’s 21st Century Cures rule (known as the (g)(10) criteria). There is also a Community Edition of Inferno with generalized, customizable capabilities for the validation of specialized implementation guides and other FHIR-enabled data exchanges. If you are interested in certifying your health IT and API technology to ONC's 2015 Edition Cures Update API criteria--use the Program Edition of Inferno. All others will want to focus on the Community Edition.
You touched on the missing link: having to hand-stitch tests in Ruby. We recognize that limitation and have been working on automation. @Reece Adamson can elaborate more on the IG to test case automation.
Robert Scanlon (Jun 16 2020 at 20:13):
@Ryan Harrison -- thanks for writing up your research on the history of these projects. You are largely on the mark. The Crucible / Inferno relationship has always been a point of confusion, and our hope has always been to either sufficiently differentiate the two tools and maintain both, or unify them under the Inferno name. Inferno began principally as a 'SMART on FHIR' + 'Argonaut Query IG' testing tool (and now what is known as the g10 criteria), and at the time that left room for Crucible to exist as a more generic FHIR testing service. But over time Inferno has been made more generic (as @John Snyder pointed out, under the 'Community Edition' banner), and therefore more similar to Crucible, as it looks to address testing needs of other Implementation Guides. So it seems more likely that we will retire Crucible as a standalone test platform, and pull in the most useful pieces into Inferno. I'd love to modernize the conformance visualization and pull that in, for example.
Robert Scanlon (Jun 16 2020 at 20:16):
One thing to note is that Crucible really is a suite of Ruby FHIR tools and libraries, many of which are still being very actively maintained. fhir_models and fhir_client are Ruby FHIR tools designed for broad use, beyond testing. And Inferno uses both of them under the hood (as well as a number of other projects out there).
Robert Scanlon (Jun 16 2020 at 20:45):
Also, Crucible's plan_executor has some support for TestScript execution, though it hasn't been maintained since FHIR STU3. We chose to not use TestScripts in Inferno in part because many of the requirements we needed to test (most notably SMART on FHIR) could not be represented in TestScripts. If there is strong demand from the IG authoring community to author tests in that manner then we could pull in that component from Crucible.
Robert Scanlon (Jun 16 2020 at 20:54):
I'm not aware of other public TestScript engines, besides Touchstone -- there has been talk on #implementers to make some upgrades to TestScripts, so I think there is at least one other TestScript engine out there. And yes, Touchstone is a good choice if you want to use TestScripts, but it also has a lot of other things not supported by Crucible/Inferno (client testing comes to mind).
Ryan Harrison (Jun 16 2020 at 21:44):
Thanks for the kind responses, @John Snyder and @Robert Scanlon.
Crucible's plan_executor has some support for TestScript execution, though it hasn't been maintained since FHIR STU3.
I'm less concerned about the use of TestScript as opposed to Ruby.
Is there a FHIR R4 testing tool that uses TestScript, other than Touchstone?
2-cents: Automating profile/extension testing of arbitrary IGs
Automation to produce an Inferno test script from an IG would be a game changer. I can see IGs, reference servers and production applications coming online. The missing link is lowering the bar for testing. In part so IG developers can use TDD. In part so client application developers can contract-test without relying on a reference server. In part to avoid drift between the IG and test suite (which is why the derivation of the test suite from the IG is so appealing).
I imagine an end-state similar to that of API contracts, where a "single source of truth" (Swagger/OAS spec; here the IG) serves as the documentation and contract test. I suspect that the SHALL/MAY language won't be readily convertible to code, so punt. Validating responses against the IG profiles and extensions is the low hanging fruit, and is a useful 50% solution in and of itself. As a first pass, reuse the SMART discovery sequence to enumerate the supported resources, call each one, and validate against the IG profiles/extension definitions.
Do you know if anyone has attempted?
2-cents: Separation of concerns between Inferno and Crucible
I think the distinction between a one-off contract test (Inferno) and tracking conformance over time (Crucible) is a good separation of concerns. So, while plan_executorcan be retired in favour of the class-based hierarchical organisation of Inferno, I think it would be a unwise to extend the Inferno codebase with the CI-aspects of Crucible. Rather, swap-out plan_executor for Inferno ( more specifically the non-GUI library portion of Inferno, presumably percolating in PR385.
Robert Scanlon (Jun 17 2020 at 01:30):
The majority of our US Core tests are generated from the IG's machine-readable metadata. That IG very helpfully tags SHALL/SHOULD/MAY conformance expectations in the capability statements (and search parameters). That way we know which operations and search parameters combos are required by the server under test, and can use it to essentially crawl the server using only the search parameters combinations that must be supported. See https://www.hl7.org/fhir/us/core/CapabilityStatement-us-core-server.json.html which defines the SHALLs, and then our generator that uses that (along with the profiles, search params, and other IG content) here: https://github.com/onc-healthit/inferno-program/tree/development/generator/uscore
Robert Scanlon (Jun 17 2020 at 01:38):
We'd like to make that more generic, so that it can broadly generate tests that expect to be able to perform a RESTful crawl of a server based on which resources / search params / etc are required. And if 'RESTful crawl' is one kind of test pattern, create other patterns based on common interactions (for example, if writes and reads are required, it is straightforward to do round-trip tests, etc).
Robert Scanlon (Jun 17 2020 at 01:45):
Inferno took the 'code generation' route, with code be written to ruby files https://github.com/onc-healthit/inferno-program/tree/development/lib/modules/uscore_v3.1.0. Crucible does something similar for vanilla FHIR, but using metaprogramming instead, and generates resource write/read tests based on core FHIR structuredefs essentially. It is a little hard to read, which defeats the purpose of the nice testing DSL we have I think. But there are other benefits to that approach. See https://github.com/fhir-crucible/plan_executor/blob/master/lib/tests/suites/resource_test.rb
Last updated: Apr 12 2022 at 19:14 UTC
