Stream: ihe
Topic: IHE Conformity Assessment
Grahame Grieve (Jun 14 2017 at 22:23):
Today I discovered that not all IHE profiles are supported for conformity assessment. I had thought that was the whole point, so my knowledge is one step less inadequate than yesterday
Grahame Grieve (Jun 14 2017 at 22:23):
how does IHE decide which profiles are subject to this?
Grahame Grieve (Jun 14 2017 at 22:23):
would IHE consider doing this for profiles that Hl7 develops?
Elliot Silver (Jun 15 2017 at 03:19):
This is most definitely not my area of expertise, so take this all with a grain of salt.
As I understand it, IHE conformity assessment targets profiles that users (and funding agencies/governments) want a higher level of assurance than an IHE connectathon can provide. IHE conformity assessment targets shipping products, while IHE connectathon allows works-in-progress, prototypes, etc.
I expect that the included profiles are widely implemented , widely included in government, etc. procurments, and have caused problems before in terms of implementation quality.
I don't know if IHE will continue to expand the list of profiles, or keep it focused on a core set.
Hopefully, someone more connected to the assessments can provide a better response.
Elliot Silver (Jun 15 2017 at 03:22):
As for IHE doing conformity assessment for HL7 profiles, I'd expect that would have to be decision of the IHE board of directors, and I have no idea what way they'd go.
René Spronk (Jun 15 2017 at 07:04):
If one has a look at https://conformity.ihe.net/summary-reports then one can see that uptake has been relatively low.
It was hoped this would generate a lot of revenue for IHE - but that hasn't turned out to be the case. This may just fade away silently and slowly (that's what I've been hearing thru the grapevine), or if suddenly some big government starts to require it the process may flare up again.
Grahame Grieve (Jun 15 2017 at 09:16):
so what do IHE connectathons offer beyond our ones. I was under the impression that they did certification? or this some extra certification step?
Alexander Berler (Jun 15 2017 at 12:34):
IHE Conformity assessment is thorough testing of specifications based on ISO 17025. IHE CAS is a machine to machine testing session monitored by an ISQTB certified professional. it delivers a certified test report that ANY certification authority can accept and create certification programs under ISO17065. this could be a government, an association, anybody that has an ISO17065 certificate. this is why IHE CAS is driven for products or prototype that will be shipped as products in a 6 month period of time.
IHE Connectathons is a peer to peer testing session based on common test plans that all parties are aware of. it "certifies" that two systems have managed to exchange information based on a specific setting (profile). succeeding this three times will for the same test plan (actor pair) will provide a gold star to the system under test usually materialised into an integration statement.
so IHE CAS and IHE CAT are two different things for a different purpose. IHE CAS was not proposed by IHE International as way to make money but to support Vendors from being tested and retested for each project and tender separately and users to reach a defined quality of acceptable specifications. countries such as Switzerland are basing their certification program on top of IHE CAS, the European Commission has issued a program (EuroCAS) to create an extension of IHE CAS for its European Interoperability Framework. other adopters will soon join.
IHE CAS is NOT fading away, quite the contrary, Grapevine will collaborate with IHE to promote IHE CAS not replace it with something else. IHE CAS started in 2015 with 6 profiles and now has 16 profiles included. selection is not done by any board of directors, it is not a political decision but a decision based on end users needs and maturity of specifications. the rule is to include profile that are final text with stable specifications, adding other profiles is possible but is the exception.
for me reusing IHE processes and tools for FHIR Connectathons is a great idea that can be built jointly between both organisations that have a signed MoU already. this is my 2 cent to this very interesting stream.
Grahame Grieve (Jun 15 2017 at 12:36):
thanks for the explanation. We're seeing more and more things crossing the borders in either direction, and we are already being asked about certification for things we've defined
Alexander Berler (Jun 15 2017 at 12:45):
do not hesitate to ask for more information of any type. collaboration is key to healthcare interoperability.
Grahame Grieve (Jun 15 2017 at 12:45):
indeed.
Grahame Grieve (Jun 15 2017 at 12:46):
I'm trying to figure out how I can make an IHE meeting, but none of the ones on the schedule are possible at the moment
John Moehrke (Jun 15 2017 at 13:06):
The difference between an IHE Connectathon and a FHIR Connectathon is slowly disappearing. The main difference is the depth at which the testing occurs. FHIR Connectathon is much like early IHE Connectathon, in that testing is mostly driven by the participants and there might be a script and there might be test tools. The IHE Connectathon has formal test scripts designed to test realistic success/failure, and test tooling. This is not hugely different right now, yes under conformance testing it would be far more formal. FHIR Connectathon is self-reporting, IHE has supervision. Lastly, FHIR connectathons today are less about testing and more about creation of a scenario, they are working sessions. -- no absolutes can be drawn as these are both living organizations, with fickle populations..
John Moehrke (Jun 15 2017 at 13:07):
I had forgot about the conformity assessment pathway, which had test scenarios with complete traceability to the profile requirements. So it was yet more formal in traceability, and oversite. This level of work is... work... it isn't a small effort, and must be done in a way that proves to everyone that no bias exists in the tests or their execution.
René Spronk (Jun 16 2017 at 08:12):
As John summarizes: FHIR connectathons are more like hackathons: one has tracks/themes, but testing is pretty loose, nothing is ever verified by a neutral third party. As such the statement that one participated doesn't mean that much (in the eye of a potential customer/user of ones product).
At the other extreme end we have formal certification (see Alexanders blurb), which is very formal, and which vendors will try to avoid like the plague because of the effort it takes, and the limited value that it has. IMHO they'll only do it if they are forced to do so by law.
IHE connectathons sit somewhere in the middle of that spectrum, with scenario based testing (IHE is about the standardization of workflows, so that is what you'd like to test, not just one single message/data exchange out of its wider context). IHE workflow profiles still have lots of optionalities when it comes to data structures and terminologies, and the test data is limited in scope. The test are being verified by neutral third parties. So even though the testing is not as rigorous as formal certfication, it kinds of hits the sweet spot, both for software developers as well as for end users.
If we were to apply the 80% rule to testing, connectathons would be the thing to go for. The process covers the most useful stuff, acknowledging that there'll be the need for flexibility at the customer side - there will always be a requirement for local adjustments at each and every customer site.
Mario Hyland (Jun 16 2017 at 21:00):
Hello everyone - I would like to correct @René Spronk during the last two (2) HL7 FHIR Connectathon - we have been making progress to introduce more formalize testing at the HL7 FHIR Connectathon - @Ron Shapiro worked with our first foray the "Patient Track (not just for beginners)" - Level 2 - Formal Testing - Participants with FHIR experience (please read more here) http://wiki.hl7.org/index.php?title=201705_Patient_Track. For San Diego we will be expanding this to include the Terminology track. Our objective remains the same - with each proposed HL7 FHIR Connectathon Track, we are encouraging the Track Leads to work with us and generate FHIR TestScripts (http://www.hl7.org/FHIR/testing.html) to test out each use case/test case which is proposed as part of the track. The HL7 FHIR Connectathon become a location we can "test-the-test". This approach has seen TestScripts used to support Patient, Scheduling, Provider Directory, Patch, Genomics and CDSi to name a few. The benefit includes supporting future pre-testing prior to any HL7 (or IHE) FHIR Connectathon, testing at the HL7 (potentially) FHIR Connectathon, and most importantly testing while away from (or those unable to attend) the HL7 FHIR Connectathon. The FHIR Test Platform(s) in use today provide several advantages - we can track statistics and use of resource/profiles - pass/fail - and implementation approaches. As we have demonstrated at the HL7 FHIR Connectathons these results are being reviewed by a third party in a totally automated setting (with immediate feedback). The benefits to the FHIR Community is that they gain ready access 24x7 365 days to ready test environments - which support test early and test often (any time any where). If anyone wants to learn more - please reach out to @Richard Ettema Once we have these formal testing tracks down with comprehensive test cases (which test both Happy Path, and include Negative / Exception handling). The tests can be nominated for a formal Certification (with the "Resources", "Profiles", "IG" and "TestScripts" all having been thoroughly tested by the community). Any number of third parties can then audit/attest to these (auditable) test results from the test platform(s) (and the audit can happen any time from any where). Making FHIR Conformance and Interoperability truly a Global objective.
René Spronk (Jun 17 2017 at 07:15):
Excellent - although this isn't really workflow-scenario based testing (ok, there are dependencies between the various steps, but only minimalistically so) it's (IMHO) certainly a step in the right direction.
John Moehrke (Jun 19 2017 at 14:52):
Oh Mario, now I need to correct you... IHE Connectathon test tools -- Gazelle -- https://gazelle.ihe.net/ is available 24x7x365 on the internet. And there are point people available, and community available..... BUT, I agree that the FHIR test tools are advancing, and surpassing in some specific cases. The FHIR test tools, based on TestScripts, might even be more appropriate for "Test Driven Development"...
Mario Hyland (Jun 19 2017 at 15:00):
@René Spronk when time permits we would welcome an opportunity to share with you the power of Test Script and how it will support many test cases including workflow; this includes complex "Multi-actor" and "Multi-Organziational" test scenarios such as Care Coordination. @John Moehrke yes, understand about Gazelle being online - yes, the objective is to push TDD. Health IT Testing Tools available to run tests, and get results 24x7 365 days any time any where.
Jens Villadsen (Jun 21 2017 at 07:41):
FYI - I've attended as one of the IHE FHIR Monitors in Cleveland 2016 and in Venice 2017. Both times I've had to write tools that automated 99% of the validation effort, because Gazelle did not provide any such tooling.
Last updated: Apr 12 2022 at 19:14 UTC
