Stream: inferno
Topic: Inferno 2.0
James Threatte (Mar 07 2022 at 18:08):
Hello
I see Inferno 2.0 is now live on inferno.healthit.gov. I have an EHR vendor preparing to test tomorrow and without release notes I can't force them to use 2.0. I see Inferno 1.9.0 is also still available on inferno.healthit.gov. Is this instance of Inferno 1.9.0 fully functional?
Robert Scanlon (Mar 07 2022 at 18:12):
Hi James -- yes v1.9 is fully functional and supported on inferno.healthit.gov, and nothing should have changed for them (e.g. same URLs, etc). If you plan to use the inferno.healthit.gov instance for certification, could you please have them walk through everything just to ensure that nothing inadvertently changed?
Mat Davis (Mar 08 2022 at 01:02):
Hello. My name is Mat Davis
I am a Quality Analyst with Dynamic Health IT.
I am writing today about the recent and unexpected updated Inferno versions below:
- (g)(10) Standardized API Test Kit
- (Legacy) Inferno Program Edition
We are fully aware of the notifications of 1) "Data is removed every Sunday..." and 2) "This URI is not guaranteed to be permanently accessible."
The issue is that the update to the newer versions was so abrupt that there was no time to prepare or be aware that any changes of this nature were taking place.
Q: Is the UI the same in the newer version?
A: No
Q: Is the process the same?
A: Appears to be but we will need to test to confirm.
Q: Are we required to use the new version or can we still use the Legacy version if we prefer?
A: This is not clear.
Q: What is the reason for this change in / additional versions?
A: Unknown. As a daily user of Inferno, having an idea of this change would've been very informative and even exciting.
Q: Will the saved URIs from previous testing be accessible?
A: No. Clicking previous links results in a page can't be found.
While there is a clear notification that "This URI is not guaranteed to be permanently accessible.", there have been URI's being used and accessed by our team that were created as far back as Nov 2021.
These URI's are used to easily access previous test, test results and test configurations.
Q: Is there any recourse for users who use these URI's as a reference point?
A: I would hope so but if not, we can do what needs to be done to move on and continue using this super helpful tool.
There are also no release notes associated with the (g)(10) Standardized API Test Kit version of Inferno which would be at the very least, a very helpful starting point for new adoption.
While improvements are always welcome and even necessary, the point is that not having any notification of this change is disruptive, frustrating and just so unexpected.
I look forward to your feedback and the feedback of others.
Thanks - Mat
Robert Scanlon (Mar 08 2022 at 17:55):
Hi @Mat Davis --thanks for reaching out and letting us know about these issues / concerns, and how they are disruptive. We are working on updating our language throughout our repos / sites to better explain the changes. Bottom line is that v1.9 -> v2.0 represents a major update to the technical foundation of these certification tests. While the test changes themselves are intended to be minimal, so much else is changing in v2.0 (including the name) that we will continue to support v1.9 for the near future. You can still use Inferno Program Edition v1.9 at the same URL as before, and besides the fact that we reset the stored test data on the site this weekend everything else should be identical.
Mat Davis (Mar 08 2022 at 18:11):
Thanks so much for your reply Robert. Happy to hear my concerns and likely the concerns of others are heard and understood.
Just for clarification, and based off the answer to James above, I'm assuming this will be yes,
Can v1.9 aka should v1.9 be used when certifying?
Any direct or acceptable answer would be appreciated here.
I am more than happy to welcome, accept and use newer versions of Inferno with minor and major improvements.
I'll start tO use v2.0 and give some honest compliments and feedback after multiple uses.
Robert Scanlon (Mar 08 2022 at 18:15):
I do not speak for ONC officially, but the plan is to allow systems to use v1.9 for (g)(10) certification for some time to avoid disruption for anyone planning on certifying in the near future, and to give people a chance to try out v2.0+ and report any issues before retiring v1.9.x. That is why we are now hosting both versions side-by-side on inferno.healthit.gov.
Robert Scanlon (Mar 08 2022 at 18:18):
Hopefully I have something specific I can point to from them directly on this soon
Robert Scanlon (Mar 08 2022 at 18:23):
How critical is having the previous test data for you? We may be able to recover it with some effort (I'm checking now), but we wanted to avoid having users rely on data persistence on the ONC-hosted site because it takes effort to maintain historic data.
Robert Scanlon (Mar 08 2022 at 18:28):
Regarding the release notes for v2.0, please see https://github.com/onc-healthit/onc-certification-g10-test-kit/releases/tag/v2.0.0.
Mat Davis (Mar 08 2022 at 19:00):
This makes perfect sense so thank you for answering to the best of your responsibility / purview.
Mat Davis (Mar 08 2022 at 19:01):
Thanks again on this front!
Mat Davis (Mar 08 2022 at 19:06):
Robert Scanlon said:
How critical is having the previous test data for you? We may be able to recover it with some effort (I'm checking now), but we wanted to avoid having users rely on data persistence on the ONC-hosted site because it takes effort to maintain historic data.
At this point, if it's possible, feel free to try or at least get whatever confirmation you feel comfortable pursuing. As of yesterday, I have focused on just building from scratch and building workflows that will save results to PDF or HTML to have a history of tests, results, configs, data (if possible)
"Avoiding having users rely on data persistence" is a understandable and important goal to move toward from your end and in general. The roll out of the initiative overtime, by the end of the week, by the end of the month, any progressive manner overtime could've helped with that adoption. I will definitely be proactive and prepared on this front moving forward.
Mat Davis (Mar 08 2022 at 22:44):
Robert Scanlon said:
How critical is having the previous test data for you? We may be able to recover it with some effort (I'm checking now), but we wanted to avoid having users rely on data persistence on the ONC-hosted site because it takes effort to maintain historic data.
Hi Robert.
As some additional notes to your comment above from other members of the team, accessing prior saved URI's have been very helpful in supporting our clients.
It allowed our clients to know where they left off in regards to testing, how they improved or declined in testing as well as seeing when a resource was erroring due to data.
This was especially helpful after an Inferno update to help determine if the issue was actually due to data or if it was due to recent changes in the test tool itself.
We understand the URI's can disappear but our clients and ourselves have gotten really comfortable with using inferno as a helpful tool in development and implementation not only ONC testing.
Johnny Bender (Mar 09 2022 at 20:51):
Hey everyone, wanted to post some information from ONC on this thread that may be helpful regarding the new Inferno Framework and (g)(10) Standardized API Test Kit. See below and thank you!
What is in the new release of Inferno?
This new release of Inferno is called the “Inferno Framework” and includes an upgrade of the overall platform which will allow ONC to better support standards conformance testing and development by supporting multiple versions of standards in tests to accommodate the Standards Version Advancement Process. It also includes new features, like a JSON API to help with tool integration, and a revised user interface to accommodate a variety of standards conformance testing use cases such as HL7 FHIR and supporting standards. ONC updated Inferno to make it easier for the health IT community to develop tests using the Inferno Framework without needing to directly engage with ONC. Learn more about the Inferno Framework and its features including Inferno Core, and Test Kits.
Included in the Inferno Framework is the (g)(10) Standardized API Test Kit, which supports testing for the § 170.315(g)(10) certification criterion adopted as part of the ONC Health IT Certification Program. ONC took special care to ensure that the test scenarios and tests in this new release are the same as the previous Inferno Program Edition (Release 1.9).
When was Inferno Framework released?
ONC announced the release of Inferno Framework including the (g)(10) Standardized API Test Kit on March 8, 2022. To support this release, ONC will continue to host the previous Inferno Program Edition (Release 1.9) on inferno.healthit.gov/inferno for approximately two months. ONC recommends Authorized Test Labs (ATLs) and developers continue to support Certification testing via the previous Inferno Program Edition (Release 1.9) for at least one month to help with the transition.
To support the health IT community with the rollout of Inferno Framework, ONC will host a month of health IT developer Inferno office hours, where ONC will review the Inferno Framework and (g)(10) Standardized API Test Kit releases on 3/18, 3/25, 4/1 and 4/8 from 2:00 pm – 3:00 pm ET. Please subscribe to ONC Certification email updates by entering your email into the “Sign Up for Certification Email Updates” box on healthit.gov. Instructions will also be posted on the Inferno Google Group and this thread on how to join these Inferno office hours.
What are the requirements for using Inferno Framework for ONC Health IT Certification?
Heath IT developers should consult with their Authorized Testing Laboratories (ATLs) for specific testing requirements for the § 170.315(g)(10) criterion, and ONC recommends that ATLs continue to support ONC Certification testing with Inferno Program Edition (Release 1.9) for at least one Inferno release cycle (one month). ONC will continue hosting Inferno Program Edition (Release 1.9.0) at inferno.healthit.gov/inferno for approximately two months. All future enhancements for certification testing for the (g)(10) criterion will be made in the (g)(10) Standardized API Test Kit. We encourage health IT developers to begin using (g)(10) Standardized API Test Kit and provide any feedback on the (g)(10) Standardized API Test Kit GitHub page.
Are the tests the same in previous Inferno Program Edition (Release 1.9.0) and the new (g)(10) Standardized API Test Kit in Inferno Framework?
Basically, yes. ONC took special care to ensure that the test scenarios and tests in this new release are the same as the previous Inferno Program Edition (Release 1.9). However, there may be small changes in test results for systems between the versions. The release notes for the (g)(10) Standardized API Test Kit on GitHub provide an overview of the updates in (g)(10) Standardized API Test Kit. Please report any discrepancies in GitHub Issues section of the (g)(10) Standardized API Test Kit repository so that they can be documented and fixed if necessary. For specific details on the changes and enhancements to the (g)(10) certification criterion tests, please review the release notes.
How do I learn more about Inferno Framework and (g)(10) Standardized API Test Kit?
ONC announced the release of Inferno Framework on the Health IT Developer Roundtable on March 9 and provided an overview on the Inferno Tech Talk on March 9. Please subscribe to the Inferno Google Group to receive notifications about future Inferno Tech Talks.
ONC will host a month of health IT developer office hours where ONC will review the new release of Inferno Framework including the (g)(10) Standardized API Test Kit on 3/18, 3/25, 4/1 and 4/8 from 2:00 pm – 3:00 pm ET. Please subscribe to ONC Certification email updates by entering your email into the “Sign Up for Certification Email Updates” box on healthit.gov. Instructions will also be posted on the Inferno Google Group and this thread on how to join these Inferno office hours.
Johnny Bender (Mar 14 2022 at 17:16):
Inferno Office Hours Sessions for Health IT Developers
The ONC Health IT Certification Program will be hosting a month of Inferno office hour sessions for health IT developers beginning this Friday, March 18, 2022. These weekly sessions will occur on 3/18/2022, 3/25/2022, 4/1/2022, and 4/8/2022 from 2:00 pm – 3:00 pm ET. Health IT developers implementing and certifying to the ONC Certification criterion at 170.315(g)(10) are encouraged to attend.
The Inferno team will cover the release of Inferno Framework, including (g)(10) Standardized API Test Kit. You can learn more about the changes that are part of this release by reading the release notes in the (g)(10) Standardized API Test Kit GitHub repository. The prior release of the Inferno Program Edition (Release 1.9) will be hosted on inferno.healthit.gov/inferno for at least the next two months to support health IT developers’ transition to the new version of the tool.
We encourage all Inferno users to visit and test the new Inferno Framework and please continue to provide feedback on the (g)(10) Standardized API Test Kit GitHub page, engage the developer community on the chat.fhir.org Inferno page, and send ONC any inquiries using the ONC Health IT Feedback and Inquiry Portal.
For more information, including registration, please see here.
Johnny Bender (Mar 18 2022 at 18:21):
Hi everyone! We ran into some technical difficulties with our Inferno Office Hours series today 3/18, so we are unfortunately cancelling the session today and will extend the office hours to 3/25/2022, 4/1/2022, 4/8/2022, and 4/15/2022 from 2:00 pm - 3:00 pm ET. Apologies for the inconvenience and looking forward to meeting with everyone!
Daniel Gabriel (Mar 25 2022 at 15:37):
Hey everyone,
We're looking at the Inferno 2.0 tests for bulk patient Implantable Device profile (BDGV-10) and running into a strange issue. In Inferno 1.0 the same data / response passes the test, but in 2.0 the test result claims to be missing required fields. The fields in question, related to UDI-PI are, in fact, present in some of the resources returned. These fields are designated as must-support. Is there a known issue with this test in 2.0 or something different that we must provide related to those fields other than what is required for must-support fields?
Even with the must-support fields present, Inferno 2.0 claims:
Device: us-core-9: 'For implantable medical devices that have UDI information, at least one of the Production Identifiers (UDI-PI) SHALL be present.' Rule 'For implantable medical devices that have UDI information, at least one of the Production Identifiers (UDI-PI) SHALL be present.' Failed
Yunwei Wang (Mar 25 2022 at 15:52):
Invariant (us-core-9 here) is enforced by FHIR validator. Every instance must valid against invariant rules. Can you check if any device in ndjson file does not have any of these UDI-PI elements populated:
The following parsed Production Identifiers (UDI-PI) from the UDI
the manufacture date
the expiration date
the lot number
the serial number
the distinct identifier (i.e., the distinct identification code)
Daniel Gabriel (Mar 25 2022 at 16:00):
Yes, there are some instances that don't have all of those values available. In those cases we do not (cannot since it does not exist) provide a value. We do provide a data absent reason using an extension, like so:
"_serialNumber":{"extension":[{"url":"http://hl7.org/fhir/StructureDefinition/data-absent-reason","valueCode":"unknown"}]}
Daniel Gabriel (Mar 25 2022 at 16:04):
In the case of provider-documented historical implants the UDI-PI fields may not be provided and therefore could not be present in the resulting FHIR object. Our test data contains examples which have the fields provided, which does demonstrate support of the must-support fields. I would understand the test failing if exactly none of the returned resources demonstrated support for the field, but that is not the case.
Yunwei Wang (Mar 25 2022 at 16:19):
This is not about must support validation. Each resource must be valid against invariant rules. The rule is "For implantable medical devices that have UDI information, at least one of the Production Identifiers (UDI-PI) SHALL be present." That means that if a device has UDI then it must have at least one of the UDI-PI. If you can send me a device resource that failed this test, I can manually check if that resource is valid using FHIR validator.
Cooper Thompson (Mar 25 2022 at 16:53):
This might make sense to submit as a US Core Jira, which we can mark as a patch fix to get industry agreement so Inferno can remove the invariant. Invariants like this fall into my "doesn't handle real world data" bucket that I bring up as often as anyone will listen (see Period Datatype Invariant and FHIR-25184).
Cooper Thompson (Mar 25 2022 at 16:54):
The other option is to just not provide a udiCarrier at all. Depending on what you do have, I wonder if adding the DAR is actually what is triggering the invariant.
Daniel Gabriel (Mar 25 2022 at 17:01):
This is what I was suspecting as well. Given that it passes in Inferno 1.0, I think something subtle could be happening here. For what it's worth the resource does pass the individual US Core Profile tests.
Yunwei Wang (Mar 25 2022 at 17:26):
us-core-9 is removed and replaced by several paragraph of narrative in US Core v4.0. But I am still confused by the difference behavior between Infenro v1.9 and v2.0. We didn't make any intented validation logic change in v2.0. If they are truely difference, it could be side effect from other changes.
Daniel Gabriel (Mar 25 2022 at 17:32):
I'm going to try running this without the DAR fields and see if that addresses the issue. Will report back. Thanks!
Nathan Loyer (Mar 28 2022 at 18:54):
hey all, found another bug in inferno 2, just opened a ticket in github for it. This issue is fairly minor, but still leads to some confusion and would be good to address it
https://github.com/onc-healthit/onc-certification-g10-test-kit/issues/58
Yunwei Wang (Mar 28 2022 at 21:17):
@Daniel Gabriel @Cooper Thompson @Nathan Loyer There is new requirement in US Core 4.0 about Device UDI/UDI-PI
Implantable medical devices that have UDI information SHALL represent the UDI code in Device.udiCarrier.carrierHRF.
- All of the five UDI-PI elements that are present in the UDI code SHALL be represented in the corresponding US Core Implantable Device Profile element.
I created a ticket FHIR-36655 for clarification. If you have any input, please add to that ticket.
Nathan Loyer (Mar 28 2022 at 21:20):
If it's an update for US Core 4.0, then that shouldn't affect the g10 test kit, right?
Yunwei Wang (Mar 28 2022 at 21:22):
That is correct.
Daniel Gabriel (Mar 29 2022 at 13:20):
Thanks for looking into this! So it seems like the expectation is that all 5 UDI-PI fields will be provided, or none shall be provided?
Daniel Gabriel (Mar 29 2022 at 13:40):
In the situation where we have partial information, does that mean we are expected to exclude any UDI-PI fields which may be present, or exclude the UDI entirely?
If I'm reading this correctly, unless we have all 5 UDI-PI fields, maybe we don't actually have UDI information provided as described.
Daniel Gabriel (Mar 29 2022 at 14:07):
My concern is that UDI, UDI-DI, and UDI-PI fields are marked as must support. In the case where we have partial results such that the described invariant would fail, can we exclude the information?
Daniel Gabriel (Mar 29 2022 at 14:33):
This might also be a matter of difference between US Core 3.1.1 and US Core 4.0.0. If we specify our devices as US Core 3.1.1 (which our resources should comply to) would the tests pass? :thinking:
Yunwei Wang (Mar 30 2022 at 14:57):
Here is a JIRA ticket asking US Core to clarify that requirement FHIR-36657. The previous one was closed due to workflow issue.
US Core 3.1.1 asks for that "Each Implantable Device must have" at least one UDI-PI (invariant uscore-9) for each instance.
Daniel Gabriel (Apr 04 2022 at 17:36):
Hey all, we've filed a bug in Inferno Core that looks to be related to the issue we're experiencing: https://github.com/inferno-framework/inferno-core/issues/154
Daniel Gabriel (Apr 04 2022 at 17:38):
It appears that even when a specific US Core profile is passed as part of a validation run, Inferno Core will use the validator for the most recent version of that profile.
Nathan Loyer (Apr 04 2022 at 19:17):
Here's another issue that I logged that if implemented would have helped us debug this instance failing that Dan linked to.
https://github.com/inferno-framework/inferno-core/issues/155
The proposal is to persist the HTTP requests to the fhir validator. I had tried forking and writing it myself, but ran into some issues and gave up on that.
Yunwei Wang (Apr 04 2022 at 20:05):
I have replied to issue-145. Inferno does use the correct profile version.
Yunwei Wang (Apr 04 2022 at 20:15):
Also, kind reminder, please raise new issue at onc-certification-g10-test-kit repo. (https://github.com/onc-healthit/onc-certification-g10-test-kit). That is the main repo we are using for any g10 testing related issues no matter what the root cause is. It is easier for us to track and response in a timely manner.
Nathan Loyer (Apr 05 2022 at 12:36):
ah, ok. I figured it made more sense to put the issue in the repo that would address it
Yunwei Wang (Apr 05 2022 at 13:55):
Thanks for the feedback. I would say if the issue is related to g10 test, raise the issue in g10-test repo though the root cause may lays on somewhere else, ex Device resource failed with validator error. If the issue is not related to g10 test and more fundamentally affect other test kit, then raise it in corresponding repo, ex add logging for validation request. The reason is that we have all the tracking build in g10 repo. Any status change on an issue is reflected with label changes, such as "will fix", "resolved", "release in v2.0" etc. So it is easier for issue report to know the expectation. Also it is easier for us to estimate how many issues need to be addressed and prioritize g10 issues.
Daniel Gabriel (Apr 05 2022 at 18:02):
Hi @Yunwei Wang we are still experiencing this issue. Notably, we are able to get identical test data passing on Inferno Program Edition 1.9.0 but see the Implantable Device profile as failing on ONC Certification (g)(10) Standardized API V.2.0.0 build on Inferno 0.2.0. Should we open a ticket on https://github.com/onc-healthit/onc-certification-g10-test-kit?
Nathan Loyer (Apr 05 2022 at 18:28):
So just got through debugging this some more with the mocked out validator that we deployed. We have it just logging all the input data so we can debug it, and it looks to me like inferno is stripping out our data absent reasons from our resource before sending it to the validator.
Yunwei Wang (Apr 05 2022 at 18:29):
Yes. Please raise an issue at g10-test-kit repo with any details you may provide. If you have the ndjson data set without any PHI, please include that too. You can also send the dataset to me directly if you don't want to post on the github.
Yunwei Wang (Apr 05 2022 at 18:30):
@Nathan Loyer That is interesting. Do you have an example of the original resource in the NDJSON and the modified resource sending to validator?
Nathan Loyer (Apr 05 2022 at 18:35):
details are in #154, I'll make a new ticket and link to that one
Yunwei Wang (Apr 05 2022 at 18:36):
Is that the "modified" resource that inferno sent to validator?
What is the original one in NDJSON?
Do you have an screenshot for the error message on UI?
Thanks
Nathan Loyer (Apr 05 2022 at 18:39):
https://github.com/onc-healthit/onc-certification-g10-test-kit/issues/67
Nathan Loyer (Apr 05 2022 at 18:39):
it's the error message listed in the description of #154
Last updated: Apr 12 2022 at 19:14 UTC
