Stream: inferno
Topic: Continuous Integration Systems
Guy Bertental (Jun 23 2021 at 11:30):
Hi, we are looking into ways of automating Inferno tests, is there a way to export all test results in a structured format?
Stephen MacVicar (Jun 24 2021 at 14:02):
We don't currently support that, but it would be a nice feature to have. We'd need another rake task that fetched the results from the database and formatted them.
Guy Bertental (Jul 12 2021 at 08:35):
Hi,
On the CLI version of inferno, is it possible to run all SMART on FHIR (US Core) test sequences?
we are trying to figure out the input parameters per module, whether we need to generate the token outside inferno or provide credentials in the config file for Continuous Integration systems...
Guy Bertental (Jul 13 2021 at 11:50):
Guy Bertental said:
Hi,
On the CLI version of inferno, is it possible to run all SMART on FHIR (US Core) test sequences?
we are trying to figure out the input parameters per module, whether we need to generate the token outside inferno or provide credentials in the config file for Continuous Integration systems...
I meant SMART of FHIR tests and not the US Core tests as I mentioned in the parentheses.
Robert Scanlon (Jul 14 2021 at 15:18):
Hi @Guy Bertental -- sorry for the slow response. The challenging part of the SMART on FHIR tests is that it requires a browser to be driven by a script to click all the right buttons in the authorization UI. We did experiment with this early on, and was able to get it to work, but found that it was too hard for others to configure and run so we haven't put effort into maintaining it. I recommend passing the token to the US Core inferno tests, instead of trying to run the SMART sequences. But if you are really interested in dusting off that capability and seeing if you can get it working, then I can check the current status (working?/almost working?/or completely broken?) and point you to the right spot so you can try it out.
Guy Bertental (Jul 15 2021 at 10:55):
Many thanks for your response @Robert Scanlon, we would have liked to make it work if it's possible, issuing the token outside inferno is exactly what we did eventually but manually with Postman because of the interactive nature of the authorization process, we failed to issue the token with REST calls (cURL tool) and authorization code grant type, This is something that we are missing and still trying to figure out, as this will enable a wide array of tests including SMART oriented tests to our FHIR implementation. but unfortunately, we don't have more time to invest in it now, maybe we will come back to this in the future.
Priscilla Hakel (Jul 15 2021 at 15:13):
Anyone knows if https://inferno.healthit.gov/validator/ server is down? I'm getting 504 (the server responded with a status of 504 (Gateway Time-out)) since yesterday when the online validator tried to fetch the drop-down list of IGs or Profiles.
Ozlem Kurt (Jul 19 2021 at 22:41):
Hello, does anyone know if inferno server is updated over the weekend? There are new requirements from last Friday to today and I was wondering if someone can inform us on that.
Thanks.
Robert Scanlon (Jul 20 2021 at 01:05):
Hi @Ozlem Kurt -- there haven't been any updates since June 11 to the ONC Certification tests. What are you seeing that is different from last week?
Ozlem Kurt (Jul 20 2021 at 14:43):
DocumentReference Tests - USCDR-13: Could not find content.attachment.url in the 5 provided DocumentReference resource(s). We have 5 clinical notes and all of them are plain text and coded as below:
"content": [
{
"attachment": {
"contentType": "text/plain",
"data": "UHJvZ3Jlc3MgTm90ZSBUZXh0OiBEciBBbGJlcnQgRGF2aXMgZGlhZ25vc2VkIE1zIEhhcHB5IEtpZCB0byBiZSBzdWZmZXJpbmcgZnJvbSBGZXZlciBhbmQgc3VzcGVjdGVkIFBuZXVtb25pYSBhbmQgcmVjb21tZW5kZWQgYWRtaXNzaW9uIHRvIHRoZSBDb21tdW5pdHkgSGVhbHRoIEhvc3BpdGFscy4="
},
"format": {
"system": "http://ihe.net/fhir/ValueSet/IHE.FormatCode.codesystem",
"code": "urn:ihe:iti:xds:2017:mimeTypeSufficient",
"display": "mimeType Sufficient"
}
}
],
I am also comparing to reference server and when contentType is plain/text, it does not have attachment url but still passing. We did not have tis issue last Friday as an example. There are couple more issues that I can list if you'd like.
Thanks,
Ozlem
Yunwei Wang (Jul 20 2021 at 18:59):
Hello @Ozlem Kurt Are you running tests on Inferno Program or Inferno Community?
Ozlem Kurt (Jul 21 2021 at 13:41):
Hi, I am using program edition but I am not getting the same errors starting by 2:00pm yesterday. Nothing has changed on my side though.
Thanks
Robert Scanlon (Jul 21 2021 at 14:13):
Is there any chance there is variability in the data that you are sending back to Inferno? Inferno only evaluates based on what it sees in a given test run, so if for some reason a different set of resources is returned then the results would be different.
Robert Scanlon (Jul 21 2021 at 14:15):
Note that US Core v3.1.1 requires systems to support BOTH "url" and "data" for this profile, so we need to see at least one example of using "url" in order for the test to pass. This requirement is very likely wrong and buts us in a difficult position of either enforcing a wrong constraint or relaxing a stated requirement in a consensus standard. So the tests not passing is the correct behavior in the case you gave. I do not know why there is variability in results though.
Ozlem Kurt (Jul 21 2021 at 15:35):
Hi Robert,
It was exactly the same data and same application.
Thanks,
Ozlem
Yunwei Wang (Jul 21 2021 at 16:44):
Hi @Ozlem Kurt Would you mind to save a testing session so we can further investigate this issue? Thanks.
Ozlem Kurt (Jul 26 2021 at 19:23):
Hi Yunwei, can you please see the link below for both Single and Multi patients? https://inferno.healthit.gov/inferno/7HPsNTQzLLg/
the errors I mentioned started to occur again.
Stephen MacVicar (Jul 26 2021 at 19:51):
None of the DocumentReferences in that test session contain content.attachment.url.
Ozlem Kurt (Jul 26 2021 at 21:50):
Hello Stephen, we have only text attachment for document reference and reference server has the same type wit no url which had no issues. Does url accept any string null value?
Thanks,Ozlem
Ozlem Kurt (Jul 26 2021 at 22:57):
Hello Stephen,
We had this all passing with the same data under this url: https://inferno.healthit.gov/inferno/8XK8dwqrPKs/
As you can see there are no urls for Document reference when type is a plain text and it was still passing.
Can you please help to clarify the issue here?
Thanks,
Ozlem
Ozlem Kurt (Jul 27 2021 at 14:15):
Hi,
We seem to have the same issue every Monday for some reason. For example, I was getting errors on Pulse Oximetry Test under this state: https://inferno.healthit.gov/inferno/7HPsNTQzLLg/ on Monday and without any change to Observations I am getting no errors today on Tuesday. On Mondays, without any changes to server or data, our tests start failing. I wonder if they do some sort of maintenance on Saturdays like cypress does.
Thanks,
Ozlem
Yunwei Wang (Jul 27 2021 at 14:56):
@Ozlem Kurt I would like to clarify your testing procedure. When you started a test on Tuesday, did you use "Rerun Tests" on the same test session or did you start a new test session?
Ozlem Kurt (Jul 27 2021 at 15:18):
Hi Wang,
It was the same session. You can see that we sent you this state on Monday (https://inferno.healthit.gov/inferno/7HPsNTQzLLg/ ) was failing single and multi patient APIs and same state today with no change on Observations for Pulse Oximetry Test.
Hope that helps.
Thanks,
Ozlem
Yunwei Wang (Jul 27 2021 at 18:31):
Hello @Ozlem Kurt Can you start a new test session from the beginning and send me the saved session no matter it is passed or failed. Thank you.
Yash Palkhiwala (Nov 05 2021 at 19:33):
Hi there, reviving this topic. We are also looking to parse test results from the inferno CLI in our CI pipeline. Are there plans to support outputting test results in a structured format (i.e. something that can be consumed by the Jenkins junit/xunit plugin)?
Last updated: Apr 12 2022 at 19:14 UTC
