Stream: inferno
Topic: Intermittently passing Inferno when we shouldn't be
Cooper Thompson (Sep 02 2021 at 14:54):
Is there any known issue (didn't see anything in github) where Inferno will sometimes pass tests when it shouldn't? For example, maybe test results for one server is getting mixed up in the test results of another server?
We have seen this intermittently. There are some tests we know we should NOT be passing (yet). For example, USCDR-13 (due to https://github.com/onc-healthit/inferno-program/issues/316). However, somehow, we are passing on the public 1.6.2 version.
Here is an example test session: https://inferno.healthit.gov/inferno/e7X7MhhmJlv/test_sets/test_procedure/
Cooper Thompson (Sep 02 2021 at 14:56):
Historically it seems like it was intermittent, but we've been unexpectedly passing some tests for ~3 days in a row now, so recently it has seemed less intermittent.
Cooper Thompson (Sep 02 2021 at 14:59):
Tests we don't expect to pass:
USCG-08
USCDR-13
USCID-08
USCP-13
Looking at these more closely, they all appear to be must-support checks.
Stephen MacVicar (Sep 02 2021 at 15:46):
This issue can cause the reference resolution tests to attempt to read references from the wrong server, but I'm not aware of any issues that would affect the MS tests. I will investigate further, but won't be able to get to it until next week.
Cooper Thompson (Sep 02 2021 at 15:53):
Ah - I wonder if we might have been seeing two different things. I bet the intermittent issues we saw were related to the race condition, but the must-support issue may be unrelated. Do you want me to submit a github issue?
Stephen MacVicar (Sep 02 2021 at 15:58):
Well, that race condition typically causes tests to intermittently fail rather than intermittently pass, because it's attempting to use ids from server A to read resources from server B.
A github issue would definitely be helpful for tracking this.
Cooper Thompson (Sep 02 2021 at 18:25):
https://github.com/onc-healthit/inferno-program/issues/355
Last updated: Apr 12 2022 at 19:14 UTC
