Stream: IPS
Topic: IPS support in the validator
Grahame Grieve (Oct 06 2021 at 11:55):
It's been tricky to use the validator to validate IPS documents. After the next release, due in 24-48 hours, you can just specify -ips as a parameter, and the validator will figure out all the other parameters. It defaults to the connectathon-2 branch, but you can specify -ips$[branch} for another alternative
Grahame Grieve (Oct 06 2021 at 11:55):
so just
java validator.jar -ips {filename}
Rob Hausam (Oct 06 2021 at 11:59):
Nice.
Reece Adamson (Oct 06 2021 at 11:59):
What's the difference between this and testing against the IPS Bundle Profile? Just more helpful error messages?
Context: We have tooling that uses the FHIR validator Java classes (e.g. https://inferno.healthit.gov/validator) and one thing we support is validating IPS bundles against the IPS bundle profiles so supporting better validation is important to us.
Grahame Grieve (Oct 06 2021 at 12:04):
well, it's a syntactical shortcut to validate against the IPS bundle profile using the branch of your choice. So it sounds like pretty much the same thing
Grahame Grieve (Oct 06 2021 at 12:05):
I'm not sure how you go keeping the version up to date
Reece Adamson (Oct 06 2021 at 12:07):
Keeping the version up to date is not an automated process currently so we load the IG from a local copy to make sure we're getting the correct branch version (relevant code in our app)
Reece Adamson (Oct 06 2021 at 12:08):
A user can also manually load a specific package.tgz
using the web app at https://inferno.healthit.gov/validator/
Grahame Grieve (Oct 06 2021 at 12:08):
yes it's tricky. Anyway, if you're running the java validator internally, you'll get all the fundamentals I'm doing
Rob Hausam (Oct 06 2021 at 12:11):
Is this specifying the -ig
(I assume it is)?
Grahame Grieve (Oct 06 2021 at 12:15):
-ips looks after all the details like -ig
Giorgio Cangioli (Oct 06 2021 at 12:18):
Grahame Grieve said:
It's been tricky to use the validator to validate IPS documents. After the next release, due in 24-48 hours, you can just specify -ips as a parameter, and the validator will figure out all the other parameters. It defaults to the connectathon-2 branch, but you can specify -ips$[branch} for another alternative
Great ! I'll try with the examples I have
Grahame Grieve (Oct 06 2021 at 12:31):
@Reece Adamson we need some process for reconciling when the java validator returns different errors than inferno.
Grahame Grieve (Oct 06 2021 at 12:31):
it's happening.
Reece Adamson (Oct 06 2021 at 12:58):
The differences (should) come down to the version of the validator used. Inferno is using version 5.3.14 of the validator library, but the most recent published is 5.5.4. I'm working on updating that now.
Our team should update this dependency more frequently to keep pace though and provide back unit tests if we notice any unexpected changes. Up for thoughts or suggestions for improved coordination!
Grahame Grieve (Oct 06 2021 at 12:59):
yes that's quite the gap. That will explain the difference
Grahame Grieve (Oct 06 2021 at 12:59):
what java class are you using?
Reece Adamson (Oct 06 2021 at 13:02):
primarily ValidationEngine (relevant source file)
Sumanth Bandaru (Oct 06 2021 at 13:09):
HI Grahame, Can you please validate this sample and give us feedback?
Thank you,
Sumanth Bandaru
Reece Adamson (Oct 06 2021 at 14:15):
@Grahame Grieve I made a PR to update the dependency, but it looks like it breaks Java 8 support. Not sure if we should move to another stream or a private thread for further discussion, but here's some details on what I'm seeing for now.
inferno-validator-wrapper CI results showing Java 8 unit tests failing. Still tracking down exactly what the issue is.
Interestingly, It looks like the maven compiler plugin is targeting Java 8 in the org.hl7.fhir.core pom, but the CI appears to be targeting Java 11. Is Java 8 intended to be supported?
Grahame Grieve (Oct 06 2021 at 14:22):
@Mark Iantorno handles all these things for us. Mark, can you look into this?
Mark Iantorno (Oct 06 2021 at 14:22):
Hey
Mark Iantorno (Oct 06 2021 at 14:23):
Java 8 is intended to be supported. Let me read the full chat one sec
Grahame Grieve (Oct 06 2021 at 14:23):
might move the Java specific stuff to #hapi
Mark Iantorno (Oct 06 2021 at 14:24):
We can do that. But first, can someone link me to the PR that apparently breaks Java8
Grahame Grieve (Oct 06 2021 at 14:24):
that's Reece's PR to his own code
Mark Iantorno (Oct 06 2021 at 14:25):
ok, well @Reece Adamson if you give me access and send me a link, we can work on alternatives
Mark Iantorno (Oct 06 2021 at 14:25):
also, I'm confident that we need to keep support for Java8 ongoing
Mark Iantorno (Oct 06 2021 at 14:25):
for HAPI
Reece Adamson (Oct 06 2021 at 14:26):
- My PR (just updates the validator dependency from 5.3.14 -> 5.5.4)
- CI Details showing Java 11 & 16 passing, but Java 8 failing
Mark Iantorno (Oct 06 2021 at 14:26):
Wait
Mark Iantorno (Oct 06 2021 at 14:26):
so Java8 passed before
Grahame Grieve (Oct 06 2021 at 14:26):
it's pretty old
Mark Iantorno (Oct 06 2021 at 14:27):
and the change that made it fail was bumping core from 5.3.14 to 5.5.4
Mark Iantorno (Oct 06 2021 at 14:28):
So it's this test ValidatorTest > getKnownIGs() FAILED
org.opentest4j.AssertionFailedError at ValidatorTest.java:100
Reece Adamson (Oct 06 2021 at 14:28):
Correct.
Mark Iantorno (Oct 06 2021 at 14:28):
So, I don't think this is a Java 8
issue
Mark Iantorno (Oct 06 2021 at 14:29):
I think there is a dependency in your tests for Java 8 that is no longer present in the core, let me check
Mark Iantorno (Oct 06 2021 at 14:29):
I'm cloning the project now
Reece Adamson (Oct 06 2021 at 14:29):
It passes on 11 & 16 though and all I did was update the validator dependency
Mark Iantorno (Oct 06 2021 at 14:29):
Is the code identical for all the tests other than the version of Java?
Reece Adamson (Oct 06 2021 at 14:31):
yup, everything is the same except java version. You can see the CI config here. I can reproduce it locally as well trying the docker build/test.
Mark Iantorno (Oct 06 2021 at 14:31):
I'm looking right now
Reece Adamson (Oct 06 2021 at 14:32):
gotcha let me know if I can help! Still trying to investigate myself in the meantime.
Mark Iantorno (Oct 06 2021 at 14:35):
fwiw, I'm happy you're using gradle
Reece Adamson (Oct 06 2021 at 14:36):
:laughing: me too
Mark Iantorno (Oct 06 2021 at 14:38):
So...the code works? Here's where I'm at
Mark Iantorno (Oct 06 2021 at 14:39):
the fetched list of profiles in the ig contains 30 entries
Mark Iantorno (Oct 06 2021 at 14:39):
0 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-condition-parent"
1 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-disease-status"
2 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-genetic-variant"
3 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-genomics-report"
4 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-patient"
5 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-related-medication-statement"
6 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-related-radiation-procedure"
7 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-related-surgical-procedure"
8 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-stage-parent"
9 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-comorbid-condition"
10 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-ecog-performance-status"
11 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-evidence-type"
12 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-genetic-specimen"
13 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-genomic-region-studied"
14 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-histology-morphology-behavior"
15 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-karnofsky-performance-status"
16 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-laterality"
17 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-primary-cancer-condition"
18 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-related-primary-cancer-condition"
19 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-secondary-cancer-condition"
20 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-termination-reason"
21 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-distant-metastases-category"
22 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-primary-tumor-category"
23 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-regional-nodes-category"
24 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-stage-group"
25 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-pathological-distant-metastases-category"
26 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-pathological-primary-tumor-category"
27 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-pathological-regional-nodes-category"
28 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-pathological-stage-group"
29 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-treatment-intent"
30 = "http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tumor-marker"
Mark Iantorno (Oct 06 2021 at 14:40):
and your test searches for
Mark Iantorno (Oct 06 2021 at 14:40):
"http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-comorbidities-parent",
"http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-disease-status",
"http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-genetic-variant",
"http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-genomics-report",
"http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-patient",
"http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-related-medication-request",
"http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-related-surgical-procedure",
"http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-stage-group"
Mark Iantorno (Oct 06 2021 at 14:41):
so it's missing
Mark Iantorno (Oct 06 2021 at 14:41):
"http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-comorbidities-parent",
Mark Iantorno (Oct 06 2021 at 14:41):
"http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-stage-group"
Mark Iantorno (Oct 06 2021 at 14:41):
"http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-related-medication-request",
Mark Iantorno (Oct 06 2021 at 14:42):
So...the question would be, why does it partially work
Reece Adamson (Oct 06 2021 at 14:42):
but only with Java 8? presumably are present when using Java 11 or 16 since the test passesthose
Mark Iantorno (Oct 06 2021 at 14:42):
so I'm just dumping what I find, still looking into it
Reece Adamson (Oct 06 2021 at 14:43):
appreciate it :pray:
Mark Iantorno (Oct 06 2021 at 14:43):
Now we know why the test is failing
Mark Iantorno (Oct 06 2021 at 14:43):
well, the data that is causing the test to fail
Mark Iantorno (Oct 06 2021 at 14:43):
at least
Mark Iantorno (Oct 06 2021 at 14:44):
Yeah if I comment out those three profiles it passes for sure, let me switch around my java versions and see what's up with 11 and 116
Mark Iantorno (Oct 06 2021 at 14:46):
@Reece Adamson if you are comfortable with it, could you please temporarily grant me admin access on the repo so I can force the CI to run
Mark Iantorno (Oct 06 2021 at 14:47):
not even admin
Mark Iantorno (Oct 06 2021 at 14:47):
contributor or whatever
Mark Iantorno (Oct 06 2021 at 14:48):
I'm running it locally with Java 11 and I'm getting the same failure
Reece Adamson (Oct 06 2021 at 14:49):
That should be fine, lemme just double check with @Robert Scanlon real quick that he's cool with it
Reece Adamson (Oct 06 2021 at 14:49):
I can kick the CI right now in the meantime if you want though
Mark Iantorno (Oct 06 2021 at 14:49):
Please do
Reece Adamson (Oct 06 2021 at 14:50):
rerunning now
Mark Iantorno (Oct 06 2021 at 14:51):
Ah, so, when doing this locally, I need to blow away my fhir cache to see the pass/fail on different versions
Mark Iantorno (Oct 06 2021 at 14:51):
I wonder if that is related to the error...
Reece Adamson (Oct 06 2021 at 14:51):
ah good catch
Mark Iantorno (Oct 06 2021 at 14:51):
Can you change the order they run in
Mark Iantorno (Oct 06 2021 at 14:51):
for my curiosity
Mark Iantorno (Oct 06 2021 at 14:51):
run 16, 11, 8
Reece Adamson (Oct 06 2021 at 14:51):
I can, but they run in separate containers so they shouldn't share cache
Reece Adamson (Oct 06 2021 at 14:51):
I'll do that right now
Mark Iantorno (Oct 06 2021 at 14:51):
oh forget it then
Mark Iantorno (Oct 06 2021 at 14:52):
I'm puzzled at why the version of java would affect the number of profiles that are fetched
Reece Adamson (Oct 06 2021 at 14:52):
if you want to test locally you can see it fail using docker build -t hl7_validator .
. I haven't tried it but I bet if we change the JDK version there we might see a difference
Reece Adamson (Oct 06 2021 at 14:53):
ya it's very odd
Mark Iantorno (Oct 06 2021 at 14:53):
So, I have a meeting now
Mark Iantorno (Oct 06 2021 at 14:53):
but I'm at the point now where I can replicate it locally, and I'll dive into it after my meeting
Mark Iantorno (Oct 06 2021 at 14:53):
My meeting goes until 12, then I will lunch and I'll dedicate hte first part of my afternoon to this
Mark Iantorno (Oct 06 2021 at 14:54):
will have to actually step through the code in the core that fetches the packages
Mark Iantorno (Oct 06 2021 at 14:54):
I'll keep you posted
Reece Adamson (Oct 06 2021 at 14:54):
Thanks! I also have a meeting now but I'll try and do the same and share whatever I find
Mark Iantorno (Oct 06 2021 at 14:54):
I just have to line by line step through IgResponse ig = validator.loadIg("hl7.fhir.us.mcode", null);
Mark Iantorno (Oct 06 2021 at 14:54):
that's where the difference will be
Mark Iantorno (Oct 06 2021 at 14:55):
Yeah...java 11 fetches 43 profile entries
Mark Iantorno (Oct 06 2021 at 14:55):
vs the 30 from java 8
Mark Iantorno (Oct 06 2021 at 14:55):
so strange
Mark Iantorno (Oct 06 2021 at 14:55):
You've tickled my interest here
Mark Iantorno (Oct 06 2021 at 15:41):
So I'm getting the same error on validator version 5.3.14
Mark Iantorno (Oct 06 2021 at 15:41):
as 5.5.4
Reece Adamson (Oct 06 2021 at 17:02):
I made another commit to my PR reverting to 5.3.14 and I see Java 8 passing in the CI when using validator version 5.3.14
Did you wipe your fhir cache between testing?
Mark Iantorno (Oct 06 2021 at 17:26):
yeah I did, but I ran it locally
Mark Iantorno (Oct 06 2021 at 17:27):
truthfully, I think this is related to changes in caching
Mark Iantorno (Oct 06 2021 at 17:27):
why this only affects Java 8
Mark Iantorno (Oct 06 2021 at 17:27):
I am unsure
Mark Iantorno (Oct 06 2021 at 17:27):
Just for testing purposes, could you change the order of the runs
Mark Iantorno (Oct 06 2021 at 17:28):
like 16, 11, 8
Mark Iantorno (Oct 06 2021 at 17:28):
It most likely will not do anything
Mark Iantorno (Oct 06 2021 at 17:28):
but...I just want to rule it out
Reece Adamson (Oct 06 2021 at 17:39):
yup trying it now
Reece Adamson (Oct 06 2021 at 17:44):
same result with reordering the ci builds
Fails with Java 8 on 5.3.14
Passes with Java 8 on 5.5.4
:thinking:
Mark Iantorno (Oct 06 2021 at 18:02):
I am 99.9% sure this has nothing to do with Java
Mark Iantorno (Oct 06 2021 at 18:02):
and everything to do with changes to caching
Mark Iantorno (Oct 06 2021 at 18:06):
So, wait
Mark Iantorno (Oct 06 2021 at 18:07):
you said
Mark Iantorno (Oct 06 2021 at 18:07):
Fails with Java 8 on 5.3.14
Passes with Java 8 on 5.5.4
Mark Iantorno (Oct 06 2021 at 18:07):
but above you said
Mark Iantorno (Oct 06 2021 at 18:07):
I made another commit to my PR reverting to 5.3.14 and I see Java 8 passing in the CI when using validator version 5.3.14
Mark Iantorno (Oct 06 2021 at 18:07):
is it inconsistent
Mark Iantorno (Oct 06 2021 at 18:13):
@Reece Adamson
Reece Adamson (Oct 06 2021 at 18:13):
I mistyped.
Fails with Java 8 on 5.5.4
Passes with Java 8 on 5.3.14
Mark Iantorno (Oct 06 2021 at 18:13):
kk
Mark Iantorno (Oct 06 2021 at 18:14):
that makes sense then
Mark Iantorno (Oct 06 2021 at 18:14):
was worried there
Reece Adamson (Oct 06 2021 at 18:17):
I don't think it has anything to do with my system cache since I have tested locally using docker (docker build -t hl7_validator .
) which doesn't use my system ~/.fhir
cache. Still might have to do with the caching in general though :shrug: .
Mark Iantorno (Oct 06 2021 at 18:21):
I'm just in another meeting from 2-4, will continue to look afterwardss
Reece Adamson (Oct 08 2021 at 16:35):
@Mark Iantorno not sure if you've had a chance to look at this. Would it be helpful to for me to create a simple test case targeting this in a new repo?
Reece Adamson (Oct 14 2021 at 13:35):
@Mark Iantorno I opened a PR on org.hl7.fhir.core to update the pull request CI pipeline to test with multiple JDK versions (right now 8 and 11).
Mark Iantorno (Oct 14 2021 at 21:39):
just gave some feedback on that. Looks like some tests are failing under different versions
Mark Iantorno (Oct 14 2021 at 21:39):
I will have to look into it tomorrow
Mark Iantorno (Oct 14 2021 at 21:39):
thanks for doing this
Peter Jordan (Oct 14 2021 at 21:53):
@Mark Iantorno the main problem to address is that the Validator is returning various bogus information messages with respect to ALL IPS examples when using the -ips parameter which, in turn, uses this profile http://hl7.org/fhir/uv/ips/StructureDefinition/Bundle-uv-ips. A simple fix, at this stage, might be for the -ips parameter to use the cross-border, Bundle Profile - http://hl7.org/fhir/uv/ips/StructureDefinition/Bundle-cross-border-uv-ips. It may take longer for the IPS WG to fix the other Profile and most of the activity now is around the cross-border use case.
BTW validating against the published IPS examples without using a specific profile or IG does not result in the generation of these information messages.
Mark Iantorno (Oct 14 2021 at 22:14):
The main issue is that right now that I need to fix is that we have a validator that doesn't pass tests consistently across versions. From what I looked at, the issue is less to do with the actual validation rules and more with how caching is being done. Now that the pipeline is set up to run the tests, I will look into fixing that first. Thanks for doing that @Reece Adamson
Rob Hausam (Oct 14 2021 at 22:19):
@Peter Jordan @Mark Iantorno We need to get away from relying at all on the cross-border Bundle profile (or the other cross-border profiles), as we agreed that we will not be carrying those forward into the CI build and the updated IG publication we are working on. But it seems very odd that the cross-border Bundle profile would validate correctly where the other IPS Bundle profile doesn't, as the cross-border profile is based on the other one and only adds one main constraint to close the slicing plus adding an additional slice.
Peter Jordan (Oct 14 2021 at 22:29):
OK - I didn't realise that we're moving to single version of the Profiles which is really good news!
These information messages, repeated for all examples on the majority of resource entries (excluding composition and the other other required resources if in certain ordinal positions), suggest it may be an issue with the slicing...
This element does not match any known slice defined in the profile http://hl7.org/fhir/uv/ips/StructureDefinition/Bundle-uv-ips<
Rob Hausam (Oct 14 2021 at 22:36):
Well, we came to that conclusion while you were sleeping yesterday. :)
Yes, at least some of it does have to do with the slicing - but I don't think it's actually because there is anything wrong with the slicing per se. The validator code explicitly allows what we are doing with the slicing (even though some of the wording in the spec does not). I've been getting ready (i.e. marshaling my ammunition) to enter another issue to see if we can resolve that. I think most of these messages are somewhat of a red herring, because when you leverage the "open" aspect of open slicing then you are logically going to "not match any known slice". We can suppress or ignore those, if they are reflecting expected behavior.
Grahame Grieve (Oct 20 2021 at 22:07):
you can let me know if you still think that there's outstanding issues for the validator, but the validator can't fix up the fact that the bundle profiles are not baked in IPS
Grahame Grieve (Oct 20 2021 at 22:09):
@Sumanth Bandaru I never got back to you about your sample IPS. The most obvious problem is here:
"language": {
"coding": [
{
"code": "en-US",
"display": "English (Region=United States)"
}
],
"text": "English (Region=United States)"
}
Grahame Grieve (Oct 20 2021 at 22:09):
you need a system on that code.
Grahame Grieve (Oct 20 2021 at 22:10):
the rest of the issues... kind of depend on getting the IPS bundle profile sort out and then decide whether there's a real problem
Rob Hausam (Oct 20 2021 at 22:13):
The Bundle profiles will be moved from the 'connectathon-2' branch to the CI build really soon. It has kept getting delayed, but I will try to get that and some other key items done tonight.
Grahame Grieve (Oct 21 2021 at 02:31):
the validator with the option -ips is running off the connectathon-2 branch already, so that's not the answer. The bundle profile just doesn't say enough right now
Rob Hausam (Oct 21 2021 at 02:34):
@Grahame Grieve So it doesn't say enough for what? Happy to hear suggestions from you (or anyone else).
Grahame Grieve (Oct 21 2021 at 02:35):
well, last I looked, it only profiles 4 entries, and leaves the rest to 'open', and I got a series of issues reported to me that seemed to all come from the bundle profile, not the validator
Rob Hausam (Oct 21 2021 at 02:39):
I agree with the first part - need to add the other entries for the recommended and optional sections. Not sure about the issues, though. I would like to work through that and see what's going on. I'll see what I'm getting with the updated publisher, and if you want to share what you are seeing (should be the same?) I can look at that, too.
Grahame Grieve (Oct 21 2021 at 02:48):
see this thread
Grahame Grieve (Oct 21 2021 at 02:48):
https://chat.fhir.org/#narrow/stream/207835-IPS/topic/IPS.20Validation.20Issue
Rob Hausam (Oct 21 2021 at 03:00):
Yes, I've been following that. I found the better results that @Peter Jordan was getting with the cross-border profiles to be a bit odd. I'll try to reproduce and see what we need to do.
Mark Iantorno (Nov 09 2021 at 21:17):
@David Otasek
Last updated: Apr 12 2022 at 19:14 UTC