Stream: IG creation
Topic: Multipart IGs
Elliot Silver (Aug 24 2021 at 02:01):
Is it possible to build an IG in multiple parts, and then combine them into a joint IG as part of the build or publication process?
I'm currently working on an IG that has some large valuesets in it. If I try to build it as one IG, it takes a couple of hours for IG publisher to run (if it doesn't crash from running out of memory) locally. However, if I strip the valuesets out into separate IGs, and make my main IG dependent on the valueset IGs, then I can build all of them in less than an hour. Although this works from the packaging side (anything that uses my skinny IG also brings in the vocab IGs), it means that the vocab doesn't show up in the artefact list, etc. of the main IG. Is there a way to get the faster builds of the multiple IG approach, along with the valueset visibility that comes with the single IG approach?
Grahame Grieve (Aug 24 2021 at 02:09):
no easy way. but the txCache is supposed to mean that this isn't such a big problem
Elliot Silver (Aug 24 2021 at 02:11):
Right -- doesn't cause a problem for using the vocabulary if you know about it; it just means the value set isn't "advertised" anywhere.
John Moehrke (Aug 24 2021 at 12:24):
so you want something more than .dependsOn.
John Moehrke (Aug 24 2021 at 12:24):
so your narrative can easily handle this, so are you just looking for the valueset to show up in the artifacts page?
Lloyd McKenzie (Aug 24 2021 at 13:31):
We've heard the requirement to 'import' artifacts into an IG for a wide variety of reasons. I really think we should re-explore this question.
Jose Costa Teixeira (Aug 24 2021 at 14:40):
Yes please
Jose Costa Teixeira (Aug 24 2021 at 14:40):
https://github.com/HL7/ig-template-base/issues/173 ?
Elliot Silver (Aug 24 2021 at 17:34):
John Moehrke said:
so your narrative can easily handle this, so are you just looking for the valueset to show up in the artifacts page?
Exactly. I'd like it to all appear as one IG despite the fact that I've built it as three for performance reasons.
I don't know if this is possible (well, I do know it isn't currently possible), because as soon as you get away from the hierarchical dependsOn relationship, you need to start checking crossreferences and links, and I think that's what slows everything down.
Alternatively, modifying the IG publisher so that it doesn't slow down as much with large value sets would allow me to use a single IG and solve the problem.
John Moehrke (Aug 24 2021 at 17:49):
yup. all I can think of is someone complaining to their doctor that when they hit their head against a wall that it hurts... and the doctor tells them to stop hitting their head aginst the wall
Grahame Grieve (Aug 24 2021 at 18:12):
how would I reproduce this?
Elliot Silver (Aug 25 2021 at 00:24):
The IGs I've been playing with have been put on https://github.com/ElliotSilver/infoway-tx. (I'm guessing that because I've got 5 IG in one repository, the CI build won't work.)
Grahame Grieve (Aug 25 2021 at 01:28):
right. 1 IG per repo for CI build
Elliot Silver (Aug 25 2021 at 13:54):
Do you need me to refactor it, or can you investigate as is?
Elliot Silver (Aug 25 2021 at 23:35):
@Grahame Grieve did you want me change the github layout?
Grahame Grieve (Aug 26 2021 at 01:59):
it's not my problem; you would have to change it if it was an HL7 IG but it's not. So up to you and your customer if you want CI build or not. I would personally
Elliot Silver (Aug 26 2021 at 02:03):
Well, right now it's just an experiment -- and the only one looking at it, other than me, is you. I suspect that we will restructure if we decide to go ahead, but unless you need it for your tests, I'll leave it for now.
Grahame Grieve (Aug 26 2021 at 02:04):
ok. but which one do I run to reproduce the two problems you want me to look at?
Elliot Silver (Aug 26 2021 at 02:07):
To look at the long build times, compare building "term" to building "tx.1", "tx.2", tx.3" then "tx".
To look at the odd valueset rendering, build "tx". (Which requires you to have already built "tx.1", "tx.2", tx.3", unless you remove the dependencies.)
Grahame Grieve (Aug 26 2021 at 02:15):
running term:
C:\work\org.hl7.fhir.igs\infoway-tx\term\fsh-generated\resources\ImplementationGuide-ca.infoway.tx.json
Grahame Grieve (Aug 26 2021 at 02:16):
Error Parsing File C:\work\org.hl7.fhir.igs\infoway-tx\term\fsh-generated\resources\ImplementationGuide-ca.infoway.tx.json: C:\work\org.hl7.fhir.igs\infoway-tx\term\fsh-generated\resources\ImplementationGuide-ca.infoway.tx.json (The system cannot find the path specified)
Grahame Grieve (Aug 26 2021 at 02:16):
Caused by: java.io.FileNotFoundException: C:\work\org.hl7.fhir.igs\infoway-tx\term\fsh-generated\resources\ImplementationGuide-ca.infoway.tx.json (The system cannot find the path specified)
Elliot Silver (Aug 26 2021 at 02:16):
Arg. Ok, let me see if I can track that down.
Elliot Silver (Aug 26 2021 at 02:23):
IG publisher doesn't appear to like a sushi project that doesn't actually contain any fsh files. Try running sushi manually once to create the IG resource and "fsh" and "fsh-generated" directories.
Grahame Grieve (Aug 26 2021 at 02:26):
run it where? on what?
Elliot Silver (Aug 26 2021 at 02:29):
In the term directory, run sushi without any parameters before you run the publisher.
You'll need to repeat that in the directory for each of the other projects tx, tx.1, tx.2, tx.3 as well.
Grahame Grieve (Aug 26 2021 at 02:41):
10:10 to run the term IG. Is that what you expected?
Elliot Silver (Aug 26 2021 at 02:42):
Not at all. Took me just under 2 hours.
Elliot Silver (Aug 26 2021 at 02:44):
I just kicked off building that about 18 minutes ago, and it is currently on "Validating Conformance Resources".
Grahame Grieve (Aug 26 2021 at 02:45):
how much memory are you giving it? Sounds like it's memory constrained
Grahame Grieve (Aug 26 2021 at 02:45):
Detected Java version: 11.0.9 from C:\Program Files\Java\jdk-11.0.9 on Windows 10/amd64 (64bit). 16344MB available
Elliot Silver (Aug 26 2021 at 02:45):
Detected Java version: 14.0.1 from /Library/Java/JavaVirtualMachines/jdk-14.0.1.jdk/Contents/Home on Mac OS X/x86_64 (64bit). 8192MB available
Elliot Silver (Aug 26 2021 at 02:47):
8GB of physical memory too. Is there any benefit to increasing java to 16GB?
Grahame Grieve (Aug 26 2021 at 02:51):
I don't know
Grahame Grieve (Aug 26 2021 at 02:51):
is it using all that you gave it?
Grahame Grieve (Aug 26 2021 at 02:52):
what does this say at the end?
Grahame Grieve (Aug 26 2021 at 02:52):
Memory (MB): Use = 10261, Free = 810, Total = 11072, Max =16344
Reclaiming memory...
Memory (MB): Use = 2936, Free = 6959, Total = 9896, Max =16344
Elliot Silver (Aug 26 2021 at 02:52):
I'll let you know in a couple of hours.
Rob Hausam (Aug 26 2021 at 02:54):
I've found some cases that definitely can benefit from going to 16GB (or possibly even more, if available - like with UTG).
Grahame Grieve (Aug 26 2021 at 02:54):
UTG definitely needs all the 16GB. Hammers memory
Grahame Grieve (Aug 26 2021 at 02:56):
You have two pretty big value sets
Rob Hausam (Aug 26 2021 at 02:57):
I can build UTG fine with 16GB on my M1 Mac Air, but it wasn't working too well with that on my Intel Mac Pro.
Elliot Silver (Aug 26 2021 at 03:01):
Yeah -- pCLOCD and the Canadian Clinical Drug Data Set.
Grahame Grieve (Aug 26 2021 at 03:20):
which is the value set that renders wrongly?
Elliot Silver (Aug 26 2021 at 03:21):
AcknowledgementCondition
Grahame Grieve (Aug 26 2021 at 03:23):
and what's wrong with it's rendering?
Grahame Grieve (Aug 26 2021 at 03:25):
I think there's some weird things about the value set itself, but the rendering is mostly what I expect
Elliot Silver (Aug 26 2021 at 03:31):
image.png
Should the version be the ValueSet.version value?
Where is the CLD coming from, since there is no compose element, and why does it decide that it is all values from the CodeSystem, when it isn't?
Grahame Grieve (Aug 26 2021 at 03:32):
the value set version is 0.1.0 so that's what renders
Grahame Grieve (Aug 26 2021 at 03:33):
it doesn't say it's all values from the code system. It just says that all the codes are from the code system
Elliot Silver (Aug 26 2021 at 03:36):
I see version 20160311 in the xml.
Grahame Grieve (Aug 26 2021 at 03:37):
Elliot Silver (Aug 26 2021 at 03:42):
Interesting, the source xml for the valueset has a different version.
Grahame Grieve (Aug 26 2021 at 03:43):
you've probably set the business version to overwrite that
Elliot Silver (Aug 26 2021 at 03:45):
I feel the language implies "all of the values that are in the code system X, and here they are" rather than "all of the following values are from X code system".
Grahame Grieve (Aug 26 2021 at 03:46):
you're right about the CLD though. That's a process sequence issue for me.
Grahame Grieve (Aug 26 2021 at 03:46):
well, I adjusted the language
Grahame Grieve (Aug 26 2021 at 03:46):
Elliot Silver (Aug 26 2021 at 03:47):
Thanks, that's clearer to me.
Grahame Grieve (Aug 26 2021 at 03:49):
@Lloyd McKenzie do you want to fix these two issues, or invoke the review process for them
Lloyd McKenzie (Aug 26 2021 at 04:02):
Technically, the wording change wouldn't qualify as a technical correction. It's clarifying and there could be debate as to what the wording ought to be, but obviously something like this should require a pretty light review process. I'll point to it from the other thread as an example of what the process should have a way to address expeditiously.
Lloyd McKenzie (Aug 26 2021 at 04:03):
What's the second issue?
Elliot Silver (Aug 26 2021 at 04:09):
I assume it's the erroneous generation of the CLD.
Grahame Grieve (Aug 26 2021 at 04:09):
At present, when rendering the CLD for a valueset, the rendering works like this:
- if there's a narrative, use that
- if there's a compose, use that
- if there's an expansion, use that
That's not what we want to the ValueSet CLD rendering, particularly because there always is a narrative, since it's been generated already, and it might just be the expansion. I've changed it so that it works like this:
- If there's a narrative that was provided (not generated) use that
- if there's a compose, generate from that
- other wise, use the text "This ValueSet has no definition"
Elliot Silver (Aug 26 2021 at 04:14):
What about the issue of publisher tripping over a project with no fsh files, and only a sushi-config?
Grahame Grieve (Aug 26 2021 at 04:15):
well, that's by design.
Grahame Grieve (Aug 26 2021 at 04:16):
the design is, look for /fsh and if the IG publisher sees it, run Sushi
Elliot Silver (Aug 26 2021 at 05:15):
Grahame Grieve said:
you've probably set the business version to overwrite that
Can you remind me where that flag is?
Elliot Silver (Aug 26 2021 at 05:34):
Nvm, found it -- apply-version. I didn't take any actions to set it, so unless it's defaulting to true, I don't think that's it.
Grahame Grieve (Aug 26 2021 at 05:35):
something is overwriting it. I don't know what else would be
Elliot Silver (Aug 26 2021 at 05:45):
Hmm, somewhere between the fsh-generated/resources version of the IG resource, and the temp/pages version of the IG resource, all of the apply-* parameters get added.
Elliot Silver (Aug 26 2021 at 06:46):
Grahame Grieve said:
what does this say at the end?
Running the "term" IG with 16GB of java heap:
Sending Usage Stats to Server (03:03:36.0228)
Memory (MB): Use = 5005, Free = 11206, Total = 16212, Max =16384
Reclaiming memory...
Memory (MB): Use = 2938, Free = 6917, Total = 9856, Max =16384
...
Finished. Times: loading: 00:21.0841, generate: 03:47:56.0479, narrative generation: 01:35.0256, realm-rules: 00:00.0001, previous-version: 00:00.0000, jekyll: 00:39.0131, validation: 02:33:08.0319 (#29), template: 00:16.0027 (#3) (03:48:39.0543)
context [anonymous] 4:2 attribute signposts isn't defined
context [anonymous] 4:2 attribute signposts isn't defined
Validation output in /Users/elliot/src/infoway-tx/term/output/qa.html
Errors: 180, Warnings: 0, Info: 0, Broken Links = 143 (03:48:57.0454)
Grahame Grieve (Aug 26 2021 at 06:47):
it's hard to read those numbers, but it does feel memory constrained
Elliot Silver (Aug 26 2021 at 06:51):
I don't know how you get to that conclusion, but I'll believe you.
Elliot Silver (Aug 26 2021 at 06:53):
I'll continue investigating tomorrow, starting with the apply-* parameters issue. I suspect it is a sushi thing.
Grahame Grieve (Aug 26 2021 at 07:02):
template thing?
Grahame Grieve (Aug 26 2021 at 07:02):
anyway, try running on a machine with more ram.
Elliot Silver (Aug 26 2021 at 07:07):
It's using an unadulterated fhir.base.template#current template.
Elliot Silver (Aug 26 2021 at 14:37):
Following up...
Building the same IG, but with only 6GB of java heap:
Sending Usage Stats to Server (48:34.0574)
Memory (MB): Use = 3694, Free = 2449, Total = 6144, Max =6144
Reclaiming memory...
Memory (MB): Use = 2927, Free = 3216, Total = 6144, Max =6144
...
Finished. Times: loading: 00:20.0953, generate: 01:18:54.0111, narrative generation: 00:27.0894, realm-rules: 00:00.0000, previous-version: 00:00.0000, jekyll: 00:41.0863, validation: 35:38.0986 (#29), template: 00:05.0609 (#3) (01:20:05.0177)
context [anonymous] 4:2 attribute signposts isn't defined
context [anonymous] 4:2 attribute signposts isn't defined
Validation output in /Users/elliot/src/infoway-tx/term/output/qa.html
Errors: 180, Warnings: 0, Info: 0, Broken Links = 143 (01:20:09.0915)
Interestingly, that's about 3x better performance with less memory.
Elliot Silver (Sep 24 2021 at 07:07):
Returning to a part of this discussion that we kinda skipped over...
Elliot Silver said:
Grahame Grieve said:
you've probably set the business version to overwrite that
Can you remind me where that flag is?
I'm working on a different IG now, and seeing the same behaviour of applying IG settings to the resources -- publisher, version, etc. According to the documentation the default values of IG publisher parameters apply-publisher, apply-version, etc. is false. My IG does not include those as parameters. However it seems that ig-template-base inserts those parameters into the IG during the onLoad processing if not present.
Playing around a bit, if publisher is not set in a profile, but is set in the IG resource, I see the following behaviours:
- apply-publisher not present in IG: IG publisher info present in profile
- apply-publisher set to false in IG: no publisher info present in profile
- apply-publisher set to true in IG: IG publisher info present in profile.
Similarly, if I set a version in a profile, and a version in the IG, I get:
- apply-version not present in IG: IG version present in profile
- apply-version set to false in IG: profile version present in profile
- apply-version set to true in IG: IG version present in profile.
I guess the part here that's non-intuitive is the template setting apply-* to true if it isn't present. (It might also be nice to have a third option: true, false, apply-if-absent-from-resource.)
Grahame Grieve (Sep 24 2021 at 07:33):
well, do you want to document that somewhere?
Elliot Silver (Sep 24 2021 at 07:35):
Is that the expected behaviour? It seems odd that the IG publisher defaults to off, but the standard template defaults to switching it on.
Elliot Silver (Sep 24 2021 at 14:59):
@Lloyd McKenzie , what's the expected behaviour here? This dates back to the earliest checkin of the ig-template-base.
Lloyd McKenzie (Sep 24 2021 at 19:30):
The template does indeed set the apply-* to true if it's not present. So IGs that don't want something applied need to specifically declare it. This choice is because a) most IGs didn't have the apply parameters set; and b) most IGs absolutely wanted (and needed) the apply cascading to happen. This should get documented at some point in the ig-guidance IG.
Elliot Silver (Sep 24 2021 at 20:43):
Grahame Grieve said:
well, do you want to document that somewhere?
Added to https://confluence.hl7.org/display/FHIR/Implementation+Guide+Parameters.
Last updated: Apr 12 2022 at 19:14 UTC
