Stream: implementers
Topic: FHIR IG builds
Rick Geimer (Jul 19 2017 at 22:35):
While Grahame is on leave, is there anyone who can invoke the current build process to get FHIR IGs building from their GitHub repositories?
Rick Geimer (Jul 19 2017 at 22:48):
I need to get the current development build of C-CDA on FHIR showing up here:
http://hl7.org/fhir/us/ccda/history.html
Rick Geimer (Jul 20 2017 at 01:30):
The build is crashing because it is running out of Java heap space. It builds locally when I increase the heap space for the IG publisher, but it seems the server heap space is too small for all the files in the IG.
Lloyd McKenzie (Jul 20 2017 at 01:52):
@Josh Mandel @Grahame Grieve
Josh Mandel (Jul 20 2017 at 01:56):
@Rick Geimer how much does it require?
Josh Mandel (Jul 20 2017 at 01:57):
Also note that the automatic build appear at build.fhir.org, not hl7.org/fhir
Rick Geimer (Jul 20 2017 at 02:22):
Thanks Josh. build.fhir.org is fine. I set Xms2g on my local system with no max. I could play around a bit and find a more exact value.
Rick Geimer (Jul 20 2017 at 02:50):
@Josh Mandel -Xms2g -Xmx2g seems to do the trick, and would be the best short term fix. I tried lowering it to -Xms1g -Xmx1g locally and it ran out of memory. The underlying problem though seems to be that all the files are being parsed as XHTML then stored in a HashMap before being processed. Would be better long term to take a different approach I think.
Josh Mandel (Jul 20 2017 at 03:09):
Thanks for the detailed report! Will update our defaults.
Josh Mandel (Jul 20 2017 at 03:12):
BTW for anyone interested, the build hapless in a docker container configured at https://github.com/hl7-fhir/auto-ig-builder/blob/master/images/ig-build/builder/builder.py#L35-L39
Rick Geimer (Jul 20 2017 at 12:54):
@Josh Mandel Thanks Josh, let me know when the new defaults are set and I'll try again.
Josh Mandel (Jul 20 2017 at 22:05):
I'll kick it off now
Rick Geimer (Jul 20 2017 at 22:29):
Thanks Josh. FYI, what's the URL to actually view the built IG? Thought it would be build.fhir.org/us/ccda but that's 404.
Rick Geimer (Jul 20 2017 at 22:31):
Nevermind: http://build.fhir.org/ig/HL7/ccda-on-fhir/index.html
Grahame Grieve (Jul 25 2017 at 08:00):
so did this get all sorted?
Jayashree Surnar (Aug 03 2017 at 09:08):
hello all,
to implement our own IG what are the steps i need to follow.
Eric Haas (Aug 03 2017 at 14:26):
start by reading this - http://wiki.hl7.org/index.php?title=IG_Publisher_Documentation#Publishing_Implementation_Guides
We are planning to provide a tutorial in IG publishing next year.
Jayashree Surnar (Aug 07 2017 at 04:59):
@Eric Haas in the documentation what is the controle file?is it IG resource? or
Grahame Grieve (Aug 07 2017 at 06:02):
the control file is documented here:
Grahame Grieve (Aug 07 2017 at 06:03):
http://wiki.hl7.org/index.php?title=IG_Publisher_Documentation
Grahame Grieve (Aug 07 2017 at 06:03):
it is not a reosurce
Jayashree Surnar (Aug 08 2017 at 06:31):
- we are implementing based on hl7 and us-core resources.
- do i need to download the IG build (source code) of us core ?or else?
- little bit confusing what things to include in the build
Jayashree Surnar (Aug 08 2017 at 06:32):
(deleted)
Jayashree Surnar (Aug 08 2017 at 06:33):
(deleted)
Jayashree Surnar (Aug 08 2017 at 08:52):
(deleted)
Jayashree Surnar (Aug 08 2017 at 09:12):
(deleted)
Eric Haas (Aug 08 2017 at 14:50):
here is a ig template that does all sort of things. like a sampler. It had a dependency to US -Core as well it uses spreadsheet tooling and some hand edited resources. ( You could use Forge instead to create profiles ) take a look at that and see how the config file - ig.json is set up and how the files are arranged.
Jayashree Surnar (Aug 09 2017 at 06:22):
Thank you so much @Eric Haas . i will have a look at it.
Jayashree Surnar (Aug 09 2017 at 13:06):
@Eric Haas
here ig template that does all sort of things given
There is a python file that will create the ig.json and ig.xnl file based on the content in the resources and example directories. means it will be done automatically, we no need to create the ig.json(control file) and ig.xml(ig resource file) right ?
before seeing this project what i thought was we have to create them.
now somewhat confusion whether i understood right or wrong way. so please can you clarify?
Eric Haas (Aug 09 2017 at 15:16):
the Ig.json file and ig.xml were a pain for me to maintain so I wrote a script to generate them. The script is poorly documented since was for my own work. You should create your files manually at least initially so you can understand how they work with the Ig-pub tooling.
Grahame Grieve (Aug 09 2017 at 23:01):
why were they a pain to maintain?
Eric Haas (Aug 09 2017 at 23:25):
Because I'd add an artifact and forget to manually add to either ig.json and ig.xml to the config and ig resource. It was a housekeeping chore I kept forgetting and led to some build errors that took me forever to uncover. So I made the script to generate a new ig.json and ig.xml each time I added a new resource. This of course leads to a whole new set of bugs and problems.
I don't think the config file is an issue or anything. In fact - I'd rather have a single config file and forget about the IG resource. I know that I'm in the minority there. ) Jekyll handles most of that stuff anyway... and .... I'm starting to add my own params to the config file like my working directory and publisher_url and title etc so that I can for example populate the meta in IG.xml and other resources. Which reminds me --- how hard would it be add publisher and publisher contact to the list of params like so all the conformance resource get populated with them just like is done for jurisdiction.
... and add the canonical urls with {canonical}/{type}/{id}? My script does this but is a one off before the ig-pub runs. I think it would be handy to generate these on the fly so the editor can be lazy and omit the meta from the resources.
Lloyd McKenzie (Aug 10 2017 at 00:59):
I go the other way. I maintain the IG XML and use a transform to generate the config. The former is a resource I can in theory get out of tools like Forge. The latter is something specific to the publication tooling.
Eric Haas (Aug 10 2017 at 01:04):
I go the other way. I maintain the IG XML and use a transform to generate the config. The former is a resource I can in theory get out of tools like Forge. The latter is something specific to the publication tooling.
that is funny ... I actually use a csv file and transform to the config file.
Grahame Grieve (Aug 10 2017 at 01:56):
I just use the config file. I've added support to the IG publisher to automatically scan the resources folders for resources, and then publish them. you provide a template for the output file names (e.g. {type}-{id}.html). And the IG resource is just a shell that the IG can fill out.... so I don't really see the need for all this preprocessing you are doing
Jayashree Surnar (Aug 10 2017 at 10:54):
is it nessessary to keep CapabilityStatement in ig? us-core ig contains 2 capabilitiystatements(client and server)
Grahame Grieve (Aug 10 2017 at 11:21):
only if you want to
Eric Haas (Aug 10 2017 at 14:50):
I just use the config file. I've added support to the IG publisher to automatically scan the resources folders for resources, and then publish them. you provide a template for the output file names (e.g. {type}-{id}.html). And the IG resource is just a shell that the IG can fill out.... so I don't really see the need for all this preprocessing you are doing
Maybe I don't need it then but perhaps we can walk through this and see why I kept tripped up enough to have to create the script and then I can polish up the the documentation so that is clearer how the magic works.
Rick Geimer (Aug 10 2017 at 15:25):
I agree with Lloyd, I prefer the IG. I'd rather see the config file become optional. If the IG contains all the info needed to publish, the tooling should be able to run from it directly (i.e. the config becomes an override mechanism).
Rick Geimer (Aug 10 2017 at 15:27):
And ideally I should be able to point to an IG resource on a FHIR server and have it run fully RESTfully :)
Eric Haas (Aug 10 2017 at 15:57):
The config is needed for publishing on a particular platform ( in this case Jekyll ) The IG resource serves a different purpose of listing the pages and resources so you can publish on a different platform (e..g ???WordPress???). One is for tooling and the other for sharing the contents.
Rick Geimer (Aug 10 2017 at 16:14):
I guess my point is that there should be a way to just point at an ImplementationGuide resource and build a generic no-frills IG (i.e. no customization, just a basic out of the box IG with the current FHIR look and feel, etc.). While I can get the IG process to work, it is overly complicated to the point of being nearly insurmountable for many new users, even when tools like Forge and Trifolia are used. It needs to be much easier to publish a basic IG once you have created your profiles.
David Hay (Aug 10 2017 at 19:33):
From the perspective of a consumer I agree with Rick: regardless of how the IG is built, my interest is in representing (and analysing) the contents...
Eric Haas (Aug 10 2017 at 20:29):
The ig -pub does just that and quite a bit more. I am not sure what Rick is advocating.
Grahame Grieve (Aug 10 2017 at 21:30):
There is already a way to do this. You get a default out of the box publication based on your resources.
Grahame Grieve (Aug 10 2017 at 21:31):
use the jar with these parameters
-source [source] -destination [dest] -tx [url]
Grahame Grieve (Aug 10 2017 at 21:31):
the IG publsiher template used in this case is the one found in the build at /tools/ig
Grahame Grieve (Aug 10 2017 at 21:33):
the IG resource does not - and will not - contain all the things that the IG publisher needs to know
Grahame Grieve (Aug 10 2017 at 21:34):
The complexity is a direct result of all the things IG publishers have wanted the publisher to be able to do
Grahame Grieve (Aug 10 2017 at 21:35):
note, though, that this option uses the International version of SCT, not the US version
Grahame Grieve (Aug 10 2017 at 21:35):
that's what default means
John Moehrke (Aug 11 2017 at 13:22):
It is understood that this is an evolving part of FHIR, right? I am finding many opportunities for improvement given my efforts to represent IHE Profiles (on FHIR) as FHIR conformance resources.
Rick Geimer (Aug 11 2017 at 17:28):
@Grahame Grieve is [source] above an IG resource, or the config file? If it is an IG resource then I withdraw my comment. If it is the config file then I think my comment stands. I understand that there are folks that want all the customization the current process provides, but others would be much happier to just be able to click and run and accept whatever comes out as long as it looks FHIR-ish.
Where you say that "the IG resource does not - and will not - contain all the things that the IG publisher needs to know", how much of that can be defaulted? As Lloyd said, he generates the config from the IG resource anyway, so is it really necessary or can the tooling auto-generate it if missing?
Grahame Grieve (Aug 11 2017 at 20:04):
[source] there is a folder that contains FHIR resources.
Grahame Grieve (Aug 11 2017 at 20:05):
it may contain a FHIR IG.
Lloyd McKenzie (Aug 11 2017 at 22:11):
I hardcover a bunch of stuff in my transform based on how I'd like it to work. However, we could use either extensions or a new "publication configuration" section in the IG resource to handle that information
Grahame Grieve (Aug 11 2017 at 22:28):
we don't want to add tool specific publication details to the IG resource
Grahame Grieve (Aug 11 2017 at 22:29):
I don't really understand much of this discussion. You don't need to say anything much in the IG config file
Grahame Grieve (Aug 11 2017 at 22:29):
unless you don't like the defaults.
Grahame Grieve (Aug 11 2017 at 22:29):
so it seems circular to me
Grahame Grieve (Aug 11 2017 at 23:43):
you must provide a canonicalBase. And you probably want to specify a version
Grahame Grieve (Aug 11 2017 at 23:43):
and you might need to specify dependencies
Grahame Grieve (Aug 11 2017 at 23:44):
I can't default those things (well, I default version to current build, but that's probably not what people want)
Grahame Grieve (Aug 11 2017 at 23:45):
and then you have to tell the IG publisher how you layout the html.
Eric Haas (Aug 12 2017 at 03:35):
use the jar with these parameters
-source [source] -destination [dest] -tx [url]
This is described as "Validating without publishing" in the documentation ... I've tried this a couple of time and it gives me a Null error.
Publishing Content Failed: null (22.0320sec)
(22.0321sec)
Use -? to get command line help (22.0323sec)
(22.0324sec)
Stack Dump (for debugging): (22.0327sec)
java.lang.NullPointerException
at org.hl7.fhir.igtools.publisher.Publisher.copyTemplate(Publisher.java:980)
at org.hl7.fhir.igtools.publisher.Publisher.initialize(Publisher.java:794)
at org.hl7.fhir.igtools.publisher.Publisher.execute(Publisher.java:321)
at org.hl7.fhir.igtools.publisher.Publisher.main(Publisher.java:3853)
Is there an example of it? If I could get it to work I would create one.
Jayashree Surnar (Aug 18 2017 at 08:30):
hi,
cloned the fhir-svn project to my local system and trying to run the publish.bat.
in this project given build.sh , publish.sh publish.bat but no build.bat what is the diff b/w the build process and the publish? or build.sh and publish.sh?
Jayashree Surnar (Aug 18 2017 at 08:43):
when i run the publish.bat getting this error:
Publisher:
[java] Error: Could not create the Java Virtual Machine.
[java] Error: A fatal exception has occurred. Program will exit.
[java] Invalid maximum heap size: -Xmx10000m
[java] The specified size exceeds the maximum representable size.
BUILD FAILED
C:\Users\Gayatri\fhir-svn\build.xml:30: The following error occurred while executing this line:
C:\Users\Gayatri\fhir-svn\tools\java\org.hl7.fhir.tools.core\build.xml:156: Java returned: 1
Total time: 7 seconds
Press any key to continue . . .
Jayashree Surnar (Aug 18 2017 at 08:54):
and what is the use of spredsheet/ bundles in IG i didn't get.
Lloyd McKenzie (Aug 18 2017 at 16:58):
Sounds like the build proces is asking for more memory than your machine has. (It wants 10GB). We might be able to trim that down a little bit, but probably not enought to make a difference for you. (The build holds all of SNOMED, LOINC and a few other code systems in memory among other things...)
Jayashree Surnar (Aug 19 2017 at 04:48):
checkout the code from http://gforge.hl7.org/svn/fhir/trunk/build and trying to build the project, i'm using windows 10. but here there is no build.bat file available what to do? any suggesions please.
after runnig the build.bat should i need to run publish.bat? to publish the ig is it neccessory to follow the build process and the publishing doc? i'm confusing.
Thank you.
Jayashree Surnar (Aug 19 2017 at 07:35):
getting the below error while running the _runant.bat file(https://github.com/HL7/ccda-on-fhir)
ccda-error.PNG
Rick Geimer (Aug 19 2017 at 13:33):
@Jayashree Surnar For C-CDA on FHIR, you need to first download the IG publisher jar file from http://build.fhir.org/org.hl7.fhir.igpublisher.jar and run that using the IG publisher documentation at http://wiki.hl7.org/index.php?title=IG_Publisher_Documentation. For convenience, there is a publish.bat file included with C-CDA on FHIR, so if you put the igpublisher.jar file in the same directory as the publish.bat file and run that bat it should all work.
Lloyd McKenzie (Aug 19 2017 at 13:35):
If you're trying to build the overall spec, you'll want to run trunk/build/publish.bat
Jayashree Surnar (Aug 21 2017 at 04:51):
there is no build folderin any one of the project i cloned.
two ways i cloned the project
1. git clone git url(ex: https://github.com/HL7/ccda-on-fhir.git OR https://github.com/hl7-fhir/fhir-svn.git)
2.in eclipse new-->project-->other-->svn-->checkout project from svn
don't know whether i'm missing somewhere ?
Grahame Grieve (Aug 21 2017 at 04:55):
let's start with a simple question: what are you actually trying to achieve?
Jayashree Surnar (Aug 21 2017 at 05:08):
we want to implement our own IG.
- present we have only profiles and extension with us.i tried to write the ig resource file and control files by following the publish documentation. i'm confusing while specifying them in the resources.
in the daf-research project they written definitions.html,base and source. and in other project it's somewhat diff.i failed to define these things.
so decided 1st to run the existing project then see what are the generated things and what we need to write for our ig.
Jayashree Surnar (Aug 21 2017 at 05:09):
and some of the projects contains the builds.xml too.
Jayashree Surnar (Aug 21 2017 at 05:15):
lets start with 1 example:
i have a profile called vitals pannel and i want to implement our IG with this profile what are the things i need to do?
Eric Haas (Aug 21 2017 at 06:05):
I do these four steps to add a new profile - assuming there are no newly defined valuesets or code systems.
- add your profile (as an xml file) to the resources directory
- add the following the ig.json file (so that it will publish) to:
"StructureDefinition/[Profile-id]": {
"base": "StructureDefinition-[Profile-id].html",
"defns": "StructureDefinition-[Profile-id]-definitions.html",
"source": "[profile-filename].xml"
},
- add the following as a child node to
packagein the ig.xml file (so that the ig publisher will find it):
<resource>
<example value="false"/>
<sourceReference>
<reference value="StructureDefinition/[profile-id]"/>
</sourceReference>
</resource>
- You need to create three new markdown files in the pages/_includes directory as [Profile]-intro.md, [Profile]-summary.md, [Profile]-search.md -- the content doesn't matter right now. This is unique to my profile template and is kind of pain but there are three markdowns files per profile. You can see several examples in the pages/_includes directory us core repo.
Eric Haas (Aug 21 2017 at 06:07):
always works
Jayashree Surnar (Aug 21 2017 at 06:08):
@Eric Haas thank you somuch for your help.
Eric Haas (Aug 21 2017 at 06:10):
the last step depends on whether the sd template needs them. you will know because the publisher will yell at you if they are missing.
Jayashree Surnar (Aug 21 2017 at 06:12):
what is the use of spredsheet or bundles ? some of the projects have.
Jayashree Surnar (Aug 21 2017 at 06:12):
and build.xml?
Eric Haas (Aug 21 2017 at 06:14):
spreadsheets are a profiling tool. so instead of loading xml files in resources you load the profiling spreadsheets.
Eric Haas (Aug 21 2017 at 06:17):
deleted
Grahame Grieve (Aug 21 2017 at 06:18):
bundles are for examples
Jayashree Surnar (Aug 21 2017 at 06:19):
okay.
Jayashree Surnar (Aug 21 2017 at 06:20):
what is difference b/w build process and the publish?
is it nessessary to follow the bothe docs to implement our IG based on US-core and hl7?
Eric Haas (Aug 21 2017 at 06:23):
only need the publisher for an IG. Assuming by build you mean the FHIR specification build - you don't need that for an IG.
Jayashree Surnar (Aug 21 2017 at 06:24):
"StructureDefinition/[Profile-id]": {
"base": "StructureDefinition-[Profile-id].html",
"defns": "StructureDefinition-[Profile-id]-definitions.html",
"source": "[profile-filename].xml"
},
these are the not actual htmls i'm i correct?(i mean ,i no need to creat them,as i have profiles so just need to specify StructureDefinition-[my profile id].html) ?
Jayashree Surnar (Aug 21 2017 at 06:25):
okay, then whats the purpose of that build?
Eric Haas (Aug 21 2017 at 06:25):
The IG- publisher generates this : http://hl7.org/fhir/us/core/ for example
and the build generates this: http://build.fhir.org/
Jayashree Surnar (Aug 21 2017 at 06:28):
okay.
Eric Haas (Aug 21 2017 at 06:29):
"StructureDefinition/[Profile-id]": {
"base": "StructureDefinition-[Profile-id].html",
"defns": "StructureDefinition-[Profile-id]-definitions.html",
"source": "[profile-filename].xml"
},these are the not actual htmls i'm i correct?(i mean ,i no need to creat them,as i have profiles so just need to specify StructureDefinition-[my profile id].html) ?
The IG-Publisher magically creates them. This tells the publisher what their address should be.
Jayashree Surnar (Aug 21 2017 at 06:29):
okay. confused here alot.
thank you.
Eric Haas (Aug 21 2017 at 06:30):
e.g., http://hl7.org/fhir/us/core/StructureDefinition-us-core-careteam.html
and
http://hl7.org/fhir/us/core/StructureDefinition-us-core-careteam-definitions.html
Eric Haas (Aug 21 2017 at 06:31):
the Profile id is 'us-core-careteam'
Jayashree Surnar (Aug 21 2017 at 06:31):
and what baout the cannonical url? is it git url or the fhir server url?
Eric Haas (Aug 21 2017 at 06:31):
as is shown here: http://hl7.org/fhir/us/core/StructureDefinition-us-core-careteam.xml.html
Eric Haas (Aug 21 2017 at 06:42):
that is the final destination for your IG.
Since us core is an HL7 artifact is hosted by hl7
US-Core current release is http://hl7.org/fhir/us/core the content changes when the version changes
US-Core STU1 is http://hl7.org/fhir/us/core/STU1 this content won't ever change
If I were to host an IG on fhir.org then the canonical base would be something like: http://www.fhir.org/guides/my-amazing-ig
Jayashree Surnar (Aug 21 2017 at 06:44):
okay.
Jayashree Surnar (Aug 21 2017 at 06:48):
what about the build.xml?
Eric Haas (Aug 21 2017 at 06:50):
where did you see that?
Jayashree Surnar (Aug 21 2017 at 06:51):
fhir-svn projectbuild.xml other guides also have that.
Eric Haas (Aug 21 2017 at 06:53):
@Lloyd McKenzie ? I think its for preprocessing
Jayashree Surnar (Aug 21 2017 at 06:57):
what about the pages?
Eric Haas (Aug 21 2017 at 07:04):
the pages directory? an IG is basically a static website ( Hence using Jekyll) the pages directory is where the pages you create as html files live. I write everything in markdown and insert them in the html using include syntax so all of the content for my pages live in pages/_include.
Jayashree Surnar (Aug 21 2017 at 07:15):
is it generated based on the pages we specified in the ig resource file?
ex:ig-page.PNG
Eric Haas (Aug 21 2017 at 07:25):
Some pages are autogenerated (list, example), and other kinds are manually created and some are hybrid. they can all be entered in the IG resource of so that tools can navigate the user to the page of interest. ( I personally would rather have a tool to web crawl through the Ig and generate the the IG resource content.)
Grahame Grieve (Aug 21 2017 at 07:30):
a tool could do that, but it would propulate 100s of html pages, with no knowledge of what has significance, and what any structure should be
Eric Haas (Aug 21 2017 at 09:14):
I know I tried one
Jayashree Surnar (Aug 23 2017 at 04:59):
hi,
in IG what are the other things we need to write/create? other than the ig resource file, controle file, profiles,extensions, .mds files,header,footer,nav htmls and using (fhir css)
Jayashree Surnar (Aug 23 2017 at 12:38):
hi everyone,
how much heap memory is needed to run this IG(https://github.com/HL7/US-Core)?
i have given 6GB still getting this error
heap.PNG
Grahame Grieve (Aug 23 2017 at 13:27):
it says 247MB is available - that's not enough. You need to run it with more memory available. Search Java help on line for assistance
Eric Haas (Aug 23 2017 at 16:26):
I never had that issue running an IG on my MacBook Air?? are you running on a virtual instance?
Jayashree Surnar (Aug 24 2017 at 04:54):
hi,
memory details: increased 512m to 2048m still getting same issue.
java-heap.PNG
Jayashree Surnar (Aug 24 2017 at 04:55):
running the ig by keeping only a single resource.
Jayashree Surnar (Aug 31 2017 at 10:33):
hi,
getting the below warning while running the us-core-ig by keeping a single resource.and it's not generating the index.html
us-core-al.ig.PNG
and getting 63 errors if i run as it is.
i'm unable to fined out the warnings.
Lloyd McKenzie (Aug 31 2017 at 14:13):
To see the warnings, look at the qa.html file in your publish folder
Eric Haas (Aug 31 2017 at 15:28):
If you are running the US Core out of the box there are about 2 dozen warnings and a handful of errors. let me know if you have a question about any of these. They could not be resolved at time of publishing.
Jayashree Surnar (Sep 01 2017 at 06:48):
thank you @Lloyd McKenzie , @Eric Haas . i will have a look at it.
Jayashree Surnar (Sep 04 2017 at 09:04):
(deleted)
Jayashree Surnar (Sep 04 2017 at 09:30):
(deleted)
Last updated: Apr 12 2022 at 19:14 UTC
