Stream: ig publishing requirements
Topic: -publish-update vs -go-publish
John Moehrke (Jan 11 2022 at 12:39):
what is the difference between the ig builder -publish-update vs -go-publish?
I am noticing that https://confluence.hl7.org/display/FHIR/HL7+Process+for+Publishing+a+FHIR+IG now uses -go-publish, yet when I first created the IHE equivalent process -publish-update was the last step. Which should I use?
Jose Costa Teixeira (Jan 13 2022 at 09:56):
@John Moehrke I'm trying to get the "quick procedure" fixed/adapted to the go-publish option
John Moehrke (Jan 13 2022 at 13:00):
what is the meaning of -publish-update vs -go-publish?
Grahame Grieve (Jan 13 2022 at 19:29):
I don't know what -publish-update is for any more. -go-publish is the only one I support. Have you looked at the documentation I wrote for it?
John Moehrke (Jan 24 2022 at 19:37):
I can switch to -go-publish. where is this documentation that I should read>?
Jens Villadsen (Jan 24 2022 at 21:28):
Probably here: https://confluence.hl7.org/pages/viewpage.action?pageId=81027536#MaintainingaFHIRIGPublication-Runningthebuild
John Moehrke (Jan 24 2022 at 21:43):
yup, have been referring to that.
Jens Villadsen (Jan 24 2022 at 22:47):
@Grahame Grieve - I'm seeing errors running go-publish:
onGenerate:
Exception generating page /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/ncludes/menu.xml for _includes/ncludes/menu.xml in /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages: /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/ncludes/menu.xml (No such file or directory) (00:27.0157)
Exception generating xslt page /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/index.xml for _includes/content/index.xml in /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages: /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/index.xml (No such file or directory) (00:27.0183)
Exception generating xslt page /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/downloads.xml for _includes/content/downloads.xml in /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages: /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/downloads.xml (No such file or directory) (00:27.0198)
Exception generating xslt page /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/background.xml for _includes/content/background.xml in /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages: /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/background.xml (No such file or directory) (00:27.0215)
Exception generating page /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/spec.md for _includes/content/spec.md in /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages: /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/spec.md (No such file or directory) (00:27.0219)
Exception generating xslt page /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/changes.xml for _includes/content/changes.xml in /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages: /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/changes.xml (No such file or directory) (00:27.0234)
Exception generating page /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/p/pages/_includes/artifacts.xml for p/pages/_includes/artifacts.xml in /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages: /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/p/pages/_includes/artifacts.xml (No such file or directory) (00:27.0238)
Generating Summary Outputs (00:27.0242)
Publishing Content Failed: Template based HTML file index.html is missing source file index.xml (00:29.0534)
Jens Villadsen (Jan 24 2022 at 23:01):
And here's another:
ig-release % java -jar publisher.jar -go-publish -source hl7.fhir.dk.core -destination web-root -registry ig-registry/fhir-ig-list.json -history ig-history
Exception publishing: Cannot invoke "String.equals(Object)" because the return value of "org.hl7.fhir.utilities.json.JSONUtil.str(com.google.gson.JsonObject, String)" is null
java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because the return value of "org.hl7.fhir.utilities.json.JSONUtil.str(com.google.gson.JsonObject, String)" is null
at org.hl7.fhir.igtools.publisher.utils.PublicationProcess.publishInner(PublicationProcess.java:140)
at org.hl7.fhir.igtools.publisher.utils.PublicationProcess.publish(PublicationProcess.java:61)
at org.hl7.fhir.igtools.publisher.Publisher.main(Publisher.java:8851)
Full log in /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/publication-process.log
John Moehrke (Jan 25 2022 at 14:06):
I have trouble with go-publish not recognizing my destination {root}. Is it because go-publish presumes there is a /uv folder in that destination?
John Moehrke (Jan 25 2022 at 14:26):
Note, I had previously built IHE process off of this confluence that uses publish-update https://confluence.hl7.org/pages/viewpage.action?pageId=104580055
Jose Costa Teixeira (Jan 25 2022 at 16:06):
@Jens Villadsen this really seems like your IG or your setup
Jose Costa Teixeira (Jan 25 2022 at 16:10):
the go-publish is working fine - almost
Jens Villadsen (Jan 25 2022 at 19:28):
@John Moehrke are you running on windows?
Jens Villadsen (Jan 25 2022 at 19:30):
Jens Villadsen said:
Grahame Grieve - I'm seeing errors running
go-publish:onGenerate: Exception generating page /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/ncludes/menu.xml for _includes/ncludes/menu.xml in /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages: /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/ncludes/menu.xml (No such file or directory) (00:27.0157) Exception generating xslt page /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/index.xml for _includes/content/index.xml in /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages: /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/index.xml (No such file or directory) (00:27.0183) Exception generating xslt page /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/downloads.xml for _includes/content/downloads.xml in /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages: /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/downloads.xml (No such file or directory) (00:27.0198) Exception generating xslt page /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/background.xml for _includes/content/background.xml in /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages: /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/background.xml (No such file or directory) (00:27.0215) Exception generating page /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/spec.md for _includes/content/spec.md in /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages: /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/spec.md (No such file or directory) (00:27.0219) Exception generating xslt page /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/changes.xml for _includes/content/changes.xml in /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages: /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/_includes/content/changes.xml (No such file or directory) (00:27.0234) Exception generating page /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/p/pages/_includes/artifacts.xml for p/pages/_includes/artifacts.xml in /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages: /var/folders/6w/8shtbtss7yjcm2_y9yv2x1hh0000gn/T/hl7.fhir.be.core#0.0.1/temp/pages/p/pages/_includes/artifacts.xml (No such file or directory) (00:27.0238) Generating Summary Outputs (00:27.0242) Publishing Content Failed: Template based HTML file index.html is missing source file index.xml (00:29.0534)
I was hit by this error on two different mac's. Has the go-publish been tested on macOS?
Jose Costa Teixeira (Jan 25 2022 at 19:30):
does this happen also when running _genonce?
Jens Villadsen (Jan 25 2022 at 19:43):
it happens when running java -jar publisher.jar -go-publish ...
Jens Villadsen (Jan 25 2022 at 19:44):
As can be seen, the temp folder has changed - which could very well be the reason. The temp folder used to be in the working dir - at least in 1.1.94 I believe - which is not the case as far as I can tell
Jens Villadsen (Jan 25 2022 at 19:54):
hmmm ... explicitly stating another temp directory made it work
Jose Costa Teixeira (Jan 25 2022 at 20:26):
There was some note a few weeks ago about Mac and temp folders
Jose Costa Teixeira (Jan 25 2022 at 20:28):
It's the Mac curse.. :)
Jens Villadsen (Jan 25 2022 at 20:30):
so if it a curse on mac, why not default to a designated temp folder always?
Jose Costa Teixeira (Jan 25 2022 at 20:33):
I was joking about the Mac curse (or was I? Perhaps the Mac curse is much deeper than a temp folder ...)
John Moehrke (Jan 25 2022 at 20:33):
I use windows, yes.. but I am not getting anywhere with go-publish
John Moehrke (Jan 25 2022 at 20:34):
I have, for now, gone back to publish-updates
Jose Costa Teixeira (Jan 25 2022 at 20:38):
@John Moehrke we can get it to work for ihe in a call - but I would prefer to wait for the fix of the bug I think I found
John Moehrke (Jan 26 2022 at 01:15):
Next time IHE does formal pub will be end of February. So we have time.
John Moehrke (Jan 26 2022 at 01:15):
Still wonder what's wrong with what I'm using today?
Grahame Grieve (Jan 27 2022 at 03:23):
it's some bug I've not figured out on the mac - the default temp folder is mis-reported somewhere somehow, so you have to specify your own temp folder. I've never understood why you get those failures, but you do. A path length thing? Something weird for sure.
Jose Costa Teixeira (Jan 27 2022 at 10:43):
@Grahame Grieve https://github.com/HL7/fhir-ig-publisher/issues/380 I think this needs to be fixed (I can't see how this would work)
Jens Villadsen (Feb 10 2022 at 13:10):
@Grahame Grieve I've been digging into an issue that neither me or @Jose Costa Teixeira understand. We see that packages that depend on eg. DK Core seem to have the wrong Web base. As can be seen on eg. image.png - all references in the MedCom core that points to the DK Core has the /ImplementationGuide/hl7.fhir.dk.core as part of the reference which is wrong - eg. https://hl7.dk/fhir/core/ImplementationGuide/hl7.fhir.dk.core/1.1.0/StructureDefinition-dk-core-patient.html should have been https://hl7.dk/fhir/core/1.1.0/StructureDefinition-dk-core-patient.html. As far as both me and @Jose Costa Teixeira can tell, this is something that happens within the publisher as the string (/ImplementationGuide/hl7.fhir.dk.core) cannot be found within the outcome generated by sushi.
Jens Villadsen (Feb 11 2022 at 18:48):
This is one of those "I scratch your back and you scratch mine", as this bug needs to be sorted out for the affiliates to be able to successfully publish their own IG's.
Jens Villadsen (Feb 11 2022 at 18:49):
Just also to say - this is not a DK specific issue. This is a general issue
Jens Villadsen (Feb 14 2022 at 21:47):
We really could need an extra set of :eyes: on this matter @Grahame Grieve
Grahame Grieve (Feb 14 2022 at 21:59):
it's on my todo list. It blew out last week because the transition to osx was way more traumatic than I imagined
Jens Villadsen (Feb 14 2022 at 22:17):
I'll try to remember to send you some :candy: next christmas so that you can sweeten up the endeavours you end up taking once in a while
Jens Villadsen (Feb 15 2022 at 11:24):
If we could have this sorted out within a month I would be very grateful
Jens Villadsen (Feb 21 2022 at 20:50):
:eyes:
John Moehrke (Feb 21 2022 at 21:42):
I am in need THIS week. It doesn't seem that -publish-update is fully working anymore, like it seemed to do in the past.
John Moehrke (Feb 21 2022 at 21:42):
for example -publish-update does not seem to update the ig-registry/fhir-ig-list.json that I indicate for it to update.
John Moehrke (Feb 21 2022 at 21:48):
what is the proper way to use -go-publish? I tried to guess based on a message in this stream above, but that blewup looking in my destination folder for a zip
Jose Costa Teixeira (Feb 22 2022 at 06:33):
Belgium is about to release soon too
Jose Costa Teixeira (Feb 22 2022 at 06:34):
I can make a workaround for the missing part (the dist-hist folder copy), but since Belgium needs also another thing to be fixed to release, it would be nice to get this all sorted out.
Jens Villadsen (Feb 22 2022 at 15:51):
@Grahame Grieve can we move this a bit further up on your todo-list?
John Moehrke (Feb 23 2022 at 17:02):
I am trying to use -go-publish, but can't quite figure out all the parameters. Can someone provide an example of use of -go-publish that works with a typical IG?
John Moehrke (Feb 23 2022 at 17:02):
specifically, i am not sure what to put in for history
John Moehrke (Feb 23 2022 at 17:05):
it also seems to be looking for ig-build-zips, and I have no idea where that is
Jose Costa Teixeira (Feb 23 2022 at 17:07):
https://github.com/costateixeira/ig-release
Jose Costa Teixeira (Feb 23 2022 at 17:07):
I am convinced that there is an issue with the xxx-hist folders
John Moehrke (Feb 23 2022 at 17:09):
so the destination directory should be empty? (Previous procedure had me copy the history template there, then the output of a build, and the ini files.
Lynn Laakso (Feb 23 2022 at 17:10):
yes, that changed. when I publish an IG now I create the destination directory and a copy of the package-list.json without the new version about to be published (ci-build entry only)
John Moehrke (Feb 23 2022 at 17:11):
does that destination folder have the old version sub-folders?
John Moehrke (Feb 23 2022 at 17:12):
meaning.. the destination folder is empty except for the package-list.json?
Lynn Laakso (Feb 23 2022 at 17:12):
that's how I do it, as with a new IG there are no "old version sub-folders"
John Moehrke (Feb 23 2022 at 17:13):
hmm, Ill give it atry
John Moehrke (Feb 23 2022 at 17:13):
did this new procedure get written somewhere? I keep finding the old one (which mostly works for me).
Lynn Laakso (Feb 23 2022 at 17:14):
I use https://confluence.hl7.org/display/FHIR/HL7+Process+for+Publishing+a+FHIR+IG
John Moehrke (Feb 23 2022 at 17:23):
Lynn Laakso said:
that's how I do it, as with a new IG there are no "old version sub-folders"
but, what about when this is a new release of something that has been published?
Jose Costa Teixeira (Feb 23 2022 at 17:27):
@John Moehrke the repo I sent contains the setup that is explained in the wiki page that Grahame wrote (at least as far as I can understand it). I don't recall well, but I think there was no issue with adding versions
John Moehrke (Feb 23 2022 at 17:33):
It may be obvious to you, but it is not to me. sorry. I have read that page multiple times. Even when I follow it, I get errors that things are missing
John Moehrke (Feb 23 2022 at 17:37):
I think I understand now that the destination folder should be a full copy of the current web site content, all publications. (In IHE this is in a github repo, so not a problem). right?
Jose Costa Teixeira (Feb 23 2022 at 17:41):
in the repo, run the command line there
Jose Costa Teixeira (Feb 23 2022 at 17:42):
sorry this is just for debugging, it's not for documenting.
Jose Costa Teixeira (Feb 23 2022 at 17:42):
You asked for an example, that is one
Jose Costa Teixeira (Feb 23 2022 at 17:43):
John Moehrke said:
I think I understand now that the destination folder should be a full copy of the current web site content, all publications. (In IHE this is in a github repo, so not a problem). right?
if IHE is capturing all the content from all IGs built (i.e. the output folder), then it should be fine, same as FTP.
Jose Costa Teixeira (Feb 23 2022 at 17:44):
of course, a different repo from the IG itself.
Jose Costa Teixeira (Feb 23 2022 at 17:45):
that is also how I see it working. the github for "website" is only edited by updating to add a new release
John Moehrke (Feb 23 2022 at 17:48):
will continue to try. (I ran into a roadblock as the IG I was working on was PDQm which is going through a change of id from uppercase to lowercase and the -go-publish does not like that). So will not try with MHD which does not have that problem.
John Moehrke (Feb 23 2022 at 17:49):
so I do still put the MHD publish.ini in the root of the website folder? Eventhough that folder is common to all publications? Why is go-publish not looking for that in the source?
John Moehrke (Feb 23 2022 at 17:51):
Jose Costa Teixeira said:
You asked for an example, that is one
it did help a bit, but I still had to make assumptions about the various paths... and go-publish seems to want absolute paths, it didn't seem to traverse relative paths (specifically paths with ../ or ./).
John Moehrke (Feb 23 2022 at 17:57):
It seems -publish-update is more reliable and more clear. go-publish seems fragile and imposing publication folder expectations.
Jose Costa Teixeira (Feb 23 2022 at 18:04):
John Moehrke said:
so I do still put the MHD publish.ini in the root of the website folder? Eventhough that folder is common to all publications? Why is go-publish not looking for that in the source?
no, you push the IHE publish.ini
Jose Costa Teixeira (Feb 23 2022 at 18:05):
John Moehrke said:
It seems -publish-update is more reliable and more clear. go-publish seems fragile and imposing publication folder expectations.
I don't think so. I'm 1 or 2 issues away from having a more thorough example.
John Moehrke (Feb 23 2022 at 18:06):
what do you mean push?
Jose Costa Teixeira (Feb 23 2022 at 18:12):
in the repo I sent, the webroot is the publication folder. It will contain the content that is in ihe.net/profiles (?)
Jose Costa Teixeira (Feb 23 2022 at 18:12):
in there, and only there, you have the publish.ini
Grahame Grieve (Feb 23 2022 at 23:35):
to be clear, -publish-update is not supported. -go-publish is the right way to do it. And it is more reliable and robust precisely because it is complaining that not everything lines up how it thinks it should.
John Moehrke (Feb 24 2022 at 00:10):
I am willing to change. I just reporting here the struggles I have had with using it. If it does not work and there is no way for me to understand how to make it work; then it is not working.
Grahame Grieve (Feb 24 2022 at 00:16):
@Jens Villadsen going back to the problem with dk.core references, the problem is in the dk.core package:
Grahame Grieve (Feb 24 2022 at 00:17):
{
"name": "hl7.fhir.dk.core",
"canonical": "http://hl7.dk/fhir/core",
"url": "http://hl7.dk/fhir/core/ImplementationGuide/hl7.fhir.dk.core/1.1.0"
}
here, the package is saying that the web URL for the package is at http://hl7.dk/fhir/core/ImplementationGuide/hl7.fhir.dk.core/1.1.0, so all the references are made relative to that
Jens Villadsen (Feb 24 2022 at 07:04):
That suffix is added by the Publisher AFAIK
Jens Villadsen (Feb 24 2022 at 07:05):
And it shouldnt
Jose Costa Teixeira (Feb 24 2022 at 07:09):
I think it's the implementationguide
Jose Costa Teixeira (Feb 24 2022 at 07:09):
in which case, I think I mislead you, @Jens Villadsen
Jens Villadsen (Feb 24 2022 at 07:11):
@Jose Costa Teixeira so where is it typed?
Jose Costa Teixeira (Feb 24 2022 at 07:11):
I think it was in the IG itself as per discussions on sushi-yaml and url vs canonical
Jose Costa Teixeira (Feb 24 2022 at 07:12):
i just fixed the dk code in my test repo, the issue seems to be gone
Jens Villadsen (Feb 24 2022 at 07:17):
URL for the commit?
Jose Costa Teixeira (Feb 24 2022 at 07:18):
https://github.com/costateixeira/ig-release
Jens Villadsen (Feb 24 2022 at 07:30):
I don't follow. There isn't any of the commits on https://github.com/costateixeira/ig-release/commits/main that refers to the DK core part
Jose Costa Teixeira (Feb 24 2022 at 07:40):
it was ok before. so my local commit was just undoing a wrong commit.
Jose Costa Teixeira (Feb 24 2022 at 07:41):
if you run the publish now, do you get the error? I didn't
Jens Villadsen (Feb 24 2022 at 10:38):
Could you add a readme that states how to run?
Jose Costa Teixeira (Feb 24 2022 at 11:06):
just run the publishdk.cmd
Jose Costa Teixeira (Mar 15 2022 at 07:12):
@Jens Villadsen is the problem with the canonical url mismatch solved for you?
Last updated: Apr 12 2022 at 19:14 UTC
