Stream: committers
Topic: IG Publisher enhancements
Grahame Grieve (Dec 07 2016 at 00:47):
ok, after that round of publishing IGS, there's several areas where we need to change either the IG Publisher itself, or our publishing process. Here's the high level summary:
- need to resolve IG dependencies that very across editors
- need to resolve how FHIR spec reversioning is handled
- when IGs publication requests (NIBs) are approved, we have to approve their canonical URL at that point
- need some advance deadline to provide a structurally complete IG for publishing approval (e.g. templates in place, json file versions & dependencies all filled out) - at least a week before full deadline
- need a clear policy in relationship to broken links; broken links are both significant and hard to resolve, and you cannot expect them to get resolved at the last minute, nor can you ignore them. We need some minimum level of broken links as official policy. Hopefully 0 broken links
Grahame Grieve (Dec 07 2016 at 00:56):
i
Grahame Grieve (Dec 07 2016 at 00:57):
I'll clarify each of these
Grahame Grieve (Dec 07 2016 at 00:58):
and I missed one:
- we need a consistent easy to set up CI build pattern for IGs
Grahame Grieve (Dec 07 2016 at 01:13):
#1: IG dependencies.
in the JSON file where you configure your IG, you can specify that it depends on another IG. Quoting from the CCDA on FHIR IG:
"dependencyList": [{
"name": "USCore",
"location": "http://hl7.org/fhir/us/core/2017Jan",
"source": "C:\\work\\org.hl7.fhir.us\\core\\output"
}],
Grahame Grieve (Dec 07 2016 at 01:14):
there's 3 elements here:
- name: the name used throughout the IG to refer to the IG.
- location: the canonical URL for the implementation guide
Grahame Grieve (Dec 07 2016 at 01:16):
the name is used in one other place - there's a Jekyll variable name fhir.[name] which has the URL of the IG - for referring to it in manually written html or markdown. Not that people are acutally using that (see the next problem)
Grahame Grieve (Dec 07 2016 at 01:17):
the location is used for
- generate links that refer to the content in the dependency IG
- get the validator.pack for the IG
Grahame Grieve (Dec 07 2016 at 01:18):
however you might not want to get the validator.pack from the canonical location of the IG - your IG might be based on a pre-release version of the dependency.
Grahame Grieve (Dec 07 2016 at 01:19):
to get around that, you can nominate a file location where the validator.pack for the dependency IG is
Grahame Grieve (Dec 07 2016 at 01:20):
the question is, how do you get that? So far, editors have done one of 2 things:
- grab a copy manually from the editor of the other spec, and put that into the the version controlled source (say, in \resources as currently recommended in the doco)
or
- set the source property to refer to the direct location of their copy of the IG (the output folder)
Grahame Grieve (Dec 07 2016 at 01:21):
the problem with the first is that the content is not current / maintained, and so discrepencies arise.
the problem with the second is that everyone has their own layout, and no single source folder is possible
Grahame Grieve (Dec 07 2016 at 01:21):
I'm not sure how to resolve this
Rick Geimer (Dec 07 2016 at 05:06):
Grahame, for C-CDA on FHIR, I have opted for now to actually copy the output folder from US Core into ccda2/us-core-output, then change the dependency to just be "us-core-output". This lets me keep a relative URL in the IG JSON file, and a static copy of the output that I can update when I am sure the US Core build is clean. Working for now, not ideal, but better than a machine-specific absolute reference.
Grahame Grieve (Dec 07 2016 at 05:14):
you just need validator.pack, not the rest of the output
Eric Haas (Dec 07 2016 at 21:56):
I don't understand why the second is problem. what do you mean by layout?
Grahame Grieve (Dec 07 2016 at 21:57):
layout?
Eric Haas (Dec 07 2016 at 21:58):
re "- we need a consistent easy to set up CI build pattern for IGs" When it is working Josh's Github hook to s3/lambda is super easy.
Eric Haas (Dec 07 2016 at 21:59):
I wish I was clever enough to set it up on my account
Grahame Grieve (Dec 07 2016 at 21:59):
it might be easy but it's not consistent. I need to work with @Josh Mandel to make this just normal
Eric Haas (Dec 07 2016 at 22:04):
zero bad links is ideal but many are devilish hard to track down. Its not always a matter of clicking on a link and seeing a 404. Those are the easy ones. I'm suspicious that my local system is more forgiving than the qa engine.
Grahame Grieve (Dec 07 2016 at 22:04):
we talked about this: your local system is more forgiving with regard to case. but not otherwise
Eric Haas (Dec 07 2016 at 22:05):
I don't understand what you mean by this "the problem with the second is that everyone has their own layout, and no single source folder is possible"
Grahame Grieve (Dec 07 2016 at 22:05):
but leaving broken links to the end... that's a recipe for trouble. you need to stay on top of them through the process
Grahame Grieve (Dec 07 2016 at 22:05):
I'm wondering whether I should allow the IG to complete if there are broken links
Rick Geimer (Dec 07 2016 at 22:06):
Actually needed definitions.json.zip too, not just validation.pack, but thanks for the hint that's much less bloat than all of the us-core output.
Grahame Grieve (Dec 07 2016 at 22:06):
it shouldn't need both. Definitely not. I'll have to investigate
Eric Haas (Dec 07 2016 at 22:06):
I'm just saying ... not trivial to find some of those links.
Grahame Grieve (Dec 07 2016 at 22:07):
the problem with external dependencies is that say I have the other IG in c:\work\[ig] on my system. For sure, you don't
Eric Haas (Dec 07 2016 at 22:07):
you would think it would be too.
Grahame Grieve (Dec 07 2016 at 22:07):
agree it's far from trivial
Josh Mandel (Dec 07 2016 at 22:12):
Happy to work on the automatic triggering for the publication tool. Do we know what's broken? Can we set up a process to get it updated whenever the way to invoke the publisher changes?
Josh Mandel (Dec 07 2016 at 22:13):
Also let's get this moved to an official FHIR.org AWS account like build.fhir.org
Grahame Grieve (Dec 07 2016 at 22:13):
nothing is broken, we just haven't properly set it up for the IGs.
Grahame Grieve (Dec 07 2016 at 22:13):
can we use the same AWS server?
Josh Mandel (Dec 07 2016 at 22:14):
It runs on lambda
Grahame Grieve (Dec 07 2016 at 22:15):
I have no idea whether this is a yes or a no
Josh Mandel (Dec 07 2016 at 22:15):
We could rewrite a version to run on our own build server, and host an API to trigger it..
Josh Mandel (Dec 07 2016 at 22:15):
But that's more moving parts
Josh Mandel (Dec 07 2016 at 22:17):
It's a "no"
Josh Mandel (Dec 07 2016 at 22:17):
Lambda is Amazon's "functions as a service"
Josh Mandel (Dec 07 2016 at 22:17):
It's an alternative to running code on your own vn
Grahame Grieve (Dec 07 2016 at 22:17):
hang on, what runs on lambda?
Josh Mandel (Dec 07 2016 at 22:18):
The ig publication
Grahame Grieve (Dec 07 2016 at 22:18):
it's not a travis build?
Grahame Grieve (Dec 07 2016 at 22:19):
but I was actually thinking of, can the target of the process be the exsiting web server. not where the process itself runs
Josh Mandel (Dec 07 2016 at 22:21):
Once the process is done building the guide to come up it can do anything it wants to. Publish to S3, copy it to our server, whatever.
Grahame Grieve (Dec 07 2016 at 22:24):
I'd like to standardise that
Grahame Grieve (Dec 07 2016 at 22:25):
http://uscore.build.fhir.org or http://build.fhir.org/uscore
Grahame Grieve (Dec 07 2016 at 22:25):
can you update the lambda code to publish to one of those locations when it completes?
Josh Mandel (Dec 07 2016 at 22:26):
Right. I was thinking it would be ig.fhir.org hosted on S3. But let's decide (separately) two things: 1) whether it is on S3 or our EC2 instance, and 2) what URL we want to expose it at.
Josh Mandel (Dec 07 2016 at 22:26):
Hmm. If we don't include the GitHub org as well as repo, how will we resolve conflicts?
Josh Mandel (Dec 07 2016 at 22:27):
We'd have to keep and update a mapping somewhere
Grahame Grieve (Dec 07 2016 at 22:27):
well, in the interests of minimising the moving parts, I'd like to use to same server
Grahame Grieve (Dec 07 2016 at 22:27):
(and cost)
Grahame Grieve (Dec 07 2016 at 22:27):
don't understand the gitub conflict issue
Josh Mandel (Dec 07 2016 at 22:28):
Sure, but we still have to decide whether *any* GitHub repo can be configured to use the publisher (as it currently works) or whether we have some kind of rules.
Josh Mandel (Dec 07 2016 at 22:28):
I'd prefer a transparent mapping to GitHub org+project names
Grahame Grieve (Dec 07 2016 at 22:29):
so we decided yes or no, what difference would it make? I'm expecting that the answer is: there's a single script that any IG can use, but one of a few of us has to issue a key to enable ot to actually happen?
Eric Haas (Dec 07 2016 at 22:30):
The CI build is down for all five of my IGs. Its weird it runs initially then stops usually on Friday. by stops I means no more logging nothing. So I think there is an issue there that need would need to be fixed.
Grahame Grieve (Dec 07 2016 at 22:30):
also, unless it's build.fhir.org/[ig], I have to set up the DNS
Josh Mandel (Dec 07 2016 at 22:31):
How much effort is adding ig.fhir.org? Also, more to my point: I was saying /org/ig would be a cleaner path
Josh Mandel (Dec 07 2016 at 22:32):
Then any repo could opt in
Josh Mandel (Dec 07 2016 at 22:32):
Where would the "key" live?
Grahame Grieve (Dec 07 2016 at 22:33):
it's about a 5 minute task for me to add to the DNS
Grahame Grieve (Dec 07 2016 at 22:33):
but better if I don't have to. by /org/ig do you mean build.fhir.org/[ig]?
Grahame Grieve (Dec 07 2016 at 22:33):
that's my preference
Josh Mandel (Dec 07 2016 at 22:34):
I'm saying an IG starts at a URL like github.com/ghorg/ghrepo. It should be published to http://ig.fhir.org/ghorg/ghrepo
Grahame Grieve (Dec 07 2016 at 22:36):
I'm ok with that
Lloyd McKenzie (Dec 07 2016 at 22:38):
If you have an external link, that could occasionally get broken intermitently if the external site goes down. You wouldn't want the build to fail in that situation
Grahame Grieve (Dec 07 2016 at 22:39):
why would the build fail?
Lloyd McKenzie (Dec 07 2016 at 22:40):
If we say broken links would stop the ig build from completing, then a temporary inavailability of an external site could stop you from publishing your ig.
Grahame Grieve (Dec 07 2016 at 22:41):
the IG pbulisher does not check the availability of an external site. I have no idea what rules i could use to determine whether any http response represents success or not
Grahame Grieve (Dec 07 2016 at 22:42):
the only external links that are checked are links that refer into the dependency list, and you don't need the site for that, only the validator.pack
Grahame Grieve (Dec 07 2016 at 22:48):
back to how this works - I guess the lamda function has they key to post to the server? So all you have to do is submit a github repository containing a ... ig.... ... a build script....? I'm happy for any output that's been through the current IG Publisher to be published to the web server
Grahame Grieve (Dec 07 2016 at 22:48):
so any one with a public repostitroy can just set it up....
Grahame Grieve (Dec 07 2016 at 22:49):
that would reduce it down to:
- configure your github repo to ping URL XXXX
- bingo, your IG will be published.
yes?
Josh Mandel (Dec 07 2016 at 23:04):
Correct. The instructions would be basically just like https://github.com/jmandel/fhir-ig-builder/blob/master/README.md
Grahame Grieve (Dec 07 2016 at 23:06):
looks good to me. what do we need to do to set it up?
Grahame Grieve (Dec 07 2016 at 23:07):
and what needs to be true about my github repository in order for it to work?
Josh Mandel (Dec 07 2016 at 23:07):
I sent you a private message about setting up an AWS IAM account
Josh Mandel (Dec 13 2016 at 17:00):
OK, the auto-publisher is up and seems to be running fine. Updated instructions at https://github.com/hl7-fhir/auto-ig-builder/blob/master/README.md — the noteworthy bits (especially for @Eric Haas) are:
1. New webhook URL (https://icbe5lqbof.execute-api.us-east-1.amazonaws.com/prod/publish?)
2. New publication URL scheme (http://github.com/test-igs/simple :arrow_right: http://build.fhir.org/ig/test-igs/simple)
Josh Mandel (Dec 13 2016 at 17:01):
Let's see how it breaks :-)
Josh Mandel (Dec 13 2016 at 17:19):
OK, and I'm happy to say that it worked on US Core (which I forked on GitHub to test this). Results at http://build.fhir.org/ig/test-igs/US-Core/
Grahame Grieve (Dec 13 2016 at 20:11):
thanks Josh. Great work
Grahame Grieve (Dec 13 2016 at 20:25):
@Josh Mandel just updating the IG publisher documentation; what happens if the IG build fails?
Grahame Grieve (Dec 13 2016 at 20:35):
I'd like to get this set up for all the IGs now
- US IGs @Eric Haas @Lloyd McKenzie @Rick Geimer @Bryn Rhodes (means moving CCDA/SDC/QICore Ig out to github now
- AU Guides (me)
Lloyd McKenzie (Dec 13 2016 at 20:50):
When does it need to be done?
Grahame Grieve (Dec 13 2016 at 20:57):
soonish. definitely before San Antonio
Josh Mandel (Dec 13 2016 at 21:10):
Or depends on when/how it fails. Is it got this far as running the publisher jar file, it should manage to produce a debug.tgz containing a build log, and publish that. The lambda logs are pretty instructive, though, and I expect we may be referring to them to answer questions from time to time.
Grahame Grieve (Dec 13 2016 at 21:11):
can we set up a fixed file in the IG structure that defines (a) a list of email addresses to notify about outcome and (b) a Zulip channel to notify as well?
Grahame Grieve (Dec 13 2016 at 21:11):
don't hestitate to tell me if I'm asking too much.
Nathan Davis (Dec 13 2016 at 21:18):
I'll work on getting the HSPC IG moved.
Grahame Grieve (Dec 13 2016 at 21:19):
ok thanks
Josh Mandel (Dec 13 2016 at 23:17):
Those features totally make sense. Let's get a bit of experience with this system and then think through how to extend it. Also, if anyone else wants to hack on this project with me... that 'd be cool :-)
Grahame Grieve (Dec 14 2016 at 00:14):
hmm, first problem, @Josh Mandel, the 2 australian guides are in the same github repository, a directory for each
Grahame Grieve (Dec 14 2016 at 00:15):
yur doco doesn't say anything about requirements for IG layout - how do you find the json file?
Josh Mandel (Dec 14 2016 at 01:56):
My guide says you need a file called ig.json :)
Josh Mandel (Dec 14 2016 at 01:56):
The model is one repo per IG, which keep the mapping straightforward
Grahame Grieve (Dec 14 2016 at 06:58):
missed that sentence
Josh Mandel (Dec 14 2016 at 17:35):
OK. I broke this out into a separate step in the instructions :-)
Eric Haas (Dec 14 2016 at 18:23):
works great thanks again Josh ( BTW - Can any Jekyll expert tell me how to get the little tab in Chrome not to display the entire Path? my header has <title>{{[title]}}</title>)
Grahame Grieve (Dec 14 2016 at 20:19):
[title] - what is that?
Eric Haas (Dec 14 2016 at 21:47):
page front matter is empty. so whatever has been defined by the IG builder
Grahame Grieve (Dec 14 2016 at 22:03):
but the IG builder doesn't know about page titles
Eric Haas (Dec 14 2016 at 22:05):
mmm interesting because I get the path to the page. OK I'll add page titles in the front matter. I thought this was a built in feature.
Grahame Grieve (Dec 14 2016 at 22:07):
you get an empty title in the html, and so the browser displays the path instead
Eric Haas (Dec 17 2016 at 01:22):
OK so 3 of my 5 IGs are publishing fine the other 2 ...Nada... no log file nothing....webhook says interaction was successful. IGs are daf-research and Argo-PD...
Josh Mandel (Dec 17 2016 at 01:53):
Thanks @Eric Haas ! Clearly we need better visibility into failures. But manually inspecting, I see:
Building docs 00:50:40 FHIR Implementation Guide Publisher (1.9.0-10581) @ Saturday, December 17, 2016 12:50:40 AM 00:50:40 Load Configuration from /tmp/lambdatemp-dir6875000414454386710/ig.json (00.0000sec) 00:50:40 Root directory: /tmp/lambdatemp-dir6875000414454386710 (00.0073sec) 00:50:40 Publishing Content Failed: Error: folder /tmp/lambdatemp-dir6875000414454386710/examples not found (00.0077sec) 00:50:40 (00.0077sec) 00:50:40 Use -? to get command line help (00.0077sec) 00:50:40 (00.0077sec) 00:50:40 Stack Dump (for debugging): (00.0077sec) 00:50:40 java.lang.Exception: Error: folder /tmp/lambdatemp-dir6875000414454386710/examples not found 00:50:40 at org.hl7.fhir.igtools.publisher.Publisher.checkDir(Publisher.java:925) 00:50:40 at org.hl7.fhir.igtools.publisher.Publisher.initialize(Publisher.java:588) 00:50:40 at org.hl7.fhir.igtools.publisher.Publisher.execute(Publisher.java:256) 00:50:40 at org.hl7.fhir.igtools.publisher.Publisher.main(Publisher.java:3156)
Josh Mandel (Dec 17 2016 at 01:54):
Is there a missing examples dir for daf-research?
Josh Mandel (Dec 17 2016 at 03:32):
OK @Eric Haas there's now public access to build logs at http://ig-build.fhir.org.s3-website-us-east-1.amazonaws.com/logs/:org/:repo (e.g. http://ig-build.fhir.org.s3-website-us-east-1.amazonaws.com/logs/Healthedata1/daf-research). I've documented this in the setup guide.
Eric Haas (Dec 18 2016 at 22:42):
re DAF research and all my IGs there should be an examples dir. I'll check
Eric Haas (Dec 18 2016 at 22:52):
Oh I forgot Git doesn't recognize empty folders..
Grahame Grieve (Dec 19 2016 at 01:03):
I think this is a problem with Git...
Eric Haas (Dec 19 2016 at 01:04):
I added a .keep file to the folders I want.
Josh Mandel (Dec 19 2016 at 01:56):
Yes. .gitignore is convention.
Grahame Grieve (Dec 19 2016 at 02:39):
.gitignore inorder to tell git not to ignore it?
Josh Mandel (Dec 19 2016 at 03:13):
Precisely. What could be more intuitive? https://git.wiki.kernel.org/index.php/GitFaq#Can_I_add_empty_directories.3F
Bryn Rhodes (Dec 20 2016 at 01:03):
Trying to get QI-Core moved out to GitHub but I can't get past some build errors. I'm getting several "Implementation Guide - Error - The link 'XYZ' cannot be resolved". No idea what I'm doing wrong, I've checked against several other IGs and it seems like I'm specifying the resources correctly in the IG.json and IG.xml. Any thoughts on what direction to look next?
Bryn Rhodes (Dec 20 2016 at 01:05):
The latest source for the IG is here: https://github.com/cqframework/qi-core
Eric Haas (Dec 20 2016 at 01:09):
case matters. is a linux server. so check your case that tripped me up several times
Bryn Rhodes (Dec 20 2016 at 02:27):
Yeah, that will probably be an issue once I get it building off GitHub, but I'm just building locally on a Windows box and can't get it to resolve. It's not clear from the documentation what's supposed to be in base, and even if I leave base out, it still won't resolve the resulting link.
Grahame Grieve (Dec 20 2016 at 02:51):
what does 'can't resolve' mean?
Bryn Rhodes (Dec 20 2016 at 02:55):
When I run the publisher, I get build errors like this one:
Bryn Rhodes (Dec 20 2016 at 02:55):
The link 'Basic-basic-adverseevent-example.html' cannot be resolved
Grahame Grieve (Dec 20 2016 at 02:55):
in qa.html? all those chckes *are case sensitive* because you need to get the case correct
Bryn Rhodes (Dec 20 2016 at 02:55):
I can see fragments of that in the _includes directory, but even if I set base to the name of one of those fragments it still won't resolve.
Bryn Rhodes (Dec 20 2016 at 02:55):
Yeah, I've matched case.
Grahame Grieve (Dec 20 2016 at 02:56):
I'm on the phone, will look once I get off
Grahame Grieve (Dec 20 2016 at 04:22):
org.hl7.fhir.exceptions.DefinitionException: error reading display hint: 'dicom='
Grahame Grieve (Dec 20 2016 at 04:30):
there's no page 'Basic-basic-adverseevent-example.html' so you can't refer to it
Grahame Grieve (Dec 20 2016 at 04:31):
more generally, there's a mismatch in your json file
Grahame Grieve (Dec 20 2016 at 04:31):
"StructureDefinition/qicore-allergyintolerance": { "defns": "qicore-allergyintolerance-definitions.html", "base": "qicore-allergyintolerance.html" },
Grahame Grieve (Dec 20 2016 at 04:32):
so you're saying that for the resource "StructureDefinition/qicore-allergyintolerance": { "defns": "qicore-allergyintolerance-definitions.html", "base": "qicore-allergyintolerance.html" },
, you're going to put the output in those 2 files, but you actually generate them in
Grahame Grieve (Dec 20 2016 at 04:33):
no, you don't generate the second one at all
Eric Haas (Dec 20 2016 at 04:58):
here is a snip from your ig.json file...
"Basic/basic-adverseevent-example": { "base": "basic-adverseevent-example.xml", "template-base": null },
Eric Haas (Dec 20 2016 at 04:59):
These are Jekyll errors I'm assuming. All the base should be the published html files. e,g "MedicationAdministration/medadmin-1": {
"base": "MedicationAdministration-medadmin-1.html"
}
Eric Haas (Dec 20 2016 at 05:00):
not xml file - those are generated by the IG for each base file by the publisher
Bryn Rhodes (Dec 20 2016 at 05:01):
So base should be "Basic-basic-adverseevent-example.html", but I've tried it with that and I still get the link error.
Eric Haas (Dec 20 2016 at 05:02):
'basic-adverseevent-example.html' is the link
Eric Haas (Dec 20 2016 at 05:03):
not Basic-basic-...
Bryn Rhodes (Dec 20 2016 at 05:03):
Hmmm... in your example the link is constructed as "<resource type>-<resource id>.html", so I was just following that pattern.
Eric Haas (Dec 20 2016 at 05:04):
OK let me download and run. you have the latest ig-publisher right?
Bryn Rhodes (Dec 20 2016 at 05:04):
Right, what's there now is definitely wrong.
Grahame Grieve (Dec 20 2016 at 05:05):
the problem for me is I don't know what you are trying to do. How do you want the IG laid out?
Bryn Rhodes (Dec 20 2016 at 05:06):
The same way that it was when it built under the FHIR tooling.
Bryn Rhodes (Dec 20 2016 at 05:06):
That's what I was going for anyway.
Bryn Rhodes (Dec 20 2016 at 05:08):
And the thing is, I haven't changed anything other than cleaning up references to the new us-core instead of daf from when it was building for the September ballot.
Bryn Rhodes (Dec 20 2016 at 05:08):
http://hl7.org/FHIR/us/qicore/2016Sep/qicore-allergyintolerance.html
Bryn Rhodes (Dec 20 2016 at 05:08):
So like that page.
Bryn Rhodes (Dec 20 2016 at 05:09):
I suppose the IG source was always wrong, it's just that the tooling is tighter now so it's exposing all the ways the source was wrong :)
Grahame Grieve (Dec 20 2016 at 05:10):
possibly. I'll look at it shortly
Grahame Grieve (Dec 20 2016 at 05:13):
can you make me a committer? you haven't followed the rules in how you created the respository....
Eric Haas (Dec 20 2016 at 05:14):
it builds for me. you have a few thousand broken links is that the problem?
Bryn Rhodes (Dec 20 2016 at 05:16):
@Grahame Grieve, just invited you
Bryn Rhodes (Dec 20 2016 at 05:16):
Eric no, it doesn't build for me, the output I get is that I have 75 Build Errors and they're all that it cannot resolve the base page links.
Grahame Grieve (Dec 20 2016 at 05:17):
thx. I also get 6500+ broken links
Grahame Grieve (Dec 20 2016 at 05:17):
what broken links are you getting?
Grahame Grieve (Dec 20 2016 at 05:17):
where are you seeing them?
Bryn Rhodes (Dec 20 2016 at 05:18):
Jekyll: done in 21.668 seconds. (00:02:29.0630sec)
Checking Output HTML (00:02:29.0844sec)
... 1 html file, 0 pages invalid xhtml (0%) (00:02:29.0847sec)
... 75 links, 75 broken links (100%) (00:02:29.0847sec)
Bryn Rhodes (Dec 20 2016 at 05:18):
That's the output I get from the builder
Bryn Rhodes (Dec 20 2016 at 05:18):
They're in the qa.html, want me to commit that output?
Grahame Grieve (Dec 20 2016 at 05:18):
1 html file? weird.
Bryn Rhodes (Dec 20 2016 at 05:18):
Note that I'm building with the publisher that is included in the 1.8 candidate, is that the problem?
Grahame Grieve (Dec 20 2016 at 05:18):
yes use the 1.9 publisher.
Grahame Grieve (Dec 20 2016 at 05:19):
what's in the temp directory?
Bryn Rhodes (Dec 20 2016 at 05:19):
Temp has what looks like all the right stuff, but it never puts anything in output but the qa validation pages.
Grahame Grieve (Dec 20 2016 at 05:19):
Jekyll is failing.
Grahame Grieve (Dec 20 2016 at 05:20):
I wonder why
Grahame Grieve (Dec 20 2016 at 05:20):
can you search your whole system for one of the files that should be produced? Jekyll is taking 20 seconds to do nothing.... smells
Bryn Rhodes (Dec 20 2016 at 05:22):
Searching...
Bryn Rhodes (Dec 20 2016 at 05:22):
Using the 1.9 publisher now I'm getting the same errors you're seeing.
Bryn Rhodes (Dec 20 2016 at 05:23):
But now it's loading the 1.9 fhir definitions.
Grahame Grieve (Dec 20 2016 at 05:23):
what OS you running? and what path, and and does Jekyll produce any output?
Bryn Rhodes (Dec 20 2016 at 05:25):
Windows.
Bryn Rhodes (Dec 20 2016 at 05:25):
And I get more errors, but I still get the 1 html file with 75 broken links output, so nothing gets produced.
Grahame Grieve (Dec 20 2016 at 05:25):
no. once I realised Jekyll is failing, upgrading to the latest publisher isn't it
Bryn Rhodes (Dec 20 2016 at 05:26):
Hmmm.... I have a space in my path.
Grahame Grieve (Dec 20 2016 at 05:26):
try without.
Bryn Rhodes (Dec 20 2016 at 05:32):
Yep, that was it, something about the process doesn't like spaces in the path.
Bryn Rhodes (Dec 20 2016 at 05:32):
Now I get 6600 broken links.
Grahame Grieve (Dec 20 2016 at 05:33):
hmm ok. it will be in my invocation pathway. I'll try to fix it
Bryn Rhodes (Dec 20 2016 at 05:34):
Thanks for helping me work through that @Grahame and @Eric Haas
Bryn Rhodes (Dec 20 2016 at 05:34):
I'll get it building with the webhook now.
Brett Marquard (Jan 03 2017 at 15:31):
I am way late to party -- but space in path caused problems for me previously. Added to wiki troubleshooting tip IG publisher troubleshooting
Lloyd McKenzie (Apr 02 2017 at 06:42):
I've made some changes to how the IG Publisher works. None of the changes *should* have any impact on existing IGs. However, you may wish to re-run yours, just to make sure.
Eric Haas (Apr 03 2017 at 22:08):
Does the IG-publisher already generate a site map? ( Browsing through the output and temp files - I did not see any but thought I'd ask anyway )
Grahame Grieve (Apr 03 2017 at 22:19):
I considered this, but don't know how to
Grahame Grieve (Apr 03 2017 at 22:19):
@Lloyd McKenzie generates some things, but based on input not available to the generator itself
Eric Haas (Apr 03 2017 at 22:23):
I noticed a jekyll plugin. https://github.com/jekyll/jekyll-sitemap but I haven't looked at it closely. That is why I thought maybe already done. My IGs are ridiculously simple (2-3 layers) but a DAF-R comment made me consider it.
Lloyd McKenzie (Apr 03 2017 at 22:47):
The IG-publisher spits out a toc.xml fragment based on the page hierarchy in the IG
Lloyd McKenzie (Apr 03 2017 at 22:48):
It also spits out a data file that lists all of the pages and the section numbers and breadcrumbs for each page
Eric Haas (Apr 07 2017 at 14:35):
where do the titles for examples come from? I only get "??" are they defined in ig.xml or shall make a task to define it in ig.json?
Grahame Grieve (Apr 07 2017 at 20:55):
I'll have to look - will do sometime in the next 48 hours
Eric Haas (Apr 10 2017 at 18:41):
The hamburger menu is still broken. (When you have a small format and click on the hamburger menu - nothing happens. - its is supposed to expand and show navbar) I looked into it and it seems to be a bootstrap issue and the scss stuff, but that is as far as I could figure out.
Grahame Grieve (Apr 10 2017 at 22:32):
it's not actually an issue with the IG publisher?
Eric Haas (Apr 10 2017 at 23:11):
It is an issue how the publisher uses Jekyll and bootstrap. Thats the part I get lost on.
Rick Geimer (Apr 11 2017 at 13:27):
Regression: There is a bug in the IG publisher when trying to publish IGs that depend on other IGs that was not there before. When I try to publish ccda-on-fhir, which depends on US core, I get the following error:
Unable to Parse HTML - starts with 'null::table' not 'div'
The JSON looks like this:
"div": "<table border=\"0\" xmlns=\"http://www.w3.org/1999/xhtml\" ...
When I change it to look like this (i.e. add a div element that is not in any namespace) it works:
"div": "<div><table border=\"0\" xmlns=\"http://www.w3.org/1999/xhtml\"
Rick Geimer (Apr 11 2017 at 13:49):
The bug seems to be in JsonParserBase, line 185. Here is the relevant method:
protected XhtmlNode parseXhtml(String value) throws IOException, FHIRFormatError {
XhtmlParser prsr = new XhtmlParser();
try {
return prsr.parse(value, "div").getChildNodes().get(0);
} catch (org.hl7.fhir.exceptions.FHIRFormatError e) {
throw new FHIRFormatError(e.getMessage() + " Offending string: " + value, e);
}
}
For now, just hacking the output of us-core in case other stuff depends on the existing behavior of parseXhtml, but this will need looking into.
Rick Geimer (Apr 11 2017 at 14:08):
Done hacking US core, now getting this error:
java.lang.Exception: Unable to determine type for C:\Users\rickg\git\ccda-on-fhir\resources\structuredefinition\CCDA-on-FHIR-US-Realm-Header: Not a JSON Object: null
Did we remove support for XML StructureDefinitions?
Lloyd McKenzie (Apr 11 2017 at 14:47):
The div element has always been expected to start with <div....> (and you should be able to have the namespace present too). So that's not a bug, that's detecting something that was previously broken.
Eric Haas (Apr 11 2017 at 15:05):
@Rick Geimer The IG publisher Support US Core for me. check out my IG-Template I have both a source location and a reference...
Eric Haas (Apr 11 2017 at 15:06):
'''
"dependencyList": [
{
"source": "resources",
"name": "uscore",
"location": "http://hl7.org/fhir/us/core/"
}
],
'''
Eric Haas (Apr 11 2017 at 15:10):
I think I never tried it without both source and location though...
Eric Haas (Apr 11 2017 at 15:12):
btw the CI build definitions.json.zip is in my resources directory
Rick Geimer (Apr 11 2017 at 15:36):
@Lloyd McKenzie Ok, then why does US Core build successfully in such a way that it does not start with div?
Rick Geimer (Apr 11 2017 at 15:36):
@Eric Haas will try that in my ig.json file and see if it works.
Rick Geimer (Apr 11 2017 at 15:45):
@Eric Haas That got me past the div error. Still getting the Not a JSON object error since the C-CDA on FHIR StructureDefinition files are XML not JSON. Locally hacking Publisher.java and FetchedFile.java to see if I can find a workaround.
Lloyd McKenzie (Apr 11 2017 at 15:46):
@Rick Geimer I don't know . . . @Grahame Grieve ?
Rick Geimer (Apr 11 2017 at 15:51):
@Lloyd McKenzie thanks. I was getting the error when referencing the definitions.json.zip file locally. When I used the method suggested by @Eric Haas that error went away. Still having issues with XML StructureDefinition resources though. Will see if my latest hack gets me any further.
Rick Geimer (Apr 11 2017 at 16:55):
Still having issues with XML StructureDefinitions. Need to go to the HL7 Value-Based Care mini-connectathon now. Checking in some changes to help debug the XML StructureDefinition problems. Will continue my investigation when I get to the event.
Eric Haas (Apr 13 2017 at 01:37):
So there is an html parsing error when have a dependency *AND you have ig build generated narrative*:
java.lang.Exception: Unable to parse StructureDefinition-us-core-procedure.json from IG uscore
at org.hl7.fhir.igtools.publisher.Publisher.loadIg(Publisher.java:997)
at org.hl7.fhir.igtools.publisher.Publisher.initialize(Publisher.java:787)
at org.hl7.fhir.igtools.publisher.Publisher.execute(Publisher.java:299)
at org.hl7.fhir.igtools.publisher.Publisher.main(Publisher.java:3583)
Caused by: org.hl7.fhir.exceptions.FHIRFormatError: Unable to Parse HTML - starts with 'null::table' not 'div' at line 1 column 7 Offending string: <table border="0" xmlns="http://www.w3.org/1999/xhtml" ...
You have this error with both a local source file or the locationL
here is what the narrative looks like in json:
"div":"<table border=\"0\" xmlns=\"http://www.w3.org/1999/xhtml\" cellpadding=\"0\" cellspacing.....
You don't get this error when the narrative in json looks like this - so either need to add the div element back in or modify the parser.
"div":"<div xmlns=\"http://www.w3.org/1999/xhtml\"><table border=\"0\" cellpadding=\"0\" cellspacing...
Lloyd McKenzie (Apr 13 2017 at 04:41):
The spec is quite clear that the string must start with "<div....". "<table..." is not legal
Lloyd McKenzie (Apr 13 2017 at 04:42):
So "fixing" the parser would not be appropriate in this situation. The parser is complaining appropriately.
Eric Haas (Apr 13 2017 at 14:41):
So the Ig-pub narrative generator is guilty.
Rick Geimer (Apr 13 2017 at 14:48):
Looks that way.
Lloyd McKenzie (Apr 13 2017 at 14:56):
Indeed. Committed a fix, but don't have an IG to test with - do you want to update and give it a try?
Eric Haas (Apr 13 2017 at 15:01):
OK lemme give it a spin
Eric Haas (Apr 13 2017 at 15:03):
On another topic... There is an issue with spreadsheet generated valuesets as illustrated in the qa.html report.....
Eric Haas (Apr 13 2017 at 15:04):
Screen-Shot-2017-04-13-at-8.02.03-AM.png
Eric Haas (Apr 13 2017 at 15:07):
Options are:
- fix it
- document that it doesn't work and to create valueset and codesystems some other way..
Lloyd McKenzie (Apr 13 2017 at 15:11):
There's enough information in the spreadsheet or generation context to populate version and publisher, but I'm not sure how we'd populate experimental without adding an additional column to the spreadsheet. I know for SDC I just did all of the value sets as XML
Eric Haas (Apr 13 2017 at 15:21):
I did all mine as XML too except for this one and was trying out the spreadsheet since is easier and made sense for the demo ig.
Eric Haas (Apr 13 2017 at 15:51):
OK so Lloyd fixed the xhtml and is working locally. I committed the change to the CI build and then this dependency ought to work... @Rick Geimer
Eric Haas (Apr 13 2017 at 15:52):
"dependencyList":[{"name":"uscore","location":"http://build.fhir.org/ig/Healthedata1/US-Core"}]
Eric Haas (Apr 13 2017 at 15:53):
GF#13204 to update US-Core with technical correction too then
Eric Haas (Apr 13 2017 at 15:54):
"dependencyList":[{"name":"uscore","location":"http://hl7.org/fhir/us/core/"}]
should work as well
Eric Haas (Apr 13 2017 at 16:05):
It works!! (Don't forget to clear your cache first.)
Eric Haas (Apr 25 2017 at 19:46):
ig-publisher + spreadsheet support logical models? logical - model putting 'logical' in type causes an error?
Grahame Grieve (Apr 25 2017 at 21:11):
yes, it is supported - look at the AU RCPA project to see how it's done
Eric Haas (Apr 25 2017 at 21:11):
ty
Eric Haas (Apr 25 2017 at 23:58):
I will update in my ig-Template a make a note to try to document in the wiki. It was a bit a steep learning curve. Messed up my definitions script too. :-P
Eric Haas (Apr 25 2017 at 23:59):
I noticed the spreadsheet was an ig spreadsheet which I used. Does this matter or can I used a profile spreadsheet.
Eric Haas (Jun 03 2017 at 01:10):
When generating examples, I am assuming there is no way to list multiple examples in a cell under the "Example" column in the spreadsheet and have them come out the other end as a list. I tried several ways (";", "," and"\n").
David McKillop (Jun 03 2017 at 02:55):
@Eric Haas FYI - @Grahame Grieve set up the structured cancer protocol's in spreadsheets, an example could be created per column and there were 10 examples with one protocol. I don't know about having >1 example per cell, but @Grahame Grieve would be best to advise on this topic.
Grahame Grieve (Jun 04 2017 at 04:10):
you can have mulitple examples - see how it's done in the spreadsheet in the RCPA specification
Eric Haas (Jun 04 2017 at 22:32):
@Grahame Grieve I've copied what you did using the spreadsheet here as examples. they don't come the other end as repeats however.
notice here in the rcpa examples:
Grahame Grieve (Jun 04 2017 at 22:42):
I'm not sure what I'm supposed to see in the image? Is this all committed somewhere?
Eric Haas (Jun 04 2017 at 22:52):
the element noLymphNodePerCassette is an array with one value with a three line feeds. I assume that is supposed to be three separate values. That is what assume you referred to above when you said look at the RCPA spec. If not, then what exactly should I be looking at?
Eric Haas (Jun 10 2017 at 22:40):
I get this error from the ig-builder...
java.lang.Exception: exception parsing pack /Users/ehaas/Documents/FHIR/Case-Reporting/resources/eicr-ccda-fhir-spreadsheet: Unknown Type Narrative
ButNarrative is in the data type list? Is this a bug?
Grahame Grieve (Jun 10 2017 at 22:41):
where do you get this?
Eric Haas (Jun 10 2017 at 22:44):
i am trying to generate examples of a profile I am working on and so I declare the type Narrativefor Composition. section.type here is the complete stack...
Eric Haas (Jun 10 2017 at 22:44):
java.lang.Exception: exception parsing pack /Users/ehaas/Documents/FHIR/Case-Reporting/resources/eicr-ccda-fhir-spreadsheet: Unknown Type Narrative at
org.hl7.fhir.igtools.spreadsheets.IgSpreadsheetParser.parse(IgSpreadsheetParser.java:204)
at org.hl7.fhir.igtools.publisher.Publisher.loadSpreadsheet(Publisher.java:1483)
at org.hl7.fhir.igtools.publisher.Publisher.loadSpreadsheets(Publisher.java:1468)
at org.hl7.fhir.igtools.publisher.Publisher.load(Publisher.java:1251)
at org.hl7.fhir.igtools.publisher.Publisher.execute(Publisher.java:315)
at org.hl7.fhir.igtools.publisher.Publisher.main(Publisher.java:3763)
Caused by: org.hl7.fhir.exceptions.FHIRFormatError: Unknown Type Narrative
at org.hl7.fhir.r4.formats.JsonParser.parseType(JsonParser.java:19908)
at org.hl7.fhir.r4.formats.JsonParserBase.parseType(JsonParserBase.java:105)
at org.hl7.fhir.r4.formats.ParserBase.parseType(ParserBase.java:67)
at org.hl7.fhir.r4.formats.ParserBase.parseType(ParserBase.java:62)
at org.hl7.fhir.igtools.spreadsheets.IgSpreadsheetParser.processValue(IgSpreadsheetParser.java:905)
at org.hl7.fhir.igtools.spreadsheets.IgSpreadsheetParser.processLine(IgSpreadsheetParser.java:802)
at org.hl7.fhir.igtools.spreadsheets.IgSpreadsheetParser.parseProfileSheet(IgSpreadsheetParser.java:268)
at org.hl7.fhir.igtools.spreadsheets.IgSpreadsheetParser.parse(IgSpreadsheetParser.java:178)
Grahame Grieve (Jun 10 2017 at 22:44):
what version?
Eric Haas (Jun 10 2017 at 22:45):
2 days ago... I was just about to say that I should download the latest and try again....
Grahame Grieve (Jun 10 2017 at 22:46):
what version is the IG for?
Eric Haas (Jun 10 2017 at 22:48):
R3
Eric Haas (Jun 10 2017 at 22:50):
just uploaded the latest publisher still not working. I'll just unroll the element and do it that way.
Grahame Grieve (Jun 10 2017 at 22:52):
umm, I'd rather debug it....
Eric Haas (Jun 10 2017 at 22:53):
OK. so do you want me to commit it? so you can see?
Grahame Grieve (Jun 10 2017 at 22:55):
yes, though I won't be able to investigate until tonight.
Grahame Grieve (Jun 10 2017 at 22:55):
if you commit, I can sync, and then you can work around it
Grahame Grieve (Jun 10 2017 at 22:55):
if you want
Eric Haas (Jun 10 2017 at 22:56):
I can work around it locallly. I just committed.
Eric Haas (Jun 10 2017 at 22:57):
it's here https://github.com/Healthedata1/case-reporting
Eric Haas (Jun 10 2017 at 22:58):
eicr-ccda-fhir-spreadsheet.xml
Grahame Grieve (Jun 10 2017 at 23:08):
got it, thanks
Grahame Grieve (Jun 13 2017 at 02:14):
ok fixed. you can use narrative now
Eric Haas (Jun 13 2017 at 16:06):
ty
Eric Haas (Jun 20 2017 at 21:00):
I believe this is a bug:
type-id (combination) = {{[uid]}} when used in a template --> OperationDefinition=Appointment-find2
notice the '=' instead of a '-'
Grahame Grieve (Jun 20 2017 at 21:18):
where is this bug?
Eric Haas (Jun 20 2017 at 21:20):
when using {{[uid]}} in templates for Ig-build I post an example soon
Grahame Grieve (Jun 20 2017 at 21:25):
ok
Eric Haas (Jun 20 2017 at 23:15):
here is an example of the faulty variable located in this repo
Eric Haas (Jun 20 2017 at 23:15):
Type-id (combination) { {[uid]}} = StructureDefinition=template-basic
Grahame Grieve (Jun 21 2017 at 03:48):
I can't check that repository out:
Grahame Grieve (Jun 21 2017 at 03:49):
Clone failed: cannot check out to illegal path c:\temp\test.map
Grahame Grieve (Jun 21 2017 at 03:51):
or 'Cloning into 'C:\work\org.hl7.fhir.us\ig-template'...
error: Untracked working tree file 'c:\temp\test.map' would be overwritten by merge.'
Grahame Grieve (Jun 21 2017 at 03:53):
@Josh Mandel google gives me no help with this error. any ideas?
Grahame Grieve (Jun 21 2017 at 03:58):
it's because in commit 72300d470948f79d53a3e649cd8ef59e3dad4661, Eric committed a file that it outside the repository. Some git clients just go ahead and process that like a file they have a right to access, whereas others won't do it at all. Seems like a big security issue to me
Josh Mandel (Jun 21 2017 at 11:40):
In general, you get that message when you have a file locally that isn't managed by git, and you're trying to merge a new commit that introduces this same file.
Josh Mandel (Jun 21 2017 at 11:41):
Here from what you're showing, the file is actually outside of the git repo root!?
Josh Mandel (Jun 21 2017 at 11:42):
Here's the thing..
Josh Mandel (Jun 21 2017 at 11:42):
What Eric checked in is https://github.com/Healthedata1/IG-Template/blob/72300d470948f79d53a3e649cd8ef59e3dad4661/c:%5Ctemp%5Ctest.map
Josh Mandel (Jun 21 2017 at 11:44):
It's not a file outside of the git repo root. It's a file at the repo root whose filename happens to be "c:\temp\test.map"
Josh Mandel (Jun 21 2017 at 11:56):
Now this is certainly quite suggestive of a security bug in the git client you're using. I wonder if it's just a spurious warning or if it really would write a file at an arbitrary path. Can you share the details of your setup, and what git client gave you this message? And if you're willing, maybe backup your current file, do the merge, and see if it really does write to your c:\temp?
Grahame Grieve (Jun 21 2017 at 12:37):
GitKracken: error when encoutnering the file.
SourceTree: error because the file already existed. So it's insecure.
Let's assume Eric is using OSX - the file is one I created in the IG publisher. I thought that the name c:\temp\test.map would be illegal in OSX - weird. But there's something more wrong thna just with SourceTree.
Grahame Grieve (Jun 21 2017 at 12:37):
https://jira.atlassian.com/projects/SRCTREEWIN/issues/SRCTREEWIN-7499?filter=allopenissues
Grahame Grieve (Jun 21 2017 at 12:41):
nowhere to report a GitKracken bug.
Grahame Grieve (Jun 21 2017 at 12:41):
it really did write to my c:\temp file - I checked
Eric Haas (Jun 21 2017 at 15:43):
sent invite for ig template repo to gg
Eric Haas (Jun 21 2017 at 15:54):
RE commitiing a file outside my directory- for some reason my working folder has the file /Users/ehaas/Documents/FHIR/IG-Template/c:\temp\test.map don't know how that happened maybe since I work on both a windows desktop and MAX airbook and transfer files back and forth??? removed.
Josh Mandel (Jun 21 2017 at 19:52):
@Grahame Grieve note that you can report issues at https://www.gitkraken.com/contact
Eric Haas (Jun 27 2017 at 16:30):
Back to the spurious c:\temp\test.map file in my Git repo. It is being created everytime I run the ig-build. It contains a mapping script and I think is a bug. GF#13576]
Last updated: Apr 12 2022 at 19:14 UTC
