Stream: implementers
Topic: IG publisher - help with troubleshooting hanging
Rob Eastwood (Dec 20 2017 at 00:16):
Hello,
We are having issues with the IG publisher hanging at a particular point and have figured out that it has something to do with our company network - not the IG publisher specifically. If we run the same inputs via our home network or via our personal mobile hotspots, all is well. As such we are hypothesising that there may have been some firewall changes that are causing the problem. Note that a few weeks ago, there were no problems and it ran well. I have logged a ticket with our internal IT to try to progress things from that end, but I am wondering if there may be some pointers from this community as to where the problem may lie.
Our implementation guide has profiles that are formally derived from various profiles in the HL7 Australia 'au-fhir-base' IG. For example, we have a patient profile derived from the HL7 Au patient profile etc. The json controller file contains this source as a dependencyList entry.
Running a recent IG publisher v3.1.0-12604 (however the version doesn't affect the problem behaviour).
We have also added full verbose logging in the json controller file to better see what is going on. Running the IG publisher on the command line gives the following output:
...... Check Generate C:\work\git\dh-participants-ig-forge\qa\tbl_spacer.png (24.0373sec) Check Generate C:\work\git\dh-participants-ig-forge\temp\tbl_spacer.png (24.0374sec) Initialization complete (24.0376sec) Load Content (24.0377sec) Processing Conformance Resources (25.0628sec) process type: NamingSystem (25.0628sec) process type: CodeSystem (25.0628sec) process type: ValueSet (25.0628sec) process type: ConceptMap (25.0628sec) process type: DataElement (25.0628sec) process type: StructureDefinition (25.0628sec) process res: patient-australianhuman-base (25.0629sec)
At the point of the first structure definition resource, the command line output just hangs there without further progress and there are no other indications of failure. At this point, if we then switch networks to our personal hotspot, the output then resumes and completes within an expected timeframe.
We are at a loss as to what is specifically occurring at this point of the process in order to inform our IT staff. Any ideas? I'd be happy to provide any further info, not sure what extra helps at the moment.
Eric Haas (Dec 20 2017 at 03:23):
Do you have the dependency as a source file as well as a url (location)?
e.g.,
"dependencyList": [
{
"name" : "uscore",
"location" : "http://hl7.org/fhir/us/core",
"source" : "definitions.json.zip"
}
also you can run the run the ig publisher - to run without a terminology server use the '-tx N/A' option and see if that is the issue
Rob Eastwood (Dec 20 2017 at 04:42):
Thank you @Eric Haas,
Re dependencyList source
Well spotted. We didn't have the source file listed. But actually, after adding it and when the publisher is then run (with no other changes) this error is given:
Connect to Terminology Server at http://tx.fhir.org/r3 (16.0757sec)
Terminology Server Version has changed from 1.0.152 to 1.0.198, so clearing txCache (18.0994sec)
Load aubase (http://build.fhir.org/ig/hl7au/au-fhir-base) from definitions.json.zip (19.0253sec)
Publishing Content Failed: C:\work\git\dh-participants-ig-forge\definitions.json.zip\definitions.xml.zip (The system cannot find the path specified) (19.0254sec)
(19.0254sec)
Use -? to get command line help (19.0254sec)
(19.0254sec)
Stack Dump (for debugging): (19.0254sec)
java.io.FileNotFoundException: C:\work\git\dh-participants-ig-forge\definitions.json.zip\definitions.xml.zip (The system cannot find the path specified)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at org.hl7.fhir.igtools.publisher.Publisher.fetchDefinitions(Publisher.java:1211)
at org.hl7.fhir.igtools.publisher.Publisher.loadIg(Publisher.java:1120)
at org.hl7.fhir.igtools.publisher.Publisher.initialize(Publisher.java:855)
at org.hl7.fhir.igtools.publisher.Publisher.execute(Publisher.java:342)
at org.hl7.fhir.igtools.publisher.Publisher.main(Publisher.java:3969)
Exception in thread "main" java.lang.NullPointerException
at org.hl7.fhir.igtools.publisher.Publisher.main(Publisher.java:3979)
This file does exist within the resources folder.
Re switching off the terminology server:
Reverting the IG json controller file to remove the dependency source file and running the IG publisher with -tx N/A then does successfully complete (along with the expected warnings about "running without terminology server" etc).
When I combine both of the above, I still get the stack dump error listed first.
Rob Eastwood (Dec 20 2017 at 05:39):
The IT staff have indicated there is a known issue with our network regarding authentication handshaking with various websites. And it seems that the FHIR terminology server is now one of them!
Brian Postlethwaite (Dec 20 2017 at 06:24):
When has this started to happen @Rob Eastwood ?
Richard Townley-O'Neill (Dec 20 2017 at 06:25):
@Brian Postlethwaite
Richard Townley-O'Neill (Dec 20 2017 at 06:25):
A couple of weeks ago. IT have been making network changes.
Richard Townley-O'Neill (Dec 20 2017 at 06:26):
We can run the ig publisher from home.
Eric Haas (Dec 20 2017 at 07:24):
C:\work\git\dh-participants-ig-forge\definitions.json.zip\definitions.xml.zip is not a valid path looks like definitions.xml.zip is appended to the definitions.json.zip
Eric Haas (Dec 20 2017 at 07:25):
if it runs w/o hitting the terminology server then that could be the network firewall issue.
Grahame Grieve (Dec 20 2017 at 10:05):
are they forcing the use of ssl?
Rob Eastwood (Dec 20 2017 at 23:34):
Thanks guys for your input.
@Brian Postlethwaite - it started on (or 1 day either side of) Tues Dec 5
@Grahame Grieve - not sure about the specifics re SSL but I will put that to the IT guys
@Eric Haas - yes this is very curious and it seems to be a second (probably related?) problem. The file definitions.json.zip is located in the resources folder but its content neither includes a definitions.xml.zipor definitions for the specific dependency; that is, it has a bunch of 'us-core' related definitions. This originates from the Healthedata1 template that we commenced using some months ago and this zip file has not been updated since. So I have followed the file structure of the Healthedata1 template re the dependencies folders, copied in the definitions.json.zip file from the actual dependency and now the error does not occur. The json controller file now has the added line indicated below:
"dependencyList": [{
"name" : "aubase",
"location" : "http://build.fhir.org/ig/hl7au/au-fhir-base",
"source": "dependencies/aubase" <<<< Added this line
}]
Thanks for pointing me in the right direction.
Rob Eastwood (Dec 20 2017 at 23:49):
Oh, just to confirm - the above changes re the dependencyList source eliminates the stack dump error listed above, but the IG publisher still only completes its run using the no terminology server switch (-tx N/A)
Rob Eastwood (Jan 08 2018 at 22:52):
Hello @Grahame Grieve - re SSL: it looks like at the time of this problem starting, SSL started being enforced (yet to be confirmed but all indications are so). We are still seeking assistance from our IT guys, but is there anything we can do via running the publisher that can help? Thank you
Grahame Grieve (Jan 08 2018 at 22:55):
remind me what the actual error is?
Rob Eastwood (Jan 08 2018 at 23:15):
Well, there isn't an actual error per se as the output just hangs at the point it starts processing the first structure definition file. With full logging turned on the output just stops at this (initial output truncated for brevity):
..... Check Generate C:\work\git\dh-participants-ig-forge\qa\tbl_spacer.png (18.0083sec) Check Generate C:\work\git\dh-participants-ig-forge\temp\tbl_spacer.png (18.0084sec) Initialization complete (18.0085sec) Load Content (18.0085sec) Processing Conformance Resources (18.0954sec) process type: NamingSystem (18.0964sec) process type: CodeSystem (18.0964sec) process type: ValueSet (18.0964sec) process type: ConceptMap (18.0964sec) process type: DataElement (18.0964sec) process type: StructureDefinition (18.0965sec) process res: patient-australianhuman-base (18.0965sec)
Grahame Grieve (Jan 08 2018 at 23:42):
maybe you'll have to run it under the debugger?
Rob Eastwood (Jan 08 2018 at 23:52):
Thanks Grahame - not sure what you mean by that (did a search on zulip and the IG publisher wiki). Not sure if you mean the logging outputs, as I have put this into the ig.json file:
"logging": ["init", "progress", "context", "html", "tx"],
Could you poke me in the right direction please
Grahame Grieve (Jan 08 2018 at 23:53):
all the source is in the main svn (under \build) - use eclipse and run the IG publisher under the java debugger.
Grahame Grieve (Jan 08 2018 at 23:53):
maybe someone in Phil's team can help
Rob Eastwood (Jan 08 2018 at 23:54):
ok thanks Grahame :)
Last updated: Apr 12 2022 at 19:14 UTC
