FHIR Chat · Unable to load template · IG creation

Stream: IG creation

Topic: Unable to load template

Sean McIlvenna (Jun 03 2020 at 16:56):

If I were to copy "ig-template-base" contents and dump it into a directory in my IG called "custom-template", and then have my ig.ini file say template = custom-template why would I get the following error?
Error loading template custom-template: Unable to load template from custom-template

Lloyd McKenzie (Jun 03 2020 at 17:07):

It would have to be #custom-template

Lloyd McKenzie (Jun 03 2020 at 17:07):

If you're planning to do that for playing around, that's fine, but don't plant to publish an HL7 IG doing that...

Sean McIlvenna (Jun 03 2020 at 18:13):

nope, don't plan to

Sean McIlvenna (Jun 03 2020 at 18:18):

@Lloyd McKenzie , I did that... now I get this:
Error loading template #custom-template: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 5 path

Lloyd McKenzie (Jun 03 2020 at 18:25):

Did you make any changes to the template? Can you provide more of a stack trace?

Sean McIlvenna (Jun 03 2020 at 18:27):

sure

Sean McIlvenna (Jun 03 2020 at 18:27):

no changes to template

Sean McIlvenna (Jun 03 2020 at 18:27):

here is stack trace:

Sean McIlvenna (Jun 03 2020 at 18:27):

Load Template from custom-template                                               (00:01.0425)
Publishing Content Failed: Error loading template custom-template: Unable to load template from custom-template (00:01.0426)
(00:01.0426)
Use -? to get command line help                                                  (00:01.0426)
(00:01.0426)
Stack Dump (for debugging):                                                      (00:01.0426)
org.hl7.fhir.exceptions.FHIRException: Error loading template custom-template: Unable to load template from custom-template
at org.hl7.fhir.igtools.templates.TemplateManager.loadPackage(TemplateManager.java:265)
    at org.hl7.fhir.igtools.templates.TemplateManager.installTemplate(TemplateManager.java:93)
    at org.hl7.fhir.igtools.templates.TemplateManager.loadTemplate(TemplateManager.java:79)
    at org.hl7.fhir.igtools.publisher.Publisher.initializeFromIg(Publisher.java:1479)
    at org.hl7.fhir.igtools.publisher.Publisher.initialize(Publisher.java:1322)
    at org.hl7.fhir.igtools.publisher.Publisher.execute(Publisher.java:690)
    at org.hl7.fhir.igtools.publisher.Publisher.main(Publisher.java:7430)
Caused by: org.hl7.fhir.exceptions.FHIRException: Unable to load template from custom-template
at org.hl7.fhir.igtools.templates.TemplateManager.loadPackage(TemplateManager.java:263)
    ... 6 more
IG Publisher finished with code 1

Sean McIlvenna (Jun 03 2020 at 18:27):

FHIR IG Publisher Version 1.0.99-SNAPSHOT (Git# 1a0cb452d00e). Built 2020-06-01T17:41:49.314+10:00 (39 hours old)

Lloyd McKenzie (Jun 03 2020 at 18:42):

I just tried doing this locally with sample-ig and it worked fine. You grabbed the content from the Git project, not from the template package, correct?

Sean McIlvenna (Jun 03 2020 at 19:07):

correct

Sean McIlvenna (Jun 03 2020 at 19:07):

from https://github.com/hl7/ig-template-base/archive/master.zip

Sean McIlvenna (Jun 03 2020 at 19:07):

extracted the sub-folder into custom-template directory

Jose Costa Teixeira (Jun 03 2020 at 19:37):

that may have to do with the way folders are structured

Jose Costa Teixeira (Jun 03 2020 at 19:44):

@Sean McIlvenna what do you have in the custom-templaet directory?

Grahame Grieve (Jun 03 2020 at 19:53):

that is not the correct stack trace. It's the stack trace for template = custom-template, not template=#custom-template

Sean McIlvenna (Jun 03 2020 at 19:58):

crap yah

Sean McIlvenna (Jun 03 2020 at 19:58):

my bad

Sean McIlvenna (Jun 03 2020 at 19:59):

@Grahame Grieve here it comes

Sean McIlvenna (Jun 03 2020 at 19:59):

Running latest IG Publisher: -jar E:\Code\trifolia-on-fhir\apps\server\src\assets\ig-publisher\1.0.99-SNAPSHOT.jar -ig C:\Users\sean.mcilvenna\AppData\Local\Temp\tmp-28580nzZfmSXw42bm\ig.ini -tx N/A
FHIR IG Publisher Version 1.0.99-SNAPSHOT (Git# 1a0cb452d00e). Built 2020-06-01T17:41:49.314+10:00 (58 hours old)
Detected Java version: 13.0.2 from C:\Program Files\Java\jdk-13.0.2 on amd64 (64bit). 12228MB available
Parameters: -ig XXX\tmp-28580nzZfmSXw42bm\ig.ini -tx N/A
dir = E:\Code\trifolia-on-fhir\dist\apps\server, path = C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\cmder;c:\apache-maven-3.6.2\bin;C:\Program Files\Java\jdk-13.0.2\bin;C:\Program Files\PuTTY\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files\Microsoft SQL Server\110\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\Binn\ManagementStudio\;C:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\;C:\Program Files\Git\cmd;C:\Program Files\nodejs\;C:\Program Files\dotnet\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\PROGRA~1\IBM\SQLLIB\BIN;C:\PROGRA~1\IBM\SQLLIB\FUNCTION;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Ruby25-x64\bin;C:\Ruby26-x64\bin;XXX\AppData\Local\Microsoft\WindowsApps;;C:\Program Files\JetBrains\WebStorm 2019.2.4\bin;;C:\Program Files\JetBrains\IntelliJ IDEA 2019.2.4\bin;;C:\Users\sean.mcilvenna\AppData\Roaming\npm;C:\Program Files\JetBrains\RubyMine 2019.3.3\bin;
Run time = Wednesday, June 3, 2020 at 11:16:34 AM Pacific Daylight Time (2020-06-03T11:16:34-07:00)
Package Cache: XXX\.fhir\packages                            (00:00.0007)
Load Template from #custom-template                                              (00:00.0525)
Publishing Content Failed: Error loading template #custom-template: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 5 path $ (00:00.0541)
(00:00.0541)
Use -? to get command line help                                                  (00:00.0542)
                                                                                 (00:00.0542)
Stack Dump (for debugging):                                                      (00:00.0542)
org.hl7.fhir.exceptions.FHIRException: Error loading template #custom-template: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 5 path $
    at org.hl7.fhir.igtools.templates.TemplateManager.loadPackage(TemplateManager.java:265)
    at org.hl7.fhir.igtools.templates.TemplateManager.installTemplate(TemplateManager.java:93)
    at org.hl7.fhir.igtools.templates.TemplateManager.loadTemplate(TemplateManager.java:79)
    at org.hl7.fhir.igtools.publisher.Publisher.initializeFromIg(Publisher.java:1479)
    at org.hl7.fhir.igtools.publisher.Publisher.initialize(Publisher.java:1322)
    at org.hl7.fhir.igtools.publisher.Publisher.execute(Publisher.java:690)
    at org.hl7.fhir.igtools.publisher.Publisher.main(Publisher.java:7430)
Caused by: com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 5 path $
at com.google.gson.JsonParser.parse(JsonParser.java:65)
    at com.google.gson.JsonParser.parse(JsonParser.java:45)
    at org.hl7.fhir.utilities.cache.NpmPackage.fromFolder(NpmPackage.java:289)
    at org.hl7.fhir.igtools.templates.TemplateManager.loadPackage(TemplateManager.java:232)
    ... 6 more
Caused by: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 5 path $
    at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1568)
    at com.google.gson.stream.JsonReader.checkLenient(JsonReader.java:1409)
    at com.google.gson.stream.JsonReader.doPeek(JsonReader.java:542)
    at com.google.gson.stream.JsonReader.peek(JsonReader.java:425)
    at com.google.gson.JsonParser.parse(JsonParser.java:60)
    ... 9 more
IG Publisher finished with code 1
The HL7 IG Publisher failed. The HL7 IG Publisher tool is not developed as part of Trifolia-on-FHIR; it is developed by HL7. If you are still having issues after having reviewed and addressed ToF errors reported in the log above, go to chat.fhir.org to get more information on how to proceed.

Sean McIlvenna (Jun 03 2020 at 20:01):

@Jose Costa Teixeira This is what I have in the custom-template folder:

.gitignore
cache.ini
config/
config.json
content/
includes/
layouts/
package/
package-list.json
README.md
scripts/

Lloyd McKenzie (Jun 03 2020 at 20:08):

Take a look at package/package.json

Lloyd McKenzie (Jun 03 2020 at 20:08):

Is it valid json?

Sean McIlvenna (Jun 03 2020 at 21:05):

yup

Sean McIlvenna (Jun 03 2020 at 21:05):

{
  "name": "fhir.base.template",
  "type": "fhir.template",
  "license": "CC0-1.0",
  "description": "FHIR IG Base Template - foundational for use by anyone",
  "author": "http://hl7.org/fhir",
  "canonical": "http://fhir.org/templates/fhir.base.template",
  "version": "0.1.0"
}

Sean McIlvenna (Jun 03 2020 at 21:06):

@Lloyd McKenzie

Grahame Grieve (Jun 03 2020 at 21:16):

I don't know what you are trying to do here, but something has gone wrong and the java stack is not enough to resolve it. You'll either have to:

• debug, and identify the actual filename that it's trying to load
• instead of trying to nest it, simply clone the git repository and refer to your local copy

Max Masnick (Apr 14 2021 at 11:06):

@Grahame Grieve @Lloyd McKenzie we are having this issue getting the mCODE IG to build on @Lynn Laakso's system.

I'm not sure why the stacktrace doesn't identify the path to the file with the JSON error -- it's really hard to debug this without that info. But in any case, I did some investigation:

1. The IG builds without this error on multiple systems: Mac, Windows, and build.fhir.org
2. I set up the IG publisher + dependencies on a fresh Windows install, and _was_ able to reproduce the error. Removing the custom template fixed it. Adding it back in caused it again.
3. I then tried to create a smaller sample IG with the same custom template to try to reproduce the issue. I _could not_ reproduce the error int he sample IG.
4. I then went back to the mCODE IG to troubleshoot further -- but I could no longer reproduce the error :thinking: Git shows no changes to the code.
5. I deleted .fhir and re-cloned the mCODE IG into a new folder. I _still_ can't reproduce the error.

Here's the stack trace from Lynn's machine. I saw the same exception on mine.

Jose Costa Teixeira (Apr 14 2021 at 11:50):

@Max Masnick this seems to build OK.

Jose Costa Teixeira (Apr 14 2021 at 11:51):

The template only have one json file - the package.json.

Lynn Laakso (Apr 14 2021 at 12:51):

@Max Masnick do you want to try removing the custom template and letting me try again?

Max Masnick (Apr 14 2021 at 16:55):

Jose Costa Teixeira said:

Max Masnick this seems to build OK.

It builds ok on my system, and on build.fhir.org. But it does not on Lynn's system, and did not on my Windows test system (as described above) -- but it does now.

Lynn Laakso (Apr 14 2021 at 16:57):

ok what did you do to get it to build on windows the second time?

Max Masnick (Apr 14 2021 at 16:58):

Lynn Laakso said:

ok what did you do to get it to build on windows the second time?

I'm not sure. The only thing I did on the system was trying to build the IG with the custom template removed, and then trying to reproduce in a sample IG with the same test template.

Max Masnick (Apr 14 2021 at 16:58):

Here's a branch with the custom template removed: https://github.com/HL7/fhir-mCODE-ig/tree/no-custom-template

Jose Costa Teixeira (Apr 14 2021 at 17:00):

to remove the custom template yo can just edit the ig.ini and change #custom-template by fhir.base.template

Jose Costa Teixeira (Apr 14 2021 at 17:01):

(I thought this was a transient issue, I did not read that it doesn't work on Lynn's machine)

Jose Costa Teixeira (Apr 14 2021 at 17:08):

@Lynn Laakso is this still broken?

Lynn Laakso (Apr 14 2021 at 17:08):

yes; working with Max and Mark now

Lynn Laakso (Apr 14 2021 at 17:26):

@Lloyd McKenzie @Grahame Grieve I am still getting the error, and also 70 new ones on my dev environment for Valudation_VAL_Profile_MatchMultiple

Lynn Laakso (Apr 14 2021 at 17:34):

@grahame these errors do not show up on the ci-build

Jose Costa Teixeira (Apr 14 2021 at 17:37):

I found a way to reproduce it :)

Jose Costa Teixeira (Apr 14 2021 at 17:39):

on my machine this breaks when i do a java -jar ..\publisher.jar -ig <ig-path> -resetTx -publish http://hl7.org/fhir/us/mcode/2021May

Jose Costa Teixeira (Apr 14 2021 at 17:39):

it does NOT break when I run .\_genonce.bat -resetTx -publish http://hl7.org/fhir/us/mcode/2021May

Jose Costa Teixeira (Apr 14 2021 at 17:40):

the _genonce.bat script does nothing but run java -jar <publisher> ig .

Jose Costa Teixeira (Apr 14 2021 at 17:40):

which makes it weird & interesting

Max Masnick (Apr 14 2021 at 17:45):

@Jose Costa Teixeira I was able to reproduce with that same command...and then I couldn't reproduce anymore (as described above). So if you can get into the Java debugger or something to see what file is the source of the JSON parsing issue while it's still reproducing, you might want to :)

Otherwise you'll be in my shoes where I can no longer reproduce, so I can't further debug.

Jose Costa Teixeira (Apr 14 2021 at 17:56):

Murphy was right.

Jose Costa Teixeira (Apr 14 2021 at 17:56):

i can get into the Java debugger, but it does not break on the debugger

Jose Costa Teixeira (Apr 14 2021 at 18:38):

I think i found it

Jose Costa Teixeira (Apr 14 2021 at 18:43):

can I make a PR?

Max Masnick (Apr 14 2021 at 18:54):

@Jose Costa Teixeira Awesome! This issue is a blocker for getting the mCODE IG to ballot (or we have to take out our custom template, which we would prefer not to do) so if it's possible to get this merged into a publisher release to allow @Lynn Laakso to build the IG to publish the ballot version, we would really appreciate it.

Jose Costa Teixeira (Apr 14 2021 at 18:55):

can i do that PR later? or @Max Masnick if you don't mind doing it:
(that was not deliberate suspense. Issue is fixed, I believe)

Jose Costa Teixeira (Apr 14 2021 at 21:08):

@Lynn Laakso is this fixed?

Lynn Laakso (Apr 14 2021 at 21:09):

was there an update to the publisher?

Jose Costa Teixeira (Apr 14 2021 at 21:09):

no, the error was in mcode

Jose Costa Teixeira (Apr 14 2021 at 21:10):

if you download it again and try, it should work

Lynn Laakso (Apr 14 2021 at 21:10):

@Max Masnick was the repo updated?

Jose Costa Teixeira (Apr 14 2021 at 21:10):

yes

Jose Costa Teixeira (Apr 14 2021 at 21:11):

45 minutes ago there was a commit to master (merging a PR)

Max Masnick (Apr 14 2021 at 21:12):

@Lynn Laakso yes the repo is updated. If you git pull or just delete and re-clone the repo it should finish building

Lynn Laakso (Apr 14 2021 at 21:29):

OK it builds, though it has 77 errors and they are not on the ci-build

Max Masnick (Apr 15 2021 at 09:38):

Lynn Laakso said:

OK it builds, though it has 77 errors and they are not on the ci-build

Grahame emailed us about this, but for anyone not on the email thread, this is what is going on:

There's a bug in the IG publisher that makes builds non-deterministic -- in other words, if you run java -jar <publisher> ig . twice in a row with exactly the same input, you may get different output.

This bug has to do with the state of the txcache/ folder: it appears a "throw-away" build is needed to get the txcache/ folder into the "correct" state, and then subsequent builds work as expected.

The mCODE IG checks the txcache/ folder into git, which means that builds using genonce and build.fhir.org already have txcache/ in the "correct" state even with a fresh clone of the mCODE GitHub repo.

But the command Lynn has to run to publish the IG includes the -resetTx option, which deletes txcache/ prior to build.

The solution is to:

1. Run the standard build command for publishing, e.g., java -jar publisher.jar -ig hl7.fhir.us.mcode#1.16.0 -resetTx -publish http://hl7.org/fhir/us/mcode/2021May
2. Run the same command again, but without the -resetTx option: java -jar publisher.jar -ig hl7.fhir.us.mcode#1.16.0 -publish http://hl7.org/fhir/us/mcode/2021May

The first build has 77 QA errors. The second build has 0.

Here's the GitHub issue for this bug: https://github.com/HL7/fhir-ig-publisher/issues/231

Lloyd McKenzie (Apr 15 2021 at 20:51):

@Jose Costa Teixeira Do you think we need something added to the publisher to catch the issue you found in the mCode IG to make it easier to diagnose the next time the same problem shows up? If so, might be good to submit a Git issue against the publisher.

Jose Costa Teixeira (Apr 15 2021 at 20:58):

The issue was a strange unexpected character in the beginning of the JSON (could not even delete it in my IDE). The exception was from the json parser.
We could try to get the extension trapping more explicit about which JSON it was parsing -if that is easy it would solve other JSONs, not only that in the template. If that is easy, it would be worth it.

Jose Costa Teixeira (Apr 15 2021 at 20:59):

but I hope this is a rare issue, so I don't know if it is worth going out of our way.

Jose Costa Teixeira (Apr 15 2021 at 21:01):

In other parts of the publisher, a more verbose output or an indication of which file it is working, would be helpful. I presume that malformed conformance JSONs would be more common than malformed JSONs for custom templates

Jose Costa Teixeira (Apr 15 2021 at 21:04):

that, or perhaps a new feature like a "pre-sanitizing / pre-check" of all the IG files for basic syntax checking (forgotten commas in JSON, quotes in XML..)

Jose Costa Teixeira (Apr 15 2021 at 21:05):

But this is just spitballing. I didn't think of a way to fix this and I presume this would be very very rare compared to other issues.

Max Masnick (Apr 15 2021 at 22:30):

@Jose Costa Teixeira @Lloyd McKenzie if the publisher could identify the path of the file with invalid JSON as part of the stack trace, that would have saved probably 10 person-hours for this bug, and would be helpful for all other JSON parsing issues as well.

Grahame Grieve (Apr 16 2021 at 07:08):

which file was it?

Jose Costa Teixeira (Apr 16 2021 at 07:35):

The local template's package.json

Jose Costa Teixeira (Apr 16 2021 at 07:36):

My PR shows the change

Grahame Grieve (Apr 16 2021 at 07:44):

which PR?

Last updated: Apr 12 2022 at 19:14 UTC