FHIR Chat · IG Publisher and Simplifier packages · IG creation

Stream: IG creation

Topic: IG Publisher and Simplifier packages

Noemi Deppenwiese (Mar 26 2020 at 13:38):

I added external dependencies to my IG (via the "dependsOn" in my IG ressource). Unfortunatly, while this works fine for IG Publisher packages, I depend on one Simplifier Package (de.basisprofil.r4#0.9.2) which causes an error when added:

java.io.IOException: Unable to find the file other/spec.internals in the package de.basisprofil.r4

Is there any solution / workaround for this error so I can get my IG to build?

Lloyd McKenzie (Mar 26 2020 at 15:12):

@Ward Weistra

Ward Weistra (Mar 26 2020 at 15:33):

@Grahame Grieve What's other/spec.internals? Can't find it in https://confluence.hl7.org/display/FHIR/NPM+Package+Specification.
I worry multiple classes of packages are emerging. With or without snapshots, with or without implementation guides. I see multiple users running into that.

Martijn Harthoorn (Mar 26 2020 at 17:50):

This issue is not related to Simplifier. But related to the IG publisher not playing nice with the Package Spec.

Grahame Grieve (Mar 26 2020 at 19:03):

I worry multiple classes of packages are emerging

yeah well, more collaboration on your part would help with that

Grahame Grieve (Mar 26 2020 at 19:04):

I've been trying to make spec.nternals go away but haven't got there yet

Ward Weistra (Mar 27 2020 at 10:08):

@Grahame Grieve Sorry if we haven't. What's the best way: set a dedicated meeting, discuss at an upcoming FHIR-I, a dedicated Zulip thread? I'll set it up. Topics:

• Should every package include snapshots or can we expect consumers to be able to generate them? Is it an issue if some do and some don't?
• Should every package contain an IG resource? Is it an issue if some do and some don't?
• Should packages just the machine-readable content (FHIR resources) of an IG or also the human-readable (pages, images) part?
• Should anything be allowed in packages (like the tools-package with images for IGs)?

And please add any requirements (spec.internals, others) you have for the IG publisher to the list, so we can discuss whether packages are the place for them and how.

Grahame Grieve (Mar 27 2020 at 10:27):

well, we can discuss it here. Some answers:

• all my packages will contain snapshots, but we haven't said that they must. OTOH, I think there's only 2 sets of code for generating them correctly, so I think that they should. I think it's important that we both use the same test cases. I know you haven't done anything about that or we'd be talking about the test cases

Grahame Grieve (Mar 27 2020 at 10:27):

• all packages that represent an IG should contain an IG resource. Other packages won't

Grahame Grieve (Mar 27 2020 at 10:28):

• I think packages should only contain the machine readable stuff. that's what the package spec says

Grahame Grieve (Mar 27 2020 at 10:29):

• anything is allowed in the packages in other folders, and it should still be the case. I use kind to tell the different kinds of packages apart

Ward Weistra (Mar 27 2020 at 13:06):

• Snapshots: I tend to agree it would be better if all packages had them. The worry has been the server load impact, I'll revisit what the impact would be and if we could manage to mandate them.

• Test cases: Yes, those are important. For the general .NET API there's the ticket you're familiar with: https://github.com/FirelyTeam/fhir-net-api/issues/1262. That's for the API team (@Ewout Kramer @Marco Visser) to decide on. Package generation is in a separate library on our side. Are there dedicated tests for package generation, can't find them in https://github.com/FHIR/fhir-test-cases. We should indeed have those to validate our works against the spec.

I think packages should only contain the machine readable stuff. that's what the package spec says

• I think you are only referring to the package folder in the package, given your next bullet. Indeed the current spec allows anything in other folders (even an empty package folder), but for size (hosting space, download size), security, clarity I'm not very comfortable with that. Packages not meant as the distribution of an API specification/data model seems like a misuse of packages to me.
• kind is only part of the listing in your feed, for now, right? If it's needed, let's make a proposal to add it to the package manifest if there's already a definition for different kinds of packages in practice.
• How about any other requirements the IG publisher has on packages right now? Let's create a proposal for additions to the package spec, perhaps for a specific kind, and discuss it.

Ward Weistra (Mar 27 2020 at 13:07):

@Noemi Deppenwiese Sorry for going deep in the weeds following your question. Grahame, can we think of a shorter-term solution for her?

Grahame Grieve (Mar 27 2020 at 18:08):

it's on my todo to finish getting rid of the dependency on spec.internals which was supposed to be gone by now

Grahame Grieve (Apr 07 2020 at 12:10):

so returning to this subject... what spec.internals does for me, that the package de.basisprofil.r4#0.9.2 doesn't, is tell me what html reference to use for a canonical URL. And the general canonical URL is not the right reference - there's version issues, and hosting issues (this one doesn't resolve at all)

Grahame Grieve (Apr 07 2020 at 12:10):

so I don't know how to depend on the package in an implementation guide

Grahame Grieve (Apr 07 2020 at 12:10):

@Ward Weistra @Noemi Deppenwiese

Noemi Deppenwiese (Apr 07 2020 at 15:19):

If i understand you corretly, you're looking for the HTML of the (rendered) IG? As far as I can tell, that is always the "url" in the ImplementationGuide resource for Simplifier IGs.

Grahame Grieve (Apr 07 2020 at 21:04):

so that references the root of the ig (with a redirect)

Grahame Grieve (Apr 07 2020 at 21:05):

that doesn't make it possible for me to find the url for the individual resources

Ward Weistra (Apr 08 2020 at 10:38):

In this case the specific canonicals also redirect to the right pages in the Simplifier project, right? E.g.

• http://fhir.de/CodeSystem/arge-ik/klassifikation -> https://simplifier.net/BasisprofilDE/klassifikation
• http://fhir.de/StructureDefinition/practitioner-de-basis -> https://simplifier.net/BasisprofilDE/practitioner-de-basis

For the more general case I get your point, this might not always be the case. I'll think about how we can provide that in general and how we could add that requirement to the spec.
What's the use of those html-URLs? Are you making web links in the IG publisher's results to where the parent profiles are described?

Grahame Grieve (Apr 08 2020 at 11:55):

interesting. the first canonical I tried didn't even resolve. those ones you have there do. but how does versioning work?

Grahame Grieve (Apr 08 2020 at 11:55):

and yes, that's the use

Grahame Grieve (Apr 08 2020 at 11:55):

the packaging, it doesn't matter, but the generated html links to the right version of the content

Grahame Grieve (Apr 08 2020 at 11:57):

I'm moving away from spec.internals because we added a way to do that in the IG resource in R4. So that means that the IG generator has to find somewhere to put an R4 IG for older versions of the spec ... that's why I haven't done it yet; it holds certain... challenges...

Ward Weistra (May 15 2020 at 14:29):

@Grahame Grieve What do packages in practice currently need to have on top of https://confluence.hl7.org/display/FHIR/NPM+Package+Specification#NPMPackageSpecification-ImplementationGuidesandpackages to be accepted as a dependency for the IG publisher?

1. other/spec.internals?
2. .index.json not anymore, right?
3. I see in a created package both ig-r4.json and ImplementationGuide-nl.wardweistra.acme.json. Is that needed (not sure why this happened, maybe misuse of IG publisher by me)?
4. In the package manifest: directories? tools-version? type? date?
5. Anything else?

Grahame Grieve (May 15 2020 at 17:31):

• not required: .index.json will be built on the fly if not not found
• required in the package manifest (I think): name | version | canonical | url | fhirVersions (if no dependency)

With regard to spec.internals, we are trying to phase it out, but haven't done so yet. It's how I locate resources to refer to them in the web. Last time round we discussed this, simplifier packages do not have a way to locate the correct version on the web, and the discussion stopped before this was resolved. So right now, I can't depend on simplifier packages because this is unresolved. One way to resolve this would be spec.internals, but I'm trying to phase that out and replace it with just using the IG resource, but I need the more recent version of the IG resource to do that. Hence, producing ig-r.json. Only, I don't use that yet, because the we didn't finish that discussion I referred to above.

Ward Weistra (May 15 2020 at 17:56):

@Grahame Grieve Simplifier has a way to find the webpage for a specific resource by providing a package name + version and canonical: eg https://simplifier.net/resolve?scope=nictiz.fhir.nl.stu3.zib2017@1.3.9&canonical=http://fhir.nl/fhir/StructureDefinition/nl-core-patient. I guess that could be a way for the IG Publisher to link to the web version.

Another way could of course be to change the IG publisher logic, to not require having a direct link to the parent profile, but linking to the base URL for the Implementation Guide (from the IG resource) and let them find the right parent profile themselves? A little less user friendly, but it more robust, not needing any data that isn't guaranteed to be there.

Grahame Grieve (May 15 2020 at 17:58):

A little less user friendly

Entirely not at all user friendly and I won't consider it

Grahame Grieve (May 15 2020 at 17:59):

why is the other method not guaranteed?

Ward Weistra (May 15 2020 at 18:02):

The resolve URL (https://simplifier.net/resolve?scope=nictiz.fhir.nl.stu3.zib2017@1.3.9&canonical=http://fhir.nl/fhir/StructureDefinition/nl-core-patient) works for every package in the package registry.

Grahame Grieve (May 15 2020 at 18:03):

so it is guaranteed then?

Ward Weistra (May 15 2020 at 18:08):

That URL is. So, I'm happy if you want to use that to create these links.

Grahame Grieve (May 15 2020 at 18:08):

ok I'll work on that next week

Grahame Grieve (May 20 2020 at 05:23):

so the problem for me is that I can't tell whether a package is a simplifier package or not. There's no way is there?

Grahame Grieve (May 20 2020 at 05:53):

@Ward Weistra

Ward Weistra (May 20 2020 at 06:55):

@Grahame Grieve Not yet. Although for now likely every package not from the IG publisher will be a Simplifier package? But that might someday change. (Still the url pattern will give a page for each registry package)

I expect the Registry work to introduce a way to show from which feed it came to the registry. But perhaps it's better if you didn't have to write code per feed and in the package spec, we add support for either adding a link per resource (already there with via the manifest of the IG resource?) or providing a url pattern?

Do you ever need web urls for examples from dependency packages? Since I believe they generally don't have a canonical the resolving URL pattern based on canonical doesn't work for them.

Grahame Grieve (May 20 2020 at 07:12):

for IGs that we publish, all resources resolve, not just the canonicals.

Grahame Grieve (May 20 2020 at 07:12):

that's convenient but not mandatory. But if people reference them, what to do but reference their web location?

Grahame Grieve (May 20 2020 at 07:14):

it's true that all packages that don't have a spec.internals currently come from simplifier.

Grahame Grieve (May 20 2020 at 07:15):

it would be good to transition to ImplementationGuide.manifest.resource.relativePath. That's still on my todo list

Grahame Grieve (May 20 2020 at 08:02):

@Ward Weistra Simplifier is not populating StructureDefinition.derivation. The Java code won't process structure definitions unless they declare whether they are a constraint or a specialization.

Technically, the element is optional since there's one type that doesn't have a base definition. Otherwise, it's supposed to be populated but we overlooked making the constraint.

I suppose Simplifier imagines that "constraint" is the default. But is it set for logical models?

Ward Weistra (May 20 2020 at 14:59):

@Grahame Grieve I asked this in the API team and it seems Forge does populate StructureDefinition.derivation with constraint for StructureDefinitions and specialization for Logical Models and snapshot generator does not touch it. Perhaps we don't fill it if people upload a SD without it.

Taking a random (not random, my goto profile actually) it does seem to be filled: https://simplifier.net/NictizSTU3-Zib2017/nl-core-patient/~xml. Any specific packages where you ran into this?

Grahame Grieve (May 20 2020 at 20:08):

The first package I worked with is de.basisprofil.r4#0.9.2 - Extension-gender-amtlich-de.json

Ward Weistra (May 20 2020 at 21:53):

Indeed, it is not filled in that Extension in that package version: https://simplifier.net/packages/de.basisprofil.r4/0.9.2/files/114506/~xml
However, it is filled in the most recent version of the package: https://simplifier.net/packages/de.basisprofil.r4/0.9.6/files/129547/~xml
Added to the extension in March: https://simplifier.net/Basisprofil-DE-R4/GenderAmtlichDE/$diff/4/5/Original
Seems @Patrick Werner added it manually, likely because the Java tooling didn't like it? Patrick, do you know how the extension was created, why derivation was missing in the first place? Or perhaps the creator @Simone Heckmann? Since Forge seems to include it with any new Extension and doesn't even allow you to change it.

Grahame Grieve (May 20 2020 at 23:53):

ok, well, the next release of the IG publisher will support simplifier packages. At least in principle. You can now declare a simplifier package in your dependencies:

  <dependsOn id="debasis">
    <uri value="http://fhir.de"/>
    <packageId value="de.basisprofil.r4"/>
    <version value="0.9.6"/>
  </dependsOn>

And the IG will run. I expect that there will be lots of downstream problems, so I'm just calling this provisional support at this time

Grahame Grieve (May 20 2020 at 23:54):

note that for simplifier packages, it's the packageId and version that matter - the canonical URL can be determined from the package

Kevin Mayfield (May 23 2020 at 05:11):

Would it be possible for me to test UK/NHS packages with this? @Ward Weistra Presume I run the IG Publisher against the simplifier package?

Grahame Grieve (May 23 2020 at 06:39):

it would be good for you to try. You can't 'run the IG publisher against the simplifier package'. What you can do is refer to a simplifier package from an IG publisher project

Ward Weistra (May 23 2020 at 08:50):

Wonderful @Grahame Grieve, thanks. I'll get testing on Monday. Both on depending on a Simplifier package with IG publisher and creating an IG publisher IG from a Simplifier project.

Ward Weistra (Jun 15 2020 at 23:09):

Grahame Grieve said:

• not required: .index.json will be built on the fly if not not found

Thoughts on why I run into this error building an IG with the publisher on de.basisprofil.r4@0.9.7?

Publishing Content Failed: Error parsing C:\Users\Ward\.fhir\packages\de.basisprofil.r4#0.9.7\package\.index.json: null (00:26.0931)
                                                                                 (00:26.0931)
Use -? to get command line help                                                  (00:26.0931)
                                                                                 (00:26.0932)
Stack Dump (for debugging):                                                      (00:26.0932)
java.io.IOException: Error parsing C:\Users\Ward\.fhir\packages\de.basisprofil.r4#0.9.7\package\.index.json: null
        at org.hl7.fhir.utilities.cache.NpmPackage.loadFiles(NpmPackage.java:251)
        at org.hl7.fhir.utilities.cache.NpmPackage.fromFolder(NpmPackage.java:213)
        at org.hl7.fhir.utilities.cache.FilesystemPackageCacheManager.loadPackageInfo(FilesystemPackageCacheManager.java:144)
        at org.hl7.fhir.utilities.cache.FilesystemPackageCacheManager.loadPackageFromCacheOnly(FilesystemPackageCacheManager.java:311)
        at org.hl7.fhir.igtools.publisher.Publisher.loadIg(Publisher.java:2533)
        at org.hl7.fhir.igtools.publisher.Publisher.initializeFromIg(Publisher.java:1745)
        at org.hl7.fhir.igtools.publisher.Publisher.initialize(Publisher.java:1325)
        at org.hl7.fhir.igtools.publisher.Publisher.execute(Publisher.java:692)
        at org.hl7.fhir.igtools.publisher.Publisher.main(Publisher.java:7533)
Caused by: java.lang.NullPointerException
        at org.hl7.fhir.utilities.cache.NpmPackage$NpmPackageFolder.readIndex(NpmPackage.java:125)
        at org.hl7.fhir.utilities.cache.NpmPackage.loadFiles(NpmPackage.java:249)
        ... 8 more

Ward Weistra (Jun 16 2020 at 14:02):

Got the same when using hl7.fhir.us.core... Removed the FHIR cache and now it runs for us.core.

Ward Weistra (Jun 16 2020 at 14:10):

Aaand it also runs for de.basisprofil.r4 now :heart:

Ward Weistra (Jun 16 2020 at 14:15):

I do get some curious errors though, especially:

unable to parse character reference [...] This structure is derived from <a href="null/https://simplifier.net/resolve?scope=de.basisprofil.r4@0.9.7&canonical=http://fhir.de/StructureDefinition/practitioner-de-basis">PractitionerDeBasis</a> [...]
image.png

Is that null the IG base URL or something?

Ward Weistra (Jun 16 2020 at 14:34):

Terminal output:

Internal error in location for message: 'Error in ?? at 1, 1: Premature ExpressionNode termination at unexpected token ":"', loc = 'C:\Users\Ward\Documents\FHIR workspace\R4\SushiProjectGerman\build\output\StructureDefinition-anId-d', err = 'unable to parse character reference 'canonical=http://fhir.de/StructureDefinition/identifier-lanr" t'
Internal error in location for message: 'Error in ?? at 1, 1: Premature ExpressionNode termination at unexpected token ":"', loc = 'C:\Users\Ward\Documents\FHIR workspace\R4\SushiProjectGerman\build\output\StructureDefinition-anId.h', err = 'unable to parse character reference 'canonical=http://fhir.de/StructureDefinition/practitioner-de-ba'

Would love to hear if this is a legitimate problem with the package or other pointers.

Ward Weistra (Jun 17 2020 at 16:23):

@Grahame Grieve Where does the IG publisher get packages, specifically the hl7.fhir.us.core package, from? From using the IG publisher, I get a different version of the hl7.fhir.us.core@3.1.0 package (with a single package root folder). From using Sushi or Forge I get the one being served from packages.fhir.org, still with the multiple root folder as it was offered originally.
Multiple versions of the same package version are really scary to me. Can we make the IG publisher look on packages.fhir.org first?

Also, the IG publisher fails with the above java.io.IOException: Error parsing C:\Users\Ward\.fhir\packages\hl7.fhir.us.core#3.1.0\package\.index.json: null error when using the package from the package cache if installed with Forge. It doesn't fail when Sushi (or the IG publisher) installs the package. I assume that is because Sushi is also updating the package/.index.json file which IG Publisher then still relies on, because I see the .index.json in the Sushi version of the package is indeed updated.

Chris Moesel (Jun 17 2020 at 16:28):

@Ward Weistra -- see this thread which speaks of the magic IG Publisher (and now SUSH) does for older package formats: https://chat.fhir.org/#narrow/stream/215610-shorthand/topic/dev.20dependencies/near/199502644

In short, when we detect the old package format, we fix it before storing it in the .fhir cache.

Ward Weistra (Jun 17 2020 at 16:37):

Thanks @Chris Moesel. I read that earlier, but didn't comprehend the impact. I've updated my sushi and indeed see the same post-processed package coming from sushi and IG publisher.

However, sushi doesn't fail on the package installed by Forge (without an post-processed package/.index.json), where the IG publisher does. That is still something that likely needs resolving.

Chris Moesel (Jun 17 2020 at 16:56):

SUSHI doesn't even look at the .index.json file, so that may be why it doesn't fail.

Grahame Grieve (Jun 17 2020 at 20:16):

how do I get forge to install a package?

Grahame Grieve (Jun 17 2020 at 20:44):

I can't reproduce this:

Internal error in location for message: 'Error in ?? at 1, 1: Premature ExpressionNode termination at unexpected token ":"', loc = 'C:\Users\Ward\Documents\FHIR workspace\R4\SushiProjectGerman\build\output\StructureDefinition-anId-d', err = 'unable to parse character reference 'canonical=http://fhir.de/StructureDefinition/identifier-lanr" t'

Grahame Grieve (Jun 17 2020 at 20:44):

and I can't figure out how to get torinox to run, or forge to install a package. but do you get a stack dump?

Grahame Grieve (Jun 18 2020 at 07:17):

@Ward Weistra The .index.json file that Torinox installs does not conform to the specification, so the IG pbulisher fails to read it. The specification is here: https://confluence.hl7.org/display/FHIR/NPM+Package+Specification#NPMPackageSpecification-.index.json

Instead, the torinox file looks like:

{
  "date": "2020-06-18T17:11:33.8586458+10:00",
  "canonicals": {
    "http://hl7.org/fhir/us/core/CapabilityStatement/us-core-client": "CapabilityStatement-us-core-client.json",
...

Grahame Grieve (Jun 18 2020 at 07:57):

IG publisher won't crash in the future. It'll just overwrite any .index.json that doesn't have "index-version": 1 as specified.

Ward Weistra (Jun 18 2020 at 12:38):

@Grahame Grieve Sorry, I was completely convinced the .NET packages library didn't touch package/.index.json, but you are right that it does. I'll look into fixing that, but regenerating it as stated in the spec should work.
Installing with Forge: Open a Profile Folder > go to Dependencies tab > Select Simplifier to search there > Search for package name > Select version and click Add.

Ward Weistra (Jun 18 2020 at 12:59):

@Grahame Grieve The only likely relevant part in the terminal on the internal error seems to be:

Generating Summary Outputs                                                       (00:39.0686)
Sending Usage Stats to Server                                                    (00:49.0863)
Jekyll: Source: C:/Users/Ward/Documents/FHIR workspace/R4/SushiProjectGerman/build/temp/pages (00:54.0547)
Jekyll: Generating...                                                            (00:54.0548)
Jekyll: done in 2.294 seconds.                                                   (00:56.0848)
Installing nl.wardweistra.acme#dev to the package cache
  Fetching:
  Installing: Checking Output HTML                                                             (00:57.0152)
found 517 files                                                                  (00:57.0207)
  ... 517 html files, 2 pages invalid xhtml (0%)                                 (01:03.0488)
  ... 31089 links, 11 broken links (0%)                                          (01:03.0489)
Build final .zip                                                                 (01:03.0489)
Final .zip built                                                                 (01:04.0013)
context [anonymous] 2:13 attribute path isn't defined
Internal error in location for message: 'Error in ?? at 1, 1: Premature ExpressionNode termination at unexpected token ":"', loc = 'C:\Users\Ward\Documents\FHIR workspace\R4\SushiProjectGerman\build\output\StructureDefinition-anId-d', err = 'unable to parse character reference 'canonical=http://fhir.de/StructureDefinition/identifier-lanr" t'
Internal error in location for message: 'Error in ?? at 1, 1: Premature ExpressionNode termination at unexpected token ":"', loc = 'C:\Users\Ward\Documents\FHIR workspace\R4\SushiProjectGerman\build\output\StructureDefinition-anId.h', err = 'unable to parse character reference 'canonical=http://fhir.de/StructureDefinition/practitioner-de-ba'
Finished. 00:34.0645. Validation output in C:\Users\Ward\Documents\FHIR workspace\R4\SushiProjectGerman\build\output\qa.html
Errors: 2, Warnings: 3, Info: 0, Broken Links = 11 (01:04.0393)
Done                                                                             (01:04.0399)
Press any key to continue . . .

Or is there a place to find more output?

My ImplementationGuide has:

  "dependsOn": [
    {
      "uri": "https://simplifier.net/guide/basisprofil-de-r4",
      "packageId": "de.basisprofil.r4",
      "id": "de_basisprofil_r4",
      "version": "0.9.7"
    }
  ]

The only resource I'm using is a very simple derived profile from the German base practitioner.

Ward Weistra (Jun 18 2020 at 13:45):

Here's the project on Github for reproduction: https://github.com/wardweistra/SushiProjectGerman

Grahame Grieve (Jun 18 2020 at 17:00):

ok thanks. I'll look into it

Grahame Grieve (Jun 19 2020 at 01:38):

ok, well, we haven't had a canonical URL that included a & before. All the errors were associated with that. I've fixed them in the next release (probably about 24 hours away)

Last updated: Apr 12 2022 at 19:14 UTC