FHIR Chat · Upload of IG · hapi

Stream: hapi

Topic: Upload of IG

Jens Villadsen (Apr 01 2020 at 13:10):

@James Agnew it looks like the upload-igpack (https://github.com/jamesagnew/hapi-fhir/blob/master/hapi-fhir-cli/hapi-fhir-cli-api/src/main/java/ca/uhn/fhir/cli/IgPackUploader.java) could use some TLC, and I have some hours to spend on it to do so. Before I do it, I would like to hear if it was more or less deliberately abandoned in its current state?

James Agnew (Apr 01 2020 at 13:48):

Hi @Jens Villadsen - We certainly would gladly accept any improvements to it, but I also consider it mostly abandoned at this point.

NPM is now the "right way" to solve the artifact packaging problem in FHIR, so what we really need to do is get support for that implemented. All of the recent validation API changes were geared towards getting that effort started. If by chance you (or anyone) is interested, Grahame has already written lots of code to parse NPMs. The next task is to create an implementation of IValidationSupport (against current HAPI FHIR Master since that interface is fairly different now) that can open and supply artifacts from an NPM module.

I'm planning on doing this soon, probably next month, but if someone beats me to it I certainly won't complain. :)

Jens Villadsen (Apr 01 2020 at 13:56):

On another project we already do the NPM package parsing and load it into HAPI (with added bonus features of whitelisting according to specific capability statements - and we do it on servlet load directly on the resource providers instead of using an HTTP client but that is easily changed) - basically boot strapping a HAPI FHIR instance with the contents of an IG, so the effort is very little for me to add this into the cli.

Jens Villadsen (Apr 01 2020 at 13:58):

My deadline for having this is the 14th of april - so may I suggest that you wait until that date and see what I end up with in a PR ;)

Jens Villadsen (Apr 01 2020 at 14:00):

Btw - end goal for me is to easily be able to instantiate a FHIR server with the 'rule set' of an IG - meaning that the FHIR server must have the IG loaded. All wrapped in docker compose (content is probably going to be postgres, HAPI FHIR JPA, cli client with IG url parameter and an elasticsearch instance)

Patrick Werner (Apr 01 2020 at 14:51):

Great development! Love to see hapi npm support!
@Jens Villadsen could this be implemented that we could specify the needed npm IG packages in the jpastarter project as well?
The hapi jpaserver would then load these packages from the package repos directly.

Patrick Werner (Apr 01 2020 at 14:52):

As not all IGs are published in the registry, an alternative config method in which you point to local IG npm packages on the filesystem would also useful.

Jens Villadsen (Apr 01 2020 at 14:55):

@Patrick Werner I'll see what I can do;)

Jens Villadsen (Apr 01 2020 at 15:00):

FYI - credits goes to MedCom (danish Non-profit org represented by @Irene Zuschlag & @Ole Vilstrup here ) for funding my hours adding this functionality once they have received my proposal ;)

Patrick Werner (Apr 01 2020 at 15:00):

thanks sooo much. If you need a helping hand, i can invest some time as well.

Jens Villadsen (Apr 02 2020 at 11:46):

@Patrick Werner - I'll let you know when we begin

Jens Villadsen (Apr 08 2020 at 20:44):

@James Agnew @Patrick Werner initially (and as a safeguard) is it then safe to say that only instances of valuesets, codesystems and structuredefinitions should be considered?

Grahame Grieve (Apr 08 2020 at 20:44):

concept maps

Jens Villadsen (Apr 08 2020 at 20:45):

right

Patrick Werner (Apr 08 2020 at 20:45):

NamingSystems ?

Jens Villadsen (Apr 08 2020 at 20:45):

hmmm ... sure, ok

Grahame Grieve (Apr 08 2020 at 20:45):

y for sure

Grahame Grieve (Apr 08 2020 at 20:46):

longer term, GraphDefinition, StructureMap, but those are very use case dependent

Jens Villadsen (Apr 08 2020 at 20:46):

I see no reason to add examples from the NPM package ... unless explicitly stated so

Jens Villadsen (Apr 08 2020 at 20:47):

and since you are listening @Grahame Grieve - you might wanna get a hold on the people behind http://build.fhir.org/ig/hl7au/au-fhir-base/downloads.html

Patrick Werner (Apr 08 2020 at 20:47):

what do you think about SearchParameter?

Jens Villadsen (Apr 08 2020 at 20:47):

they link to the validator.pack file

Jens Villadsen (Apr 08 2020 at 20:48):

SearchParameter .... well ...

Jens Villadsen (Apr 08 2020 at 20:48):

it depends ...

Patrick Werner (Apr 08 2020 at 20:49):

examples not by default. Could imagine having this optionally or as a seperate paramter could alos be nice to prime servers for connectathons.

Jens Villadsen (Apr 08 2020 at 20:49):

yep

James Agnew (Apr 08 2020 at 20:50):

I would have thought SearchParameter for sure.. those are pretty commonly included in IGs no?

Patrick Werner (Apr 08 2020 at 20:50):

Thats why i was asking about SearchParameter. I personally would include it

Jens Villadsen (Apr 08 2020 at 20:50):

ok

Grahame Grieve (Apr 08 2020 at 20:51):

why are we loading things?

• because loading them changes internal server functionality (e.g. validator...)
• because we want them available on the FHIR API
  ?

Jens Villadsen (Apr 08 2020 at 20:51):

FYI: that also includes the use of https://smilecdr.com/docs/fhir_repository/search_parameters.html#unique-composite-search-parameters

James Agnew (Apr 08 2020 at 20:53):

@Grahame Grieve at this point by default those two things are one and the same in HAPI FHIR JPA server

I.e. if you load in a bunch of StructureDefinitions, they become available to the validator for validating other resource types, but they are also regular resources available via the REST endpoint

Another enhancement I definitely want to make is to allow people to separate those functions (which is how Vonk does it).. but that's not yet done

Jens Villadsen (Apr 08 2020 at 20:53):

SearchParameters that overload existing searchparameters in order to create indexes might not be a part of an IG

Grahame Grieve (Apr 08 2020 at 20:54):

I.e. if you load in a bunch of StructureDefinitions

indeed. But loading a bunch of new patients doesn't change internal functionality

Grahame Grieve (Apr 08 2020 at 20:54):

is there a list of which resources currently have internal meaning?

Jens Villadsen (Apr 08 2020 at 20:56):

gotta go - will follow up tomorrow ;)

James Agnew (Apr 08 2020 at 21:00):

Presently in HAPI it would be: StructureDefinition, CodeSystem, ValueSet, ConceptMap, SearchParameter. Possibly NamingSystem will get added to that list too at some point soon-ish.

I do really want to get that separated behaviour coded before too long though... truthfully to me it seems like weird behaviour to load in an NPM module and then have those resources show up through the rest endpoint. I think there should be a way perhaps through the RSET endpoint to interact with the NPM packages themselves at the package level (maybe a FHIR operation or two to manage them) but they shouldn't get exploded int your database eventually

Patrick Werner (Apr 08 2020 at 21:01):

i like the idea of the npm rest endpoint!

Patrick Werner (Apr 08 2020 at 21:02):

could even only provide the npm package name and start from there

Grahame Grieve (Apr 08 2020 at 21:02):

otoh it does make sense to be able to load the NpmPackage into the rest api if that's what you want

Grahame Grieve (Apr 08 2020 at 21:03):

Possibly NamingSystem will get added to that list too at some point soon-ish.

yeah because we'll be making changes for supporting v2 / CDA that make NamingSystem required inside the terminology system operations

James Agnew (Apr 08 2020 at 21:35):

I guess Subscription should be added to this list too. I haven't yet seen someone including a Subscription in an IG, but I bet it's a matter of time...

Grahame Grieve (Apr 08 2020 at 21:36):

I have seen them, but only as examples

Jens Villadsen (Apr 09 2020 at 06:35):

Grahame Grieve said:

otoh it does make sense to be able to load the NpmPackage into the rest api if that's what you want

I'll have that need

Jens Villadsen (Apr 09 2020 at 06:36):

I my environment the Terminology Server is more often updated than the IG

Jens Villadsen (Apr 09 2020 at 06:37):

and those updates are shipped as events other an MQ

Jens Villadsen (Apr 12 2020 at 21:13):

@James Agnew If you would like the IG to end up in some special NPM address, how is it supposed to work from HAPI CLI? A custom operation where an URL is passed to the system provider that then fetches the content at the specified URL?

Grahame Grieve (Apr 12 2020 at 21:14):

If you would like the IG to end up in some special NPM address

I don't understand this.

Jens Villadsen (Apr 12 2020 at 21:15):

James Agnew said:

Presently in HAPI it would be: StructureDefinition, CodeSystem, ValueSet, ConceptMap, SearchParameter. Possibly NamingSystem will get added to that list too at some point soon-ish.

I do really want to get that separated behaviour coded before too long though... truthfully to me it seems like weird behaviour to load in an NPM module and then have those resources show up through the rest endpoint. I think there should be a way perhaps through the RSET endpoint to interact with the NPM packages themselves at the package level (maybe a FHIR operation or two to manage them) but they shouldn't get exploded int your database eventually

Jens Villadsen (Apr 12 2020 at 21:16):

To me it sounds like that the content of the NPM package should be accessible through a REST interface that is decoupled from the regular FHIR REST interface

Jens Villadsen (Apr 12 2020 at 21:17):

eg. /IG/[packageName]/CodeSystem

James Agnew (Apr 12 2020 at 23:33):

TBH, not yet sure... Need to spend some time thinking about how storage will work in the JPA server before figuring out how the API works.

For now I definitely think it's fine to break the package apart and upload the individual components.

Jens Villadsen (Apr 13 2020 at 00:44):

Ok

Jens Villadsen (Apr 17 2020 at 18:44):

We have order to go. Commence of development is monday :)

Jens Villadsen (Apr 17 2020 at 19:27):

Jens Villadsen said:

My deadline for having this is the 14th of april - so may I suggest that you wait until that date and see what I end up with in a PR ;)

I meant 14th of May ;)

Jens Villadsen (Apr 20 2020 at 09:16):

@Martin Zacho Grønhøj is on the assignment now ;)

Jens Villadsen (Apr 20 2020 at 09:37):

@James Agnew (@Grahame Grieve @Patrick Werner ) We intend to do the following:
1) Extend the the DSTU3 and R4 systemproviders with a operation that takes a parameter containing an URL. When invoking that FHIR operation, the server downloads the content (package.tgz) from that given URL. The server then unzips the content and installs it into the database
2) Extend the hapi-fhir-starter project by adding a property in the properties file that contains a list of URLs to package.tgz files. On startup, the HAPI FHIR starter server fetches the files and proceeds as in 1)
3) Extend the HAPI FHIR CLI tool to support either using the operation from 1) or locally unzip a package.tgz file and upload the resources either one by one or in a single transaction

Please comment on our intended approach and feel free to point out any missing items.

Grahame Grieve (Apr 20 2020 at 11:47):

that takes a parameter containing an URL. When invoking that FHIR operation, the server downloads the content (package.tgz) from that given URL. The server then unzips the content and installs it into the database

from my point of view it would be more natural to just provide the package id, and then ask the PackageCacheManager to resolve it. No new download if already in the cache, and you need not be concerned about where it actually is - it will all auto-resolve if it's in the package system

Jens Villadsen (Apr 20 2020 at 12:02):

Grahame Grieve said:

that takes a parameter containing an URL. When invoking that FHIR operation, the server downloads the content (package.tgz) from that given URL. The server then unzips the content and installs it into the database

from my point of view it would be more natural to just provide the package id, and then ask the PackageCacheManager to resolve it. No new download if already in the cache, and you need not be concerned about where it actually is - it will all auto-resolve if it's in the package system

Don't you then assume that the package is installed at http://packages.fhir.org ?

Jens Villadsen (Apr 20 2020 at 12:02):

which might not be the case during development

Jens Villadsen (Apr 20 2020 at 12:04):

A way to support that could be with a custom url scheme if you are really keen on that repo - eg. fhir-ig://packageId

Grahame Grieve (Apr 20 2020 at 12:06):

indeed it might not be the case then. In which case a URL - or a file - would be good. but I expected that the majority use would be a published package

James Agnew (Apr 20 2020 at 13:11):

@Jens Villadsen I'd rather have the server not need to fetch external content if we can avoid it, since that's a pretty big security risk.

Couldn't the CLI tool read in the package and handle the "explosion" itself?

Jens Villadsen (Apr 20 2020 at 13:46):

@James Agnew The primary idea was that one could start a docker image with a parameter stating an IG url which then would be loaded into the HAPI FHIR instance at startup time.

Jens Villadsen (Apr 20 2020 at 13:47):

@Grahame Grieve 2 out of three IG's I work on are not listed there - I can't see why such a strict binding (to the package repo) should be necessary?

Jens Villadsen (Apr 20 2020 at 13:48):

@James Agnew It could indeed be solved with the CLI tool. The idea however was to stay independent of any other tools

Jens Villadsen (Apr 20 2020 at 13:49):

Ideally all that a user should use was a docker image and an URL

Jens Villadsen (Apr 20 2020 at 13:52):

But I guess the explosion could be encapsulated in the client. From your comments it becomes clear that we have different needs. @James Agnew does that mean that you won't accept the current design encapsulated in a SystemProvider in PR due to security risks?

James Agnew (Apr 20 2020 at 14:24):

Hmm, I see.

If you need this to be inside the server, let's at least put it in its own dedicated service and make the URL(s) be something that are pulled from configuration as opposed to being something that comes from user input (I guess DaoConfig is probably the right spot for it). That would mitigate the security concern mostly.

Patrick Werner (Apr 20 2020 at 14:39):

@Jens Villadsen @James Agnew
I like and need both approaches. A new FHIR package service endpoint (turned off by default in the jpastarter) and a cli exploding packages and manually uploading them to a REST endpoint.
For the latter a colleague of mine already started a small project, will be glad to add this to hapi-cli

Jens Villadsen (Apr 20 2020 at 14:52):

@Patrick Werner please let your colleague reach out to my colleague @Martin Zacho Grønhøj who will also be looking at the CLI

Jens Villadsen (Apr 20 2020 at 14:53):

@James Agnew daoconfig seems like a reasonable tradeoff

Kevin Mayfield (Apr 20 2020 at 14:53):

I've done something like this but I loaded in the packages during the JPAServer initialisation.
My aim was to set a validation package in HAPI and had another IG to control server (CapabilityStatement) behaviour.

Jens Villadsen (Apr 20 2020 at 14:55):

@Kevin Mayfield we also currently load it during servlet init for validation later on

Jens Villadsen (Apr 20 2020 at 14:56):

Im sure @Martin Zacho Grønhøj will add the fork url soon that will be the base for our PR

Grahame Grieve (Apr 20 2020 at 22:21):

2 out of three IG's I work on are not listed there

@Jens Villadsen I cannot get simplifier to connect up to the wider eco-system, so I'm running a back up package server at http://test.fhir.org/packages that has all the other packages

Grahame Grieve (Apr 20 2020 at 22:22):

the PackageCacheManager sorts this stuff automatically

Kevin Mayfield (Apr 21 2020 at 07:30):

@Jens Villadsen I'm hoping I can get some resources to support this. Maybe be concentrating on simplifier to HL7 IG conversion first but our aim (developers anyway) is to support validation.

Jens Villadsen (Apr 21 2020 at 08:00):

@Grahame Grieve I still think using a local version of a package should be possible

Jens Villadsen (Apr 21 2020 at 08:02):

@Kevin Mayfield validation is supported by HAPI FHIR out of the box once an IG is loaded into it

Jens Villadsen (Apr 21 2020 at 08:02):

as far as I remember

Grahame Grieve (Apr 21 2020 at 09:40):

using a local version of a package should be possible

yes I think that makes sense too

Jens Villadsen (Apr 21 2020 at 10:21):

@Grahame Grieve where does http://test.fhir.org/packages look for packages when it scans?

Grahame Grieve (Apr 21 2020 at 10:22):

the list is here:

Grahame Grieve (Apr 21 2020 at 10:23):

https://github.com/FHIR/ig-registry/blob/master/fhir-ig-list.json

Jens Villadsen (Apr 21 2020 at 10:24):

if thats the case where is my package then downloaded from? https://build.fhir.org/ig/hl7dk/dk-medcom/branches/master/failure/build.log

Jens Villadsen (Apr 21 2020 at 10:25):

http://packages.fhir.org/ ?

Grahame Grieve (Apr 21 2020 at 10:26):

dk.fhir.ig.dk-medcom-core#current would be downloaded on the fly from the current build.

Grahame Grieve (Apr 21 2020 at 10:26):

based on the last successful master build

Grahame Grieve (Apr 21 2020 at 10:26):

I see it says 0.8.0 in the package itself, but that only resolves if you publish it

Grahame Grieve (Apr 21 2020 at 10:27):

e.g. you register it in a package feed the server follows, or you push it to the server directly

Martin Zacho Grønhøj (Apr 21 2020 at 11:01):

Hi everyone, happy to join the heat ;-)

James Agnew said:

If you need this to be inside the server, let's at least put it in its own dedicated service and make the URL(s) be something that are pulled from configuration as opposed to being something that comes from user input (I guess DaoConfig is probably the right spot for it). That would mitigate the security concern mostly.

I've created af fork for development of this feature: https://github.com/mzgtrifork/hapi-fhir/tree/install-external-ig-package-at-startup
edit: renamed branch

Patrick Werner said:

For the latter a colleague of mine already started a small project, will be glad to add this to hapi-cli

Is the project accessible somewhere?

Patrick Werner (Apr 21 2020 at 11:17):

not yet. Will talk to my student today

Patrick Werner (Apr 21 2020 at 11:18):

But this fork is for adding a backend to hapi, not for extending the cli to open a package and upload it via rest?

Jens Villadsen (Apr 21 2020 at 11:27):

@Patrick Werner aint the cli part of the repo at https://github.com/jamesagnew/hapi-fhir/tree/master/hapi-fhir-cli ?

Martin Zacho Grønhøj (Apr 21 2020 at 11:50):

Patrick Werner said:

But this fork is for adding a backend to hapi, not for extending the cli to open a package and upload it via rest?

Maybe both, but on separate branches. I'm starting development of the backend on the branch install-external-ig-package-at-startup.

Patrick Werner (Apr 21 2020 at 12:51):

ok my colleague was more focused on the uploading of conformance resources of a package via REST

Patrick Werner (Apr 21 2020 at 12:53):

because some servers will check if the referenced CS of a VS is present i suggest to have a order in which conformance resources should be upped:

Patrick Werner (Apr 21 2020 at 12:54):

NamingSystem, CodeSystem, ValueSet, StructureDefinition, ConceptMap, SearchParameter, Subscription

Patrick Werner (Apr 21 2020 at 12:55):

Also for upping the StrucDefs i would try to resolve the dependencies of contained profiles and start with the "root" profile

Patrick Werner (Apr 21 2020 at 12:56):

So the snapshots of the uploaded profiles can be built properly on the server side (if needed and implemented)

Jens Villadsen (Apr 21 2020 at 13:07):

I agree - that is the same order we already use

Jens Villadsen (Apr 21 2020 at 13:08):

Building the hierachy can be a bit cumbersome

Jens Villadsen (Apr 21 2020 at 13:09):

And we haven't touched on the issue of IG dependencies and hierachies

Grahame Grieve (Apr 21 2020 at 19:53):

Building the hierachy can be a bit cumbersome

especially since there are circular dependencies. (not in the heirarchy, but in the profile references). So don't worry about that, and have a load all before you generate snapshots

Jens Villadsen (Apr 22 2020 at 09:47):

@Grahame Grieve aren't snapshots on the profiles generated by the IG Publisher?

Grahame Grieve (Apr 22 2020 at 10:22):

Yes the IG publisher does and any packages from it will have snapshots, but other sources don’t necessarily produce snapshots

Jens Villadsen (Apr 22 2020 at 10:23):

which sources are we talking about?

Grahame Grieve (Apr 22 2020 at 12:20):

I believe that (some?) simplifier packages don't include snapshots. Or is it forge? And FSH doesn't produce snapshots

Patrick Werner (Apr 22 2020 at 12:25):

Grahame Grieve said:

Building the hierachy can be a bit cumbersome

especially since there are circular dependencies. (not in the heirarchy, but in the profile references). So don't worry about that, and have a load all before you generate snapshots

I tried to understand how profiles could have circular references and can't come up with an example. Could someone push me in the right direction please?

Grahame Grieve (Apr 22 2020 at 12:28):

the validator tests cases are derived from the real world case - two patients cross-linked, with a profile on primary and a profile on the secondary

Patrick Werner (Apr 22 2020 at 12:44):

ah ok. Thats ok. Was thinking about profiles deriving from another in a circle :octopus:

Jens Villadsen (Apr 22 2020 at 13:15):

OK - producing snapshots shouldn't be much of an issue when using HAPI FHIR and its dependencies (the core lib)

Martin Zacho Grønhøj (Apr 23 2020 at 11:28):

I've forked hapi-fhir-jpaserver-starter, and planning to begin a top-down approach:

• Adding configuration fields "igResourcesURL" in HapiProperties of jpaserver-starter and DaoConfig of jpaserver-api
• Adding a service "igResourcesInstaller" in jpaserver-base (under ca.uhn.fhir.jpa.dao or maybe a dedicated namespace?), which

1. Downloads the IG package and dependent packages/ loads them from disk (possibly using Grahams NpmPackage abstraction), if the URL is set.
2. Generates snapshots of StructureDefinitions if necessary.
3. Saves all resources of type NamingSystem, CodeSystem, ValueSet, StructureDefinition (snapshots), ConceptMap, SearchParameter and Subscription to the database.

I intend to invoke the service when the ApplicationReadyEvent is broadcastet.

Comments?

Ward Weistra (Apr 23 2020 at 13:06):

@Jens Villadsen Cool stuff :up: I assume you've found the package server documentation? https://simplifier.net/docs/package-server. You will indeed encounter packages with dependencies, which you should also be able to find on packages.fhir.org.
Indeed, having the options to point to packages which are not (yet) in the FHIR package server makes a lot of sense.

Patrick Werner (Apr 23 2020 at 13:53):

@Ward Weistra @Grahame Grieve will there be a joined package server at some point?
I think the success of npm/maven is due to a central repository.

Ward Weistra (Apr 23 2020 at 13:56):

@Patrick Werner packages.fhir.org IS the central repository! :open_mouth: It brings together the packages from Simplifier, HL7, FHIR Foundation and all the feeds in the feeds object of https://github.com/FHIR/ig-registry/blob/master/fhir-ig-list.json.

Ward Weistra (Apr 23 2020 at 13:58):

That doesn't mean the infrastructure is all perfect yet. We're still running the import from the other feeds by hand because we keep running into issues with the feeds and need a good way to automate the reporting and handling of them.

Patrick Werner (Apr 23 2020 at 13:58):

ah good to know! Thanks Ward.

Ward Weistra (Apr 23 2020 at 14:00):

It also focuses on stable packages so far. There's no central solution for the #dev, #current and x.x.x-alphabeta versions yet.

Patrick Werner (Apr 23 2020 at 14:01):

Grahame Grieve said:

2 out of three IG's I work on are not listed there

Jens Villadsen I cannot get simplifier to connect up to the wider eco-system, so I'm running a back up package server at http://test.fhir.org/packages that has all the other packages

was asking because of this statement @Ward Weistra

Ward Weistra (Apr 23 2020 at 14:09):

Yeah, I assume Grahame is referring to the package server infrastructure not yet in automated fashion picking up new feeds in the feeds object or scheduled importing of the packages? Which are both things we are working towards. We just ran the importer manually again, on all feeds in there now.

Patrick Werner (Apr 23 2020 at 14:28):

sounds great. Thanks for the explanation

Jens Villadsen (Apr 23 2020 at 15:09):

What is behind http://packages.fhir.org? A regular Nexus instance?

Jens Villadsen (Apr 23 2020 at 16:05):

Since it is an npm repo, why not use a regular Nexus for it?

Jens Villadsen (Apr 23 2020 at 16:06):

Or am i a total ignorant?

James Agnew (Apr 23 2020 at 16:47):

possibly using Grahams NpmPackage abstraction

We should defintely be using that... Dependency management is complex, no sense reinventing the wheel

Jens Villadsen (Apr 23 2020 at 16:57):

Speaking of ... @Grahame Grieve did you have a look at how sonatype handles npm? (https://github.com/sonatype/nexus-public/tree/master/plugins/nexus-repository-npm/src/main/java/org/sonatype/nexus/repository/npm/internal)

Grahame Grieve (Apr 23 2020 at 20:45):

@Ward Weistra thanks for finally uploading from the package feeds. How often will this run now?

Jens Villadsen (Apr 23 2020 at 20:51):

cURL'ing https://packages.fhir.org/ leads to HTTP 404

Jens Villadsen (Apr 23 2020 at 20:52):

anyways - is it a regular npm repo running behind the curtains? @Ward Weistra

Grahame Grieve (Apr 23 2020 at 20:52):

I'm getting that too with POSTMan - GET http://packages.fhir.org/catalog?fhirversion=STU3 with Accept: application/json, but just http://packages.fhir.org/catalog?fhirversion=STU3&prerelease=false works in a browser

Grahame Grieve (Apr 23 2020 at 20:54):

@Ward Weistra this is the list of packages I get for STU3 with prerelease = false:

SliceBugTest
Furoretestproject
Simplifier.Core.Stu3.DataTypes
Simplifier.Core.STU3.ValueSets
Simplifier.Core.STU3.CodeSystems
Simplifier.Core.STU3.Extensions
Simplifier.Core.STU3
Demo.National.PatientCare
Demo.Regional.PatientCare
gefyra.sandbox
ufp.core
dummy.first
MySimplifierProject.core
MySimplifier.Project
ProvincialClientRegi.core
PGO.02
irrs.pkg
cihi.irrs.pkg
cihi.irrs
private.package.test
test.my.first.package
TestProjekt2.test
Simplifier.Others.Stu3
nictiz.fhir.nl.stu3.zib2017
Simplifier.Core.STU3.Resources
Simplifier.Core.STU3.ProfilesOthers
HL7Norway.STU3.no.basis
kbv.fhir.vos
testontwikkel.package
testontwikkelz.package
dvmd.kdl.stu3.2019
PCR.Core.STU3
MyDemoProject.core
auw.dummy
MyDemo.core
UKKoeln.nNGM.RegistrationForm.IG
aws.dummy
dev.cihub.ClinicalSite.org
dev.cihub.CancerCareRecord
TestMenno.project
Testproject.test
TestProjectArnim.test
Mirjamstestproject.test
acme.base
hl7.fhir.r2b.core
hl7.fhir.r2b.corexml
hl7.fhir.r2b.examples
hl7.fhir.r2b.expansions
hl7.fhir.r3.core
hl7.fhir.r3.corexml

Jens Villadsen (Apr 23 2020 at 20:54):

jvi@Jenss-MBP-2 ~ % curl 'http://packages.fhir.org/catalog?fhirversion=STU3'
[{"Name":"SliceBugTest","Description":null,"FhirVersion":"STU3"},{"Name":"Furoretestproject","Description":null,"FhirVersion":"STU3"},{"Name":"Simplifier.Core.Stu3.DataTypes","Description":"These are the FHIR Core STU3 Data types. This is a temporary package until the offical HL7 FHIR Core Packages are released and published on Simplifier.","FhirVersion":"STU3"},{"Name":"Simplifier.Core.STU3.ValueSets","Description":"These are the FHIR Core STU3 ValueSets. This is a temporary package until the offical HL7 FHIR Core Packages are released and published on Simplifier.","FhirVersion":"STU3"},{"Name":"Simplifier.Core.STU3.CodeSystems","Description":"These are the FHIR Core STU3 Code Systems. This is a temporary package until the offical HL7 FHIR Core Packages are released and published on Simplifier.","FhirVersion":"STU3"},{"Name":"Simplifier.Core.STU3.Extensions","Description":"These are the FHIR Core STU3 Extensions. This is a temporary package until the offical HL7 FHIR Core Packages are released and published on Simplifier.","FhirVersion":"STU3"},{"Name":"Simplifier.Core.STU3","Description":"This is a meta package that contains the FHIR Core STU3 Specification. It does not contain resources, but it has dependencies to all packages needed to get the full specification. ","FhirVersion":"STU3"},{"Name":"Demo.National.PatientCare","Description":"Demo project to show the use of packages","FhirVersion":"STU3"},{"Name":"Demo.Regional.PatientCare","Description":"Demo project to show the use of packages ","FhirVersion":"STU3"},{"Name":"gefyra.sandbox","Description":"Release version","FhirVersion":"STU3"},{"Name":"ufp.core","Description":"The United Federation of Planet (UFP) FHIR Implementation Guide of intergalactic scope","FhirVersion":"STU3"},{"Name":"dummy.first","Description":"Dummy package testing version 2","FhirVersion":"STU3"},{"Name":"MySimplifierProject.core","Description":"My Simplifier Project","FhirVersion":"STU3"},{"Name":"MySimplifier.Project","Description":"My Simplifier Project","FhirVersion":"STU3"},{"Name":"ProvincialClientRegi.core","Description":"FHIR based Implementation Guide for provincial CR system","FhirVersion":"STU3"},{"Name":"PGO.02","Description":"WEM ZWConnect","FhirVersion":"STU3"},{"Name":"irrs.pkg","Description":"IRRS Project","FhirVersion":"STU3"},{"Name":"cihi.irrs.pkg","Description":"IRRS Project","FhirVersion":"STU3"},{"Name":"cihi.irrs","Description":"IRRS Project","FhirVersion":"STU3"},{"Name":"private.package.test","Description":"Testing private packages","FhirVersion":"STU3"},{"Name":"test.my.first.package","Description":"This is my test description, see if it gets in the package.manifest","FhirVersion":"STU3"},{"Name":"TestProjekt2.test","Description":"Test Projekt","FhirVersion":"STU3"},{"Name":"Simplifier.Others.Stu3","Description":"Profiles that are part of the core specification but are not base profiles","FhirVersion":"STU3"},{"Name":"nictiz.fhir.nl.stu3.zib2017","Description":"Nictiz NL package of FHIR STU3 conformance resources for HCIM Release 2017. Includes MedMij and HL7 NL.\r\n\r\nHCIMs: https://zibs.nl/wiki/HCIM_Release_2017(EN)","FhirVersion":"STU3"},{"Name":"Simplifier.Core.STU3.Resources","Description":"These are the STU3 Core Profile StructureDefinitions describing resources.\r\n\r\n","FhirVersion":"STU3"},{"Name":"Simplifier.Core.STU3.ProfilesOthers","Description":"Profiles that are part of the core specification but are not base profiles","FhirVersion":"STU3"},{"Name":"HL7Norway.STU3.no.basis","Description":"Basis profiles developed by HL7 Norway.","FhirVersion":"STU3"},{"Name":"kbv.fhir.vos","Description":"Beispielprojekt Version 1.1","FhirVersion":"STU3"},{"Name":"testontwikkel.package","Description":"Mijn eigen test en ontwikkelproject","FhirVersion":"STU3"},{"Name":"testontwikkelz.package","Description":"Mijn eigen test en ontwikkelproject","FhirVersion":"STU3"},{"Name":"dvmd.kdl.stu3.2019","Description":"Publikation der Terminologie-Ressourcen für die Nutzung der Klinische Dokumentenklassen-Liste (KDL) des DVMD","FhirVersion":"STU3"},{"Name":"PCR.Core.STU3","Description":"ehealth Ontario","FhirVersion":"STU3"},{"Name":"MyDemoProject.core","Description":"My demo project","FhirVersion":"STU3"},{"Name":"auw.dummy","Description":"Dummy-Projekt zur Erstellung des AuW Packages","FhirVersion":"STU3"},{"Name":"MyDemo.core","Descrip

Grahame Grieve (Apr 23 2020 at 20:54):

how do we clean this up? we don't want to see non real packages..

Grahame Grieve (Apr 23 2020 at 21:02):

I didn't look at nexus. should I? What would I look for?

Jens Villadsen (Apr 23 2020 at 21:03):

parsing for starters: https://github.com/sonatype/nexus-public/blob/master/plugins/nexus-repository-npm/src/main/java/org/sonatype/nexus/repository/npm/internal/NpmPackageParser.java

Jens Villadsen (Apr 23 2020 at 21:05):

resolving would be next

Jens Villadsen (Apr 23 2020 at 21:06):

since its an open source repo that handles (among others) npm packages I would think that they have all the logic needed

Grahame Grieve (Apr 23 2020 at 21:08):

that might have helped when I was starting, but not so much now

Jens Villadsen (Apr 23 2020 at 21:09):

actually - the whole resolving thing - shouldn't that have been carried out by npm repo?

Grahame Grieve (Apr 23 2020 at 21:10):

what resolving thing?

Jens Villadsen (Apr 23 2020 at 21:10):

when you fetch a leaf IG, you traverse one node up the tree right?

Grahame Grieve (Apr 23 2020 at 21:11):

a leaf IG?

Jens Villadsen (Apr 23 2020 at 21:12):

the most outer IG in a tree of IG's

Grahame Grieve (Apr 23 2020 at 21:13):

so you mean, based on dependencies?

Jens Villadsen (Apr 23 2020 at 21:13):

yes

Grahame Grieve (Apr 23 2020 at 21:14):

so I fetch just the package asked for in the package manager when the package is loaded. The IG loader will walk the dependencies loading the packages, which will be fetched if they are needed

Jens Villadsen (Apr 23 2020 at 21:14):

yeah ok .... well ... nevermind ... hope you didn't spend too much time on that part

Jens Villadsen (Apr 23 2020 at 21:15):

I still would like to know though, if there's a regular npm repo behind https://packages.fhir.org/

Grahame Grieve (Apr 23 2020 at 21:15):

some time. Initially, we talked about just asking the repository for the package and all it's dependencies, but the message I have been getting is that download time matters, and why fetch what you already have?

Jens Villadsen (Apr 23 2020 at 21:18):

sure ... there's no reason to do that

Jens Villadsen (Apr 23 2020 at 21:18):

but that could also be solved ... like npm and maven does it

Jens Villadsen (Apr 23 2020 at 21:18):

maven checks can check on SHA's if I remember correct

Jens Villadsen (Apr 23 2020 at 21:19):

npm probably does the same of some sort

Jens Villadsen (Apr 23 2020 at 21:19):

so the repo gives you back a list of resolved dependencies with links, versions and SHA's ...

Jens Villadsen (Apr 23 2020 at 21:20):

then you can decide for yourself about your caching strategy

Jens Villadsen (Apr 23 2020 at 21:21):

@Ward Weistra an observation - when going to http://packages.fhir.org then you are redirected to https://simplifier.net/packages - but you need to be logged in, in order to download .... that seems fishy

Grahame Grieve (Apr 23 2020 at 21:22):

you don't need to be logged to download from packages.fhir.org

Jens Villadsen (Apr 23 2020 at 21:23):

image.png

Jens Villadsen (Apr 23 2020 at 21:24):

then why does it say "Login to download" @Grahame Grieve

Grahame Grieve (Apr 23 2020 at 21:27):

because you're not using the API. But @Ward Weistra people shouldn't have to login to use the web site either, for the package registry

Jens Villadsen (Apr 23 2020 at 21:28):

I cant use an API if curl tells me 404

Jens Villadsen (Apr 23 2020 at 21:29):

the circle is closed. If regular tools are not usable and the UI locks me out it doesn't work

Jens Villadsen (Apr 23 2020 at 21:31):

If I had got a 302 in cURL it would have been sufficient to me

Patrick Werner (Apr 24 2020 at 06:33):

@Jens Villadsen the api doc is here: https://simplifier.net/docs/package-server/Search

Patrick Werner (Apr 24 2020 at 06:34):

Because we want to make package search as useful as possible for the FHIR usecase, we are not following the NPM standard, but we implement a lean endpoint that can be used for intellisense (dropdown) and a search that can deal with the most important FHIR relevant search queries: canonicals and FHIR versions.

Patrick Werner (Apr 24 2020 at 06:34):

Is this a defined interface for fhir packages?

Grahame Grieve (Apr 24 2020 at 06:35):

http://docs.simplifier.net/simplifier/simplifierPackages.html

Ward Weistra (Apr 24 2020 at 08:10):

Good morning!

@Jens Villadsen While the package spec is NPM inspired, it isn't fully compatible (no single root folder, lower case package names, not handling dependencies the same way). Also, from what I hear, NPM is not a nicely defined standard, but more a bag of conventions that you have to reengineer, making it harder. There's been an earlier conversation on this, without concrete decisions afaik: https://chat.fhir.org/#narrow/stream/179280-fhir.2Finfrastructure-wg/topic/NPM.20Package.20Spec/near/181766179

I would welcome FHIR-I proposals on what aligning fully would mean, while maintaining any FHIR-needed extras like defining and searching on FHIR version. I do see the benefit in people being able to reuse NPM servers. Do keep in there are already a large number of packages, so we will need package-spec versioning too (personally I would think this spec should someday be tied to the rest of the FHIR spec versioning, so e.g. have the proposals for FHIR spec alignment with NPM work for FHIR package spec in R5).
The Simplifier package infrastructure (from which the FHIR.org package server benefits) doesn't use an existing NPM server behind it. We do have some instructions on installing with NPM, but I've been told dependencies are not coming along (as discussed above) and NPM search is not implemented.

Simplifier's priority has been on making sure the FHIR packages endpoint does what it needs for the FHIR community. This has been documented on https://simplifier.net/docs/package-server and is fully open, just curl https://packages.fhir.org/catalog instead (see Search) to get the packages, query https://packages.fhir.org/<package-name> for the versions and query https://packages.fhir.org/<package-name>/<package-version> to download. The find the dependencies in package.json and download those with their URLs. Or use Torinox to download them all.
If you're visiting https://packages.fhir.org (or https://packages.simplifier.net) in the browser we redirect to the Simplifier UI for searching packages. This is a better experience than a 404 for someone looking for a package, right?

Ward Weistra (Apr 24 2020 at 08:28):

@Grahame Grieve We've discussed the topic of prerelease before here.

• http://packages.fhir.org/catalog?fhirversion=STU3&prerelease=false will only show you packages that have a version that is not a prerelease. According to semver versioning 0.0.1-anything is a pre-release package, 0.0.1 is not. That seems to work correctly, unless you find packages in that list that ONLY have prerelease version.
• Secondly, people could of course be marking things 'ready for release' which, let's say, are not of the level of quality of hl7.fhir.us.core. But in that sense Simplifier is the Youtube of FHIR data models (user-generated content) compared to the more tv-station approach (vetted content) of the HL7/FHIR Foundation feeds. There are two likely improvements to this: Allow the user to say (in package server and also the FHIR registry on top) from which channel they want the content to come (e.g only from 'FHIR Foundation Guides' feed or not from 'FHIR Foundation Support Packages' feed. And ranking the best packages (e.g. most referred to) higher to filter out the weeds.

Ward Weistra (Apr 24 2020 at 08:47):

(Side note: we will very soon unlist all the Simplifier.Core.* packages now the real deal hl7.fhir.*.core is there :tada:)

Jens Villadsen (Apr 24 2020 at 09:59):

I would welcome FHIR-I proposals on what aligning fully would mean, while maintaining any FHIR-needed extras like defining and searching on FHIR version. I do see the benefit in people being able to reuse NPM servers. Do keep in there are already a large number of packages, so we will need package-spec versioning too (personally I would think this spec should someday be tied to the rest of the FHIR spec versioning, so e.g. have the proposals for FHIR spec alignment with NPM work for FHIR package spec in R5).

@Ward Weistra sounds like we are aligned ;)

Jens Villadsen (Apr 24 2020 at 10:04):

I'm no npm master ninja - I would just like to not add frustration for people used to be using npm packages and repos. That would be my 5 cents.

James Agnew (Apr 24 2020 at 13:27):

According to semver versioning 0.0.1-anything is a pre-release package, 0.0.1 is not

One of the trends I hate right now in the Java ecosystem is that many big libraries have decided to deviate from this. Spring labels its releases 1.2.3.FINAL, hibernate uses 1.2.3.RELEASE, etc.

Martin Zacho Grønhøj (Apr 24 2020 at 13:29):

Hi guys, a couple of questions:

1) When installing an IG which was loaded from a file on the disk, how should its dependencies be resolved?

Use the PackageCacheManager to fetch the dependencies from the global cache, or the repository at packages.fhir.org if a dependency is not in cache?

2) What should the behavior be, if an IG or (any of) its dependencies cannot be resolved? A graceful shutdown or startup without installed packages?

3) When generating snapshots of IGs with dependencies, I assume that any StructureDefinitions in any of the dependencies, that does not already contain a snapshot, should also have its snapshot generated.

Should this happen in a specific order, eg. snapshots of dependencies should be generated before snapshots of the dependent package?

Grahame Grieve (Apr 24 2020 at 21:32):

@Ward Weistra :

While the package spec is NPM inspired, it isn't fully compatible

I don't believe that this is true

no single root folder

I reworked all the packages to only have a single root folder, and we wrote this into the rules

lower case package names

That is a convention that we made, though I see it's not a convention that Simplifier follows. None of my package infrastructure depends on following it, though all the HL7 authoring rules enforce it

not handling dependencies the same way

Other than restricting the possible values for version, I thought it was the same

Grahame Grieve (Apr 24 2020 at 21:35):

people could of course be marking things 'ready for release' which, let's say, are not of the level of quality of hl7.fhir.us.core.

Well, there's levels in between. the package server should publish packages when people are ready to publish them, irrespective of the workflow of quality. But I think that simplifier projects should not be allowed to publish packages like dummy.first - they should be at least forced to namespace. And projects like that should be removed from being published.

Grahame Grieve (Apr 24 2020 at 21:35):

but whatever, we have to figure out how to tidy this up ASAP

Grahame Grieve (Apr 24 2020 at 21:38):

@Martin Zacho Grønhøj

1 - I'm not sure what difference you intend there - my answer is yes to both unless I misunderstand what you mean by global cache

2. that's an application specific question. Are you asking about HAPI installation? I would expect that in that context, you'd ask the user

3. yes and yes. Though how I manage this in the core tools is :

• Load all profiles
• iterate them all generating the snapshots
• generate any missing snapshots in dependencies recursively
• keep a watch on the stack to guard against stack crash (and produce a better error message)

Martin Zacho Grønhøj (Apr 25 2020 at 00:41):

Grahame Grieve said:

1 - I'm not sure what difference you intend there - my answer is yes to both unless I misunderstand what you mean by global cache

Global cache meaning the local cache on the machine.

Grahame Grieve said:

2. that's an application specific question. Are you asking about HAPI installation? I would expect that in that context, you'd ask the user

Yes, in hapi-fhir-jpaserver-starter for instance. IMO it's sensible to have the default behavior be a shutdown, to inform the user of the error, but also allow a flag to be set, if the user wants to run the application anyways. So I guess my question is: Does it make sense to run a server with a package that's missing its dependencies?

Grahame Grieve said:

3. yes and yes. Though how I manage this in the core tools is :

Could you provide me with a link to the source code of this? I can't seem to find it.

Martin Zacho Grønhøj (Apr 27 2020 at 07:33):

Martin Zacho Grønhøj said:

Could you provide me with a link to the source code of this? I can't seem to find it.

@Grahame Grieve

Grahame Grieve (Apr 27 2020 at 09:36):

well, it's spread across the core code, but most of it's in org.hl7.fhir.r5.context.BaseWorkerContext

Jens Villadsen (Apr 27 2020 at 09:42):

aint much of it wrapped in https://github.com/hapifhir/org.hl7.fhir.core/blob/4316c81f7ef578510c0b255663dd06ff64b9b56e/org.hl7.fhir.r5/src/main/java/org/hl7/fhir/r5/conformance/ProfileUtilities.java#L463 @Grahame Grieve ?

Grahame Grieve (Apr 27 2020 at 09:46):

that's the logic for generating snapshots. the logic for managing the generation is in the BaseContext

Martin Zacho Grønhøj (Apr 28 2020 at 12:44):

James Agnew said:

I guess Subscription should be added to this list too. I haven't yet seen someone including a Subscription in an IG, but I bet it's a matter of time...

In our current way of uploading the resources from the IG to our server, for each resource R1 we do the following:

• search for any resource R2 with the same URL (profile?)
• if found, assign R2.id to R1 and update R1
• otherwise, create R1

The other resource types (NamingSystem, CodeSystem, ValueSet, StructureDefinition, ConceptMap and SearchParameter) all extend MetadataResource and thus have a searchable URL, whereas Subscription does not, since it only extends DomainResource.

What parameter should be used for searching for existing subscriptions, when uploading subscriptions from an IG? I guess the ID could be used, but since it's not globally unique like the MetadataResource URL, then the IG upload could introduce duplicate data.

Jens Villadsen (Apr 28 2020 at 14:12):

Martin Zacho Grønhøj said:

The other resource types (NamingSystem, CodeSystem, ValueSet, StructureDefinition, ConceptMap and SearchParameter) all extend MetadataResource and thus have a searchable URL, whereas Subscription does not, since it only extends DomainResource.

What parameter should be used for searching for existing subscriptions, when uploading subscriptions from an IG? I guess the ID could be used, but since it's not globally unique like the MetadataResource URL, then the IG upload could introduce duplicate data.

Subscriptions should be unique by their Identifier, right? @James Agnew @Grahame Grieve

James Agnew (Apr 28 2020 at 14:19):

Subscriptions don't have an identifier, so I think you'd have to use the ID. It's not ideal, but there isn't really a better way I can see..

Jens Villadsen (Apr 28 2020 at 17:08):

@James Agnew what is http://build.fhir.org/subscription-definitions.html#Subscription.identifier that then?

Jens Villadsen (Apr 28 2020 at 17:09):

oh.... that is only on bleeding edge ...

Jens Villadsen (Apr 28 2020 at 17:09):

:drooling:

James Agnew (Apr 28 2020 at 17:12):

Ah, so it is.

Jens Villadsen (Apr 30 2020 at 08:51):

Martin Zacho Grønhøj said:

I've forked hapi-fhir-jpaserver-starter

@Sean McIlvenna FYI

Jens Villadsen (Apr 30 2020 at 08:55):

@James Agnew - Can you be persuaded to have a look at the upcoming PR that @Martin Zacho Grønhøj hopefully will make in the near future in order to get this into 5.0.0? or has that deadline already passed?

Grahame Grieve (Apr 30 2020 at 08:55):

hopefully not - I have a breaking change to commit for R5 that I haven't got around to yet

Jens Villadsen (Apr 30 2020 at 08:59):

@Grahame Grieve is that a commit to something in https://github.com/hapifhir/org.hl7.fhir.core ? AFAIK, @Martin Zacho Grønhøj's work will concern loading of Igs in versions DSTU3 and R4 based on the transitive dependency that HAPI FHIR runs on

Jens Villadsen (Apr 30 2020 at 09:00):

@Grahame Grieve so (how) will your commit be able to break that?

Martin Zacho Grønhøj (Apr 30 2020 at 09:29):

PR to hapi-fhir should be ready in a few days. It uses the npm utilities from org.hl7.fhir.utilities.cache and models from org.hl7.fhir.<version>.model.

Grahame Grieve (Apr 30 2020 at 10:46):

I'm fixing some property accessors in the generated model classes

Jens Villadsen (Apr 30 2020 at 19:03):

@Grahame Grieve whats your ETA for completion of that fixing?

Grahame Grieve (Apr 30 2020 at 21:21):

I don't know. soon, but people keep dropping other urgent stuff on me

Jens Villadsen (Apr 30 2020 at 21:23):

image.png

Grahame Grieve (Apr 30 2020 at 21:24):

yeah well that's pretty topical these days

Jens Villadsen (Apr 30 2020 at 21:25):

image.png

Martin Zacho Grønhøj (May 01 2020 at 18:38):

PR for hapi-fhir is ready: https://github.com/jamesagnew/hapi-fhir/pull/1826

Let me know what you think/ improvements etc.

Jens Villadsen (May 01 2020 at 19:34):

@James Agnew

Martin Zacho Grønhøj (May 07 2020 at 20:28):

I've implemented a quick reference implementation of how to use the new IgInstallerSvc in my hapi-fhir-jpaserver-starter fork using a local built of the branch from my PR. It is still under review.

I've build a docker-image, so you can try running an instance of hapi-fhir-starter with an uploaded IG of your choice:

docker pull mzgtrifork/hapi-fhir-jpaserver-starter:0.0.1

install an implementation guide using a URL:

docker run \
       --name=hapi-fhir \
       --rm \
       -p 8080:8080 \
       -e HAPI_IMPLEMENTATION_GUIDE_URL=https://build.fhir.org/ig/hl7dk/dk-medcom/package.tgz \
       mzgtrifork/hapi-fhir-jpaserver-starter:0.0.1

or to fetch a package, say NHSD.Assets.STU3#1.0.0, from packages.fhir.org and install at startup:

docker run \
       --name=hapi-fhir \
       --rm \
       -p 8080:8080 \
       -e HAPI_FHIR_VERSION=DSTU3 \
       -e HAPI_IMPLEMENTATION_GUIDE_ID=NHSD.Assets.STU3 \
       -e HAPI_IMPLEMENTATION_GUIDE_VERSION=1.0.0 \
       mzgtrifork/hapi-fhir-jpaserver-starter:0.0.1

So far so good - a lot of packages on packages.fhir.org causes various errors. It seems like Simplifier doesn't validate packages according to the spec? @James Agnew @Ward Weistra

Jens Villadsen (May 07 2020 at 21:14):

@Grahame Grieve @Patrick Werner have a look at the docker image and let us know what you think

Grahame Grieve (May 07 2020 at 21:15):

which packages cause errors? We're still working on the packages ...

Jens Villadsen (May 07 2020 at 21:38):

https://simplifier.net/packages/de.medizininformatikinitiative.kerndatensatz.person/0.9.1 seems to be broken

Jens Villadsen (May 07 2020 at 21:39):

https://simplifier.net/packages/ehelse.fhir.no.grunndata/2.2.4 seems to be broken

Grahame Grieve (May 07 2020 at 21:43):

the packages sourced from Simplifier... yes... lots of problems in those

Jens Villadsen (May 07 2020 at 21:44):

well ... how should one be able to tell where they come from?

Jens Villadsen (May 07 2020 at 21:44):

https://simplifier.net/packages/hl7.fhir.uv.ips/0.3.0 is broken

Grahame Grieve (May 07 2020 at 21:45):

oh why?

Jens Villadsen (May 07 2020 at 21:45):

FHIR versions mismatch (expected <=4.0.1, package uses 3.2.0)

Jens Villadsen (May 07 2020 at 21:48):

wait ...

Jens Villadsen (May 07 2020 at 21:50):

no ... ips is fine

Jens Villadsen (May 07 2020 at 21:54):

us core (3.1.0) however is broken: ConceptMap[url='http://hl7.org/fhir/us/core/ConceptMap/ndc-cvx'] contains at least one group without a value in ConceptMap.group.source

Grahame Grieve (May 07 2020 at 21:55):

why would you call that broken?

Jens Villadsen (May 07 2020 at 21:56):

because it means that the concept map cannot be stored

Grahame Grieve (May 07 2020 at 21:56):

i don't follow that. The specification doesn't require it, and the author didn't provide it, so why is it the package that is broken?

Jens Villadsen (May 07 2020 at 21:58):

let me rephrase: the implementation guide contains resources that cannot be stored using the java reference implementation. whether its a bug in the reference implementation or in the IG, i cannot say for now

Grahame Grieve (May 07 2020 at 21:59):

since the package was built using the reference implementation, I think that you should look elsewhere for the source of the issue

Jens Villadsen (May 07 2020 at 22:00):

by reference implementation I mean the java reference implementation (HAPI) - not the ig tooling

Grahame Grieve (May 07 2020 at 22:02):

HAPI is not the reference implementation. The reference implementation is the HAPI FHIR Core. So no doubt the problem is some business rule in HAPI somewhere, which I think it erroneous. but in this case it's not the package that is broken

Jens Villadsen (May 07 2020 at 22:02):

right - the loading of it is

James Agnew (May 07 2020 at 22:03):

That is a rule in the current HAPI FHIR JPA server - There's a not-null on a table that means we can't store ConceptMap groups without a source

I think Diederik assumed that was a safe rule to have when he coded it but sounds like it isn't

Grahame Grieve (May 07 2020 at 22:04):

the documentation says about ConceptMap.group.source:

This is not needed if the source value set is specified and it contains concepts from only a single system

Jens Villadsen (May 07 2020 at 22:04):

run

docker run \
       --name=hapi-fhir \
       --rm \
       -p 8080:8080 \
       -e HAPI_FHIR_VERSION=R4 \
       -e HAPI_IMPLEMENTATION_GUIDE_ID=hl7.fhir.us.core \
       -e HAPI_IMPLEMENTATION_GUIDE_VERSION=3.1.0 \
       mzgtrifork/hapi-fhir-jpaserver-starter:0.0.1

and you can see for yourself

James Agnew (May 07 2020 at 22:06):

This is not needed if the source value set is specified and it contains concepts from only a single system

Heh, that feels like a weird bit of context conduction to me, but if it's allowed I'll ticket it.

Ward Weistra (May 08 2020 at 08:43):

@Martin Zacho Grønhøj @Jens Villadsen Thanks for calling out issues. I see the two packages Jens mentions are part of a group of packages having duplicate files in there, which we are working to resolve.
Any other specific issues you are finding so I can prevent them from happening?

Martin Zacho Grønhøj (May 13 2020 at 13:24):

The bug I've reported here is pretty common and occurs for instance in acme.base#0.0.5 and de.medizininformatikinitiative.kerndatensatz.fall#0.9.0, but it's supposedly been fixed.

The package hl7.fhir.au.rcpa#0.1.0 throws a parsing error:

ca.uhn.fhir.parser.DataFormatException: Unknown element 'comments' found during parse

Don't know if its an error in the parser or the package data (probably the former).

The package nictiz.fhir.nl.stu3.questionnaires#1.0.0 uses X-Range NPM semantic versioning for the dependency nictiz.fhir.nl.stu3.zib2017#1.3.x which cannot be resolved using the PackageCacheManager#loadPackage:

org.hl7.fhir.exceptions.FHIRException: Unable to resolve package id nictiz.fhir.nl.stu3.zib2017

    at org.hl7.fhir.utilities.cache.PackageCacheManager.fetchTheOldWay(PackageCacheManager.java:772)
    at org.hl7.fhir.utilities.cache.PackageCacheManager.loadFromPackageServer(PackageCacheManager.java:335)
    at org.hl7.fhir.utilities.cache.PackageCacheManager.loadPackage(PackageCacheManager.java:601)

The fhir package spec says it's allowed, and the package ID nictiz.fhir.nl.stu3.zib2017#1.3.9 has no problems resolving using the cache manager, so I guess it's a bug in the cache manager.

The package hl7.fhir.au.pd#2.0.1 has a SearchParameter, which causes the following error:

Caused by: ca.uhn.fhir.rest.server.exceptions.UnprocessableEntityException: Invalid SearchParameter.expression value "extension(‘http://hl7.org.au/fhir/StructureDefinition/au-receivingfacility’).extension(‘namespace-id’).value": Error in ?? at 1, 1: Found ‘ expecting a token name
    at ca.uhn.fhir.jpa.dao.r4.FhirResourceDaoSearchParameterR4.validateSearchParam(FhirResourceDaoSearchParameterR4.java:147)
    at ca.uhn.fhir.jpa.dao.r4.FhirResourceDaoSearchParameterR4.validateResourceForStorage(FhirResourceDaoSearchParameterR4.java:88)
    at ca.uhn.fhir.jpa.dao.r4.FhirResourceDaoSearchParameterR4.validateResourceForStorage(FhirResourceDaoSearchParameterR4.java:47)
    at ca.uhn.fhir.jpa.dao.BaseHapiFhirDao.updateEntity(BaseHapiFhirDao.java:997)
    at ca.uhn.fhir.jpa.dao.BaseHapiFhirResourceDao.doCreate(BaseHapiFhirResourceDao.java:515)
    at ca.uhn.fhir.jpa.dao.BaseHapiFhirResourceDao.create(BaseHapiFhirResourceDao.java:221)
    at ca.uhn.fhir.jpa.dao.BaseHapiFhirResourceDao.create(BaseHapiFhirResourceDao.java:185)

To me this looks like 1) an error in the package or 2) some kind of character encoding error in the json parser, used to parse the contents of the package, or in the dao, but I'm not sure, haven't looked into it.

These are the errors I encountered in todays testing of the IG installer @Ward Weistra :)

Ward Weistra (May 13 2020 at 15:46):

Thanks @Martin Zacho Grønhøj. Indeed, the Simplifier packages issue of not filling the fhirVersion has been resolved today for new newly created packages. And apparently Grahame has also updated his tooling to handle them for the few that don't :clap:

The .NET tool stack supports the limited wildcard semantic versioning that the package spec allows and resolves this dependency correctly: https://simplifier.net/packages/nictiz.fhir.nl.stu3.questionnaires/1.0.0

Perhaps @Brett Esler can help us debug the issues with the two hl7.fhir.au packages, which I assume come from his feed.

Brett Esler (May 14 2020 at 22:59):

Hi @Ward Weistra what is up with hl7.fhir.au packages?

Ward Weistra (May 15 2020 at 12:28):

@Brett Esler: Martin Zacho Grønhøj saw the following two errors when importing them into HAPI:

The package hl7.fhir.au.rcpa#0.1.0 throws a parsing error:

ca.uhn.fhir.parser.DataFormatException: Unknown element 'comments' found during parse

Don't know if its an error in the parser or the package data (probably the former).

Simplifier also has trouble parsing the package (no files shown) and looking at what's in the package folder there is no dot between the filename base and extension: eg Bundle-Colorectal-genexample-1-mapjson

The package hl7.fhir.au.pd#2.0.1 has a SearchParameter, which causes the following error:

Caused by: ca.uhn.fhir.rest.server.exceptions.UnprocessableEntityException: Invalid SearchParameter.expression value "extension(‘http://hl7.org.au/fhir/StructureDefinition/au-receivingfacility’).extension(‘namespace-id’).value": Error in ?? at 1, 1: Found ‘ expecting a token name
    at ca.uhn.fhir.jpa.dao.r4.FhirResourceDaoSearchParameterR4.validateSearchParam(FhirResourceDaoSearchParameterR4.java:147)
    at ca.uhn.fhir.jpa.dao.r4.FhirResourceDaoSearchParameterR4.validateResourceForStorage(FhirResourceDaoSearchParameterR4.java:88)
    at ca.uhn.fhir.jpa.dao.r4.FhirResourceDaoSearchParameterR4.validateResourceForStorage(FhirResourceDaoSearchParameterR4.java:47)
    at ca.uhn.fhir.jpa.dao.BaseHapiFhirDao.updateEntity(BaseHapiFhirDao.java:997)
    at ca.uhn.fhir.jpa.dao.BaseHapiFhirResourceDao.doCreate(BaseHapiFhirResourceDao.java:515)
    at ca.uhn.fhir.jpa.dao.BaseHapiFhirResourceDao.create(BaseHapiFhirResourceDao.java:221)
    at ca.uhn.fhir.jpa.dao.BaseHapiFhirResourceDao.create(BaseHapiFhirResourceDao.java:185)

To me this looks like 1) an error in the package or 2) some kind of character encoding error in the json parser, used to parse the contents of the package, or in the dao, but I'm not sure, haven't looked into it.

It seems to complain about the curly quotes (‘ and ’) being used in the expression of the three SearchParameters in there.

Keep in mind that published packages can't be changed, so if you want to bring out fixed versions of these you will have to publish new versions (higher version number) for them to be picked up from the feed. And then the older versions of the package can be unlisted (not sure if there's a way to indicate that in the feed yet).

Jens Villadsen (Aug 17 2020 at 19:08):

PR is now available on HAPI 5.1.0: https://github.com/hapifhir/hapi-fhir-jpaserver-starter/pull/126

Last updated: Apr 12 2022 at 19:14 UTC