FHIR Chat · FSH Packages · shorthand

Stream: shorthand

Topic: FSH Packages

Richard Kavanagh (Feb 20 2020 at 22:01):

Where does SUSHI resolve its packages from?
I tried to add a package that was published on SImplifier at https://simplifier.net/packages and it could not locate it.

Chris Moesel (Feb 20 2020 at 22:38):

It loads dev versions from your local cache, current versions from http://build.fhir.org, and all other versions from http://packages.fhir.org. We could probably update it to fall back to simplifier.net if it can't find it in packages.fhir.org (but we haven't implemented that currently).

Grahame Grieve (Feb 20 2020 at 22:45):

packages.fhir.org is just an alias for simplifier. You'll have to ask firely why a package isn't available

Chris Moesel (Feb 21 2020 at 01:37):

Ah, OK. I was thinking it was a separate instance of simplifier. I didn't realize (or forgot) it was an alias.

Chris Moesel (Feb 21 2020 at 01:38):

@Richard Kavanagh -- if you can tell us how you're referencing the package in package.json, we can look at it from our end.

Richard Kavanagh (Feb 21 2020 at 08:45):

Thanks @Chris Moesel I was attempting to add this package https://simplifier.net/packages/uk.core.r4/1.0.0

I amended package.json to

 "dependencies": {
    "hl7.fhir.r4.core": "4.0.1",
    "hl7.fhir.us.core": "3.1.0",
    "UK.Core.r4": "1.0.0"
 },

When I executed SUSHI the log showed

Found UK.Core.r4#1.0.0 in local cache.
error Failed to add UK.Core.r4:1.0.0 to ImplementationGuide instance. Could not determine its canonical URL from the FHIR cache.

Nick Freiter (Feb 21 2020 at 12:58):

SUSHI expects that the ImplementationGuide resource for this package in hl7fhirukcoreimplementationguider4-january2020.json does not declare a packageId. SUSHI expects that the package UK.Core.r4#1.0.0 will have packageId set to UK.Core.r4, and version set to 1.0.0 in the ImplementationGuide resource. A workaround would be to set these values locally:
1) Navigate to your local FHIR cache located at ~/.fhir/packages
2) Find the UK.Core.r4#1.0.0 folder, and navigate to ~/.fhir/packages/UK.Core.r4#1.0.0/package
3) Open the hl7fhirukcoreimplementationguider4-january2020.json file in this folder
4) Change the version field from January 2020 to 1.0.0
5) Add a packageId field that is set to UK.Core.r4

This should allow SUSHI to correctly find the UK.Core.r4 package. It seems this package should come with a packageId already set, since that appears to be a required field on the ImplementationGuide resource. We have been assuming that the version field would match the numerical version given when specifying the dependency, but maybe that is an incorrect assumption on our end.

Richard Kavanagh (Feb 21 2020 at 14:02):

Thanks @Nick Freiter that seems to have helped a bit. I'm not sure why the package has these items missing.

I don't get the error message anymore, now I get a new one.

error Cannot read property 'children' of undefined

My FSH file is

Instance:   RichardSMITH
InstanceOf: https://fhir.nhs.uk/R4/StructureDefinition/UKCore-Patient
Title: "Richard SMITH"
* gender = #female
* name.given = "Richard"
* name.family = "SMITH"
* name.use = "official"

I don't get any output from this file at all.

Mark Kramer (Feb 21 2020 at 15:01):

The RichardSMITH output should be located in build/input/examples/Patient-RichardSMITH.json

Richard Kavanagh (Feb 21 2020 at 15:10):

Thanks @Mark Kramer the folder is empty. I have generated examples from other FSH files but I have removed them all to isolate the problem. I now only have the one FSH file and nothing gets generated. I guess it has to do with the error message I posted previously.

As I am using the UK Core package I can only imagine that there is an issue with the package.

Richard Kavanagh (Feb 21 2020 at 15:15):

The following FSH generated OK so it must be the package that is the issue.

Instance:   JohnSMITH
InstanceOf: Patient
Title: "John SMITH"
* gender = #male
* name.given = "John"
* name.family = "SMITH"
* name.use = #home

Richard Kavanagh (Feb 21 2020 at 15:19):

The package was created using Simplifier not the HL7 IG Publisher - perhaps there are inconsistencies at the moment. @Ward Weistra are you aware of any?

Nick Freiter (Feb 21 2020 at 15:23):

It looks like the https://fhir.nhs.uk/R4/StructureDefinition/UKCore-Patient Structure Definition that comes from that package only has a differential, no snapshot. As of now, SUSHI is dependent on a snapshot being present in the Structure Definition. This is something we can consider addressing, but it would be complex to develop, since we would need to either modify a lot of our handling, or develop the ability to generate snapshots from differentials.

So far the assumption that IGs would have snapshots hasn't come back to bite us. Is having an IG without snapshots more common than we thought?

Chris Moesel (Feb 21 2020 at 15:26):

@Grahame Grieve , @Ward Weistra -- is it safe to assume, in general, that IGs we retrieve from packages.fhir.org will have snapshot populated in their SDs?

Richard Kavanagh (Feb 21 2020 at 15:42):

@Nick Freiter I have manually forced the generation of the Snapshot for the profile that was having the issues and now it works.
I'll feedback to the authors and hopefully, they will fix this in the next release.
Thanks for your help.

Ward Weistra (Feb 21 2020 at 17:09):

Thanks for troubleshooting this, gentlemen!

@Nick Freiter Why is SUSHI relying on the ImplementationGuide resource in a package? I don't think every package will have one.
I'm surprised to indeed see packageId to be mandatory on the IG resource though. So I'll see how we can fill that at some point.

@Chris Moesel I'm afraid that that's not the case. Since calculating snapshots is compute intensive (and the logic changes now and then) we don't include them by default in new packages. It also feels a bit like duplicating content which is contained in the dependencies, although I see the convenience and have had the request before...

Sidenote: the package registry of course delivers packages, not IGs ;)

The spec states 'All FHIR Implementation Guides are published as NPM packages, one package for each IG.' but not every package needs to have an IG included. The IG publisher stack creates this one to one link, but it can't be expected from FHIR packages in general.

Chris Moesel (Feb 21 2020 at 17:27):

Hi @Ward Weistra -- we currently rely on an ImplementationGuide resource because when we build the ImplementationGuide for the SUSHI IG, we need to populate dependsOn. Unfortunately, dependsOn requires (1..1) the uri property, which has type canonical(ImplementationGuide).
image.png

Chris Moesel (Feb 21 2020 at 17:28):

That's pretty hard to populate if there is no ImplementationGuide to point to.

Chris Moesel (Feb 21 2020 at 17:30):

Seems like maybe a hiccup in the system if the intent is to allow dependencies on non-IG packages.

Chris Moesel (Feb 21 2020 at 17:32):

Since calculating snapshots is compute intensive (and the logic changes now and then)

Ha. This is exactly why we want to rely on the snapshot being there instead of having to compute it ourselves. ;-) I guess if we need to support packages that don't include snapshots, we'll either need to revisit our approach to building SDs or we'll have to implement a snapshot generator (sigh).

Ward Weistra (Feb 21 2020 at 17:39):

Thank @Chris Moesel. Yes, that's interesting. You want to help your user to populate the depends on of their IG. But only in the HL7 balloting world there's the convention that the packages an IG depends on will have an IG in the package.

I guess the aim for FISH is to be there for a larger audience than just that world. There's no guarantee the user even wants to make an IG themselves I'd say. So possibly that logic to help create the IG resource could be optional.
Or for dependency packages without an IG you could not include that package in depends on of the IG. It will still be in the package.json. Or allow the user to specify a list of depends ons if there are IG linked to those packages but they are not IN the packages (since that's not required).

Keith Boone (Feb 21 2020 at 18:50):

I believe that someone has already created a Snapshot Generator, although it might not be in your preferred programming language. One possible solution would be to stand it up as a service somewhere though.

Chris Moesel (Feb 21 2020 at 18:53):

Yeah, I think we'll need to think on it some more @Ward Weistra. SUSHI doesn't require you to produce an IG -- we only create the ImplementationGuide JSON if the FSH Tank has an ig-data folder in it. Tanks without that folder will get the resource files only (without any IG stuff). So this is only relevant for folks who want to create an IG. But if they do, I think we still have an issue if they rely on a package that doesn't have an ImplementationGuide -- as the template-based publishing doesn't use the package.json; it only uses the ig.ini and ImplementationGuide JSON.

Grahame Grieve (Feb 21 2020 at 18:55):

@Ward Weistra is correct that packages do not require an Implementation Guide resource. OTOH, all implementation guide packages will have an one. The issue here is that there's no facility for a FHIR implementation guide to depend on profiles that are not published via an implementation guide. That's not a tooling limitation, that's baked into the definition of the implementation guide resource

Grahame Grieve (Feb 21 2020 at 18:56):

but if I track this discussion correctly, the intent is to depend on a package that is an implementation guide, but doesn't have an implementation guide resource, and there fore the problem is in the source package

Grahame Grieve (Feb 21 2020 at 18:57):

I think that we should have a rule that all implementation guide packages that represent a published implementation guide should include snapshots

Chris Moesel (Feb 21 2020 at 18:58):

I think this conversation actually started w/ a dependency on a package that had an implementation guide, but that implementation guide was not conformant (missing the packageId property). It also had resources without snapshots. The conversation then morphed into the more general discussion about our reliance on dependency IGs.

Chris Moesel (Feb 21 2020 at 19:03):

@Keith Boone -- I'm aware of a snapshot generator in Java (part of the Publisher tooling I think) and one in .NET (part of Forge maybe?). I don't know how separable they are, but standing one up as a service seems feasible. I was going to say that the downside is that SUSHI could not run offline -- but we already require network if you need to download dependencies, so in theory we could do the snapshot generation immediately after that (and cache it somewhere). Still, IMO, the fewer external network dependencies we have, the better... Gonna have to think on this one.

Chris Moesel (Feb 21 2020 at 19:09):

That said, even if I wanted to, I'm not sure I can justify building it myself just because I didn't want a network call. That said, @Grahame Grieve -- is there a published algorithm or approach for generating snapshots? Are there only a few rules to keep in mind or are there lots of complexities and edge cases? It's one of those things that sounds simple on the surface ("just a special merge") but I bet has lots of gremlins lurking beneath.

Grahame Grieve (Feb 21 2020 at 19:29):

loooots of gremlins hiding underneath.

Grahame Grieve (Feb 21 2020 at 19:29):

there's a set of test cases, and I shudder each time I have to revisit them

Grahame Grieve (Feb 21 2020 at 19:30):

agree it makes sense to generate the depedencies when installing the package. You don't need a network service for it; the validator can generate snapshots on reqeust

Chris Moesel (Feb 21 2020 at 19:32):

We're TypeScript/JavaScript -- so we'd need to make a system call to call the validator -- and that would require the user to have Java installed. Still, if they want to use the IG Publisher, I guess they need Java already anyway.

Grahame Grieve (Feb 21 2020 at 19:34):

one easy way is to die with an error if the package doesn't have snapshots and get the user to do it manually to the installed package, and I add a command line option to do that

Keith Boone (Feb 22 2020 at 16:15):

You might be able to use Google's JCL (https://www.infoq.com/news/2019/05/j2cl-java-javascript-transpiler/) to transcode the Java one to JavaScript and run it inside Sushi.

Grahame Grieve (Feb 22 2020 at 20:33):

carumba. Given the scope of the dependencies for the validator (json xml rdf apache libraries) I wouldn't hold my breath

Jens Villadsen (Feb 22 2020 at 20:39):

Chris Moesel said:

We're TypeScript/JavaScript -- so we'd need to make a system call to call the validator -- and that would require the user to have Java installed. Still, if they want to use the IG Publisher, I guess they need Java already anyway.

roll it in docker ... then you're off the hook

Jens Villadsen (Feb 22 2020 at 20:42):

@Chris Moesel we are some people that roll the entire setup in docker ... leaving the Java Runtime as well as Jekyll and Ruby inside the whale where it belongs

Jens Villadsen (Feb 22 2020 at 20:43):

Keith Boone said:

You might be able to use Google's JCL (https://www.infoq.com/news/2019/05/j2cl-java-javascript-transpiler/) to transcode the Java one to JavaScript and run it inside Sushi.

The sun would have burned out long before you get that to succeed ... that would be my estimate

Chris Moesel (Feb 24 2020 at 13:36):

@Grahame Grieve -- Do we want to encourage users to modify their installed packages like that? I assumed that, for the most part, we wanted packages to be managed more tightly than that? And when you said "and I add a command line option to do that" -- do you mean you would add a command-line option to the validator to populate snapshots in an installed package?

@Keith Boone -- I actually hadn't seen J2CL before. I'm intrigued! Even if it doesn't work out here, it might be helpful for one of my JS-based CQL projects than needs to use the Java-based CQL-to-ELM translator. Thanks for the pointer!

@Jens Villadsen -- Yes, Docker would certainly alleviate some of these concerns -- but some of the audience we're targeting probably isn't really the Docker type. Still, we certainly could consider Docker as an option for those who do use it.

Jens Villadsen (Feb 24 2020 at 13:45):

@Chris Moesel - well ... my impression is that FSH pretty much targets both kinds of audience ;)

Chris Moesel (Feb 24 2020 at 13:45):

@Jens Villadsen -- agreed.

Keith Boone (Feb 24 2020 at 14:58):

@Chris Moesel , I too am intrigued. Let me know how well it works if you get to play with it before I do, and I'll do the same.

Grahame Grieve (Feb 24 2020 at 20:02):

Do we want to encourage users to modify their installed packages like that?

Not in principle, no. But the question here appears to be least worst approach overall. So it's an option

Grahame Grieve (Feb 24 2020 at 20:02):

you would add a command-line option to the validator to populate snapshots in an installed package?

yes

Ward Weistra (Feb 25 2020 at 10:59):

I guess there's no need to 'edit the packages' themselves, but rather creating a set of temporary snapshots in the user's cache? Maybe just a phrasing difference, but lets keep packages immutable.

Grahame Grieve (Feb 25 2020 at 11:17):

yes they are immutable

Last updated: Apr 12 2022 at 19:14 UTC