Stream: bulk data
Topic: Tool to push IG NPM package into FHIR servers
Richard Stanley (Apr 21 2021 at 14:01):
(Redirect from Implementers channel) Hi folks. I'm looking for an existing (preferably open source) tool that can push FHIR IG NPM packages (.tgz) built by Publisher into a FHIR server. With IGs being fundamental to FHIR, and the NPM IG packaging being the supported approach, I would find it odd if such a tool does not already exist. SmileCDR has an API to ingest NPM packages, but I'm looking to push and prefer non-proprietary.
Josh Mandel (Apr 21 2021 at 14:35):
Can you say more about what it means to push an npm package into a server, and how this relates to block data?
Richard Stanley (Apr 21 2021 at 14:45):
Graham suggested asking here. By pushing, I mean taking apart an NPM IG package into the constituent resources, and POSTing to FHIR server. The reason I ask is because my use case requires loading FHIR servers with IGs and would expect there is existing tooling to do so. As NPM packaging is the supported format, then what workflow are folks using load them into FHIR servers.
Paul Church (Apr 21 2021 at 15:15):
I assume Grahame was thinking of bulk import, otherwise I don't see the relevance. I have an IG that consists of a bunch of resources, I want to load the resources into my server to configure it for use, how do I do that?
We use the google proprietary import operation for this general use case, but it doesn't support NPMs specifically. Might be an area for future improvement. Do other servers have support for loading resources from an NPM package?
Richard Stanley (Apr 21 2021 at 15:19):
Ahh ok. I wondered why he sent me here :) Yeah, no direct relevance.
SmileCDR supports it but there's isn't an API exposed to do so in HAPI. If this tooling doesn't exist, I can do it manually in Go or Python, but I expected perhaps incorrectly that someone had made an open tool to do so.
Martin Höcker (Apr 21 2021 at 17:58):
Did you have a look at firely terminal? I only tried it briefly, but I think it's compatible with the use-case you are describing. Most of the features are free, but the tool is not open source.
Richard Stanley (Apr 21 2021 at 18:03):
I don't have it, but thanks for the tip. Will have a look!
Oliver Egger (Apr 21 2021 at 19:09):
there is in addition python upload code and you can configure hapi, see https://chat.fhir.org/#narrow/stream/179167-hapi/topic/How.20to.20install.20the.20FHIR.20NPM.20Package.3F
Grahame Grieve (Apr 21 2021 at 20:27):
I think this is bulk import, just a variant format
Richard Stanley (May 08 2021 at 17:13):
HI folks. Just a quick update, I'm writing an open source tool to push FHIR NPM IG packages to FHIR servers. What I'm seeing in loading (PUTting) FHIR resources is that there is an order of precendence for them to be accepted by HAPI (and maybe other servers), e.g. CodeSystem before ValueSet, etc. Is there a machine parseable list somewhere which lists dependencies between resources types?
Josh Mandel (May 08 2021 at 17:39):
Is the order you're referring to just "a resource needs to be loaded before any resources that depend on it"? In general you can't do this calculation based on resource types alone; it is the actual references inside of instances that matter. For example, any resource can include an extension with a valueReference pointing to any other resource.
Josh Mandel (May 08 2021 at 17:43):
So you'd need to do some kind of topological ordering on your actual data set rather than on an abstract list of resource types. Alternatively, you may be able to disable integrity checks while loading a large data set into the servers. Alternatively, you could partition data into self-contained subsets that you represent with transaction bundles, to handle cross-resource references within each bundle.
In general, all these considerations are something that a server would take care of with a well crafted $import operation; there is work on going to define the details for a standardized import operation, but right now your best bet would be to look at server specific capabilities, since we don't yet have standards for this beyond the draft stage.
Richard Stanley (May 08 2021 at 18:56):
Thanks @Josh Mandel for a wonderfully detailed answer. Yes, that's my use case and I understand better now the constraints.
Lloyd McKenzie (May 08 2021 at 19:06):
I would expect a server would consume an NPM package the same way it consumes a transaction - and there's no ordering required for a transaction. Servers just need to manage the dependency links they find in whatever manner works best for them.
Richard Stanley (May 08 2021 at 19:22):
Thanks @Lloyd McKenzie Ok, so that implies a way to implement, put all the resources into a transaction and then let the server manage them. I wonder if there's a future feature here for FHIR NPM packages, e.g. have publisher create a transaction bundle of the resources? For now my little CLI app just grabs the package.tgz into memory and uses the .index.json to figure out what needs to be pushed to the FHIR server. (Related -- I wonder if .index.json could be included directly in /output rather than only in the tgz). Again, thanks for the advice.
Grahame Grieve (May 09 2021 at 21:15):
I'm not sure what it would mean to include .index.json outside the package, since all the names in it are the names inside the package
Christian Gulden (May 10 2021 at 12:49):
@Richard Stanley have you seen HAPI's support for loading IGs? You can specify a registry or load the IG from a .tgz: https://github.com/hapifhir/hapi-fhir-jpaserver-starter/blob/master/src/main/resources/application.yaml#L43-L52. Of course this is HAPI-specific, but the implementation may be interesting to you. E.g. there seems to be an implicit ordering by resource types used when loading them into the server: https://github.com/hapifhir/hapi-fhir/blob/75c74a22dbd1f0dde3631b540d1898eef2a2666f/hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/packages/PackageInstallerSvcImpl.java#L85-L93 (briefly skimming over the source code) - which might work well here due to the way these specific resources reference each other.
Richard Stanley (May 10 2021 at 13:06):
Thanks @Christian Gulden I'm happy for the link to the code!
Last updated: Apr 12 2022 at 19:14 UTC
