FHIR Chat · NPM Package Spec · fhir/infrastructure-wg

Stream: fhir/infrastructure-wg

Topic: NPM Package Spec

Grahame Grieve (Oct 02 2019 at 20:12):

Topic for next call: https://wiki.hl7.org/FHIR_NPM_Package_Spec describes a tooling specification that the toolsmiths need to be stable. At Atlanta firely and I discussed this and said we'd make a round of edits on the page, and then ask FHIR-I to formally take control of the spec, so that any further changes must be approved through committee.

That time has now come.

Grahame Grieve (Oct 02 2019 at 20:28):

Motion: that we leave this spec on the wiki, but add a note that any further non-trivial changes need to be approved by FHIR-I

Alternative motion is to move it into the main spec, but it's very oblique to the spec. Orsome spec somewhere else. I don't feel strongly about this

Grahame Grieve (Oct 02 2019 at 20:29):

I won't be on the call (travelling)

Grahame Grieve (Oct 18 2019 at 03:25):

I think it's time that this page moved to formal FHIR-I mgmt, and to some sort of 'tooling standard' point. (not sure what it's formal HL7 standard should be). But anyway, no changes without FHIR-I agreement now on:

https://wiki.hl7.org/FHIR_NPM_Package_Spec

(see https://chat.fhir.org/#narrow/stream/179166-implementers/topic/FHIR.20NPM.20packages)

Also https://wiki.hl7.org/FHIR_Package_Cache and https://wiki.hl7.org/FHIR_IG_PackageList_doco too (@Martijn Harthoorn )

Martijn Harthoorn (Oct 18 2019 at 09:46):

Sounds right.

Martijn Harthoorn (Oct 24 2019 at 13:46):

Under package content there is a line "All resources have a filename of [Type]-[id].json". I remember we choose to write that as a SHOULD, because it's good as a best practice, but we should refrain from making it mandatory - since it would lead to tooling having it as expectation - and that will lead to future inflexibility.

Martijn Harthoorn (Oct 24 2019 at 13:47):

@Grahame Grieve

Grahame Grieve (Oct 24 2019 at 17:24):

no it must be a SHALL

Grahame Grieve (Oct 24 2019 at 17:25):

there's no reason for flexibility on this in a package, and a reason for not: it makes it much much quicker to load a package - something you are are usually very concerned with

Martijn Harthoorn (Oct 25 2019 at 08:19):

My thoughts on this:
- In our tools we use .index.json to load a package
https://chat.fhir.org/#narrow/stream/179177-conformance/topic/Package.20rework/near/168938409
- Id is scoped and can be re-assigned per server. A package trancends that scope.
- Id has a cardinality of 0..1 and It is very common for draft conformance resources to not (yet) have an Id.

Michel Rutten (Oct 25 2019 at 08:47):

Requiring both a mandatory index file and mandatory file name patterns seems redundant? I'd prefer to require an index file and not restrict file names.

Martijn Harthoorn (Oct 25 2019 at 09:26):

Does Forge set an id on a StructureDefinition?

Michel Rutten (Oct 25 2019 at 10:03):

Forge allows authors to specify a Resource Id. However when uploading to a FHIR server, the server may reject the specified ID and assign another one. After publishing a profile to a FHIR server/Simplifier, Forge updates the Resource.id and publishing date values. If the server assigned a (new/other) id, then Forge will mark the Resource as "dirty" (i.e. having unsaved changes), to ensure that users save back the updated resource with the newly assigned id to local disk.

Lloyd McKenzie (Oct 25 2019 at 13:11):

Before we lock it, it should be moved to Confluence.

Grahame Grieve (Oct 25 2019 at 20:16):

using .index.json is not a substitute for an organised naming strategy. The key thing - absolutely critical - is that you don't have to read every resource in the package to find out what kind of resource it is. You just scan the directory for the resource types you are looking for. What is proposed for .index.json doesn't substitute for that.

Grahame Grieve (Oct 25 2019 at 20:16):

This is mandatory for me

Grahame Grieve (Oct 25 2019 at 20:17):

I don't understand either of these points:

- Id is scoped and can be re-assigned per server. A package trancends that scope.

uh what? I don't see how that matters. the id is fixed in the package itself

- Id has a cardinality of 0..1 and It is very common for draft conformance resources to not (yet) have an Id.

then they can't be part of a package since it means they are not stored anywhere

Grahame Grieve (Oct 25 2019 at 20:18):

I don't understand why it's an issue either. Humans don't read the packages

Grahame Grieve (Oct 25 2019 at 20:21):

I see that I overlooked adding anything about .index.json to the package spec btw

Michel Rutten (Oct 26 2019 at 15:40):

I understand the performance requirements.

Indexing based on mandatory file name patterns is indeed the fastest and easiest solution, at the cost of being inflexible and brittle. Package authors lose the original resource file names, depending on their internal workflow. And the proposed file name pattern provide only resource type & id, whereas clients need to index other metadata as well.

The metadata harvesting logic in the open source .NET API is much more flexible and sufficiently fast for us not to have to rely on file name patterns. The Forge project explorer depends on it. As this demonstrates that the perceived performance problems can be addressed, I'd much rather see a recommendation for a file name pattern instead of a hard requirement.

Grahame Grieve (Oct 27 2019 at 00:18):

The metadata harvesting logic in the open source .NET API is much more flexible and sufficiently fast for us not to have to rely on file name patterns

this is in .index.json? All that is in there at the moment is canonical URLs. If we can add to that, then that would be basis for going forwards

Grahame Grieve (Oct 27 2019 at 00:25):

for instance, if we had this:

Grahame Grieve (Oct 27 2019 at 00:26):

{
  "index-version" : 1,
  "canonicals": {
    "[url]": "[filename]",
    etc..
  },
  "types" : [
    "[type}" : ["filename", "filename"],
    etc..
  ],
  "date" : "[date]"
}

Grahame Grieve (Oct 27 2019 at 00:27):

index-version = 1 fixed value. if you see anything different, rebuild the .index.json
canonicals = a list of canonical URLs in the package
types = a list of resources by type
date = date the file was added to the cache it is in (purely for display purposes)

Grahame Grieve (Oct 27 2019 at 00:27):

if we had that, I wouldn't care what the files are called

Grahame Grieve (Oct 27 2019 at 00:33):

also, the canonicals should have a version using the |[ver] format

Grahame Grieve (Oct 27 2019 at 03:46):

so example of what I'm talking about - I populated all the packages here https://github.com/FHIR/packages

Grahame Grieve (Oct 27 2019 at 03:59):

they all have a .index.json now

Grahame Grieve (Oct 27 2019 at 06:33):

emerging issue: should examples have it's own .index.json?

Michel Rutten (Oct 27 2019 at 17:00):

Sounds good.

Maybe it's more convenient to restructure the index as a list of complex file info nodes? Each node would represent a single file and contain a set of metadata properties (filename, canonical url, resource type, ...). This would also allow for additional polymorphic metadata, e.g. depending on the resource type. Clients can easily generate lookup tables from the index file, e.g. profiles by url, examples by profile etc. The format is also easily extensible by adding new metadata properties.

Example:

{
  "index-version" : 1,
  "date" : "[date]",
  "files": [
    {
      "filename": "nl-core-patient.json",
      "id": "nl-core-patient",
      "resourcetype": "StructureDefinition"
      "url": "http://fhir.nl/fhir/StructureDefinition/nl-core-patient",
      "kind": "Resource",
      "type": "Patient",
      etc..
    },
    {
      "filename": "nl-core-address-official.json",
      "id": "nl-core-address-official",
      "resourcetype": "StructureDefinition",
      "url": "http://fhir.nl/fhir/StructureDefinition/nl-core-address-official",
      "kind":"complex-type",
      "type": "Extension",
      etc..
    },
    {
      "filename": "nl-core-patient-example-1.json",
      "id": "nl-core-patient-example-1",
      "resourcetype": "Patient",
      "profile": "http://fhir.nl/fhir/StructureDefinition/nl-core-patient",
      etc...
    },
    etc..
  ],
}

@Martijn Harthoorn ?

Grahame Grieve (Oct 27 2019 at 21:33):

I don't mind this, though I worry about complexity and size

Grahame Grieve (Oct 27 2019 at 22:12):

I regenerated the base packages - about twice the size. committed them, so you can look at them

Martijn Harthoorn (Oct 28 2019 at 16:10):

Grahame,
Assuming you can live without a mandatory type-id.json format
Technically I think the package spec and the package cache spec are good to go.
There are some textual improvements (without technical implications) still possible (like removing some duplications).

Grahame Grieve (Oct 28 2019 at 20:50):

Yes I no longer care about the name of the file

Grahame Grieve (Oct 29 2019 at 03:13):

ok I update the wiki page, consistent with what I think the decision was.

Lloyd McKenzie (Oct 29 2019 at 03:19):

We'll pretend that happened before the FHIR-I WG locked the page. Any further changes require a vote.

Grahame Grieve (Oct 29 2019 at 03:22):

ok we can pretend that

Grahame Grieve (Oct 31 2019 at 04:58):

so...... immediately after agreeing that the package specification was under FHIR-I control, I've had to make emergency changes to the spec in order to get the technical corrections published.

Grahame Grieve (Oct 31 2019 at 04:59):

- I added a 'fhir-version-list' property to the npm package.json file, an array of string

This is added so that a tool an always tell what fhir-versions the package is for. This was necessary because in the past, the package would simply refer to the core dependency, even if the package it was referring to didn't- and would never - exist. Now, they can't refer to packages that don't exist. So package references can no longer tell you what FHIR version except for the major releases. And many of our balloted IGs don't depend on a major release. This won't be possible in the future anyway, so there'll be no more new IGs like this

- I added "sub-packages" - array of string - to package-list.json

This was because we now publish the main spec as a set of packages, and I had to know about them somewhere - it's only a message to the publishing tool that builds the package RSS feed

Grahame Grieve (Oct 31 2019 at 05:00):

FHIR-I please approve these on Monday's call. (and sorry, but I'm cleaning up an old mess and adding functionality we already agreed to, but I hadn't seen the ramifications of)

Grahame Grieve (Oct 31 2019 at 05:01):

I already updated the documentation

Martijn Harthoorn (Oct 31 2019 at 10:14):

It's not problematic to add custom properties to the package.json, as long as they can safely be ignored by other tools and don't overlap with existing properties.

Martijn Harthoorn (Oct 31 2019 at 10:15):

And with the fhir-version-list i agree.

Martijn Harthoorn (Oct 31 2019 at 10:16):

Would it be more inline with other properties to just call the property fhirVersions?

Grahame Grieve (Oct 31 2019 at 10:18):

there was no camel case elements, so I wasn't sure. It's kind of hard to change it now

Martijn Harthoorn (Oct 31 2019 at 10:20):

devDependencies

Grahame Grieve (Oct 31 2019 at 10:20):

?

Martijn Harthoorn (Oct 31 2019 at 10:21):

is an example of a camel case property in package.json

Grahame Grieve (Oct 31 2019 at 10:21):

I'm not familiar with it?

Martijn Harthoorn (Oct 31 2019 at 10:22):

Neither am I. We don't use it for the fhir package spec. But it shows the pattern.

Grahame Grieve (Oct 31 2019 at 10:23):

oh, I only looked at what we used. If I'm going to change it, I have to change it now...

Martijn Harthoorn (Oct 31 2019 at 10:23):

It will ease your concsiousness ;)

Grahame Grieve (Oct 31 2019 at 10:23):

hah. it will cost me time...

Grahame Grieve (Oct 31 2019 at 10:45):

but done anyway

Martijn Harthoorn (Oct 31 2019 at 11:30):

I will add it to our (new) packages too.

Martijn Harthoorn (Oct 31 2019 at 14:24):

What will the format of the fhir version be? "3.0.1" or "r3" or just 3?

Grahame Grieve (Oct 31 2019 at 17:27):

whihc bit are you asking aout?

Martijn Harthoorn (Nov 01 2019 at 11:39):

The bit in package.json - fhirVersions property.

Grahame Grieve (Nov 01 2019 at 12:39):

3.0.1 - full 3 parts

Grahame Grieve (Nov 01 2019 at 12:40):

for me, r2 | r2b | r3 | r4 | r5 are codes for a 3 letter version e.g. 1.0, 1.4, 3.0, 4.0, and 4.1. These are independent of patch. But version references in packages are always version referenced. (at the moment)

Grahame Grieve (Nov 24 2019 at 16:00):

at DevDays, @Hans Buurman (I think) claimed to me that the NPM Spec is invalid because it allows siblings directories to \packages, and we have made extensive use of them

Grahame Grieve (Nov 24 2019 at 16:09):

And indeed, from http://wiki.commonjs.org/wiki/Packages/1.0#Package_File_Format:

The archive must have a single top level directory.

Grahame Grieve (Nov 24 2019 at 16:19):

So what I propose:

• update the FHIR NPM Package spec to also say this, and say that sub-folders of the package spec are not described (instead of siblings)
• regenerate all the packages on hl7.org/fhir.org to use subfolders instead of sibling folders
• reload on packages.fhir.org

@Martijn Harthoorn - thoughts?

Grahame Grieve (Nov 24 2019 at 19:37):

we should also clarify that the name of the filename in .index.json is local to the same directory as the .index.json file, and that there is one .index.json file per directory (and that it is not mandatory for directories)

Grahame Grieve (Nov 24 2019 at 19:37):

(inside the package)

Grahame Grieve (Nov 24 2019 at 20:06):

also, reviewing the packages (again) - some old SDC versions didn't use semver. I think I should retrospectively update them - both package, resources, and html files - to use semver. @Lloyd McKenzie

1.6 -> 1.6.0
2 -> 2.0
1.6 -> 1.6.0
2 - 2.0.0

Lloyd McKenzie (Nov 24 2019 at 20:12):

Ok

Lloyd McKenzie (Nov 24 2019 at 20:13):

Does the change to npm impact the way folders need to be organized in IG templates?

Grahame Grieve (Nov 24 2019 at 21:14):

no

Grahame Grieve (Nov 24 2019 at 21:14):

it's all generated

Ward Weistra (Nov 25 2019 at 10:09):

This was @Hans van Amstel's comment :up: Would be great to add his voice to this conversation, as well that of the guys from Black Pear who had an opinion on this (@Dunmail?)

Dunmail (Nov 25 2019 at 10:27):

It was me! :grinning:

Dunmail (Nov 25 2019 at 10:49):

As JS developer, making FHIR profiles available in packages that can be natively used in JS will be a massive leap forward because we have definitive source artefacts to use as the basis for tooling etc.

We understand the decision to provide the packages via a different repository. This is OK.

We have confirmed that we can use npm to pull a root package from the FHIR repo. However, the dependencies don't come down :thinking:

I wonder if we could do something clever with a post-install script - any intention to use these for FHIR packages?

Grahame Grieve (Nov 25 2019 at 12:38):

It was that Hans yes

Grahame Grieve (Nov 25 2019 at 12:39):

We’ve said that we will prohibit scripts - security nightmare

Dunmail (Nov 25 2019 at 13:02):

We’ve said that we will prohibit scripts - security nightmare

Fair enough. Will keep thinking!

Dunmail (Nov 25 2019 at 13:14):

We’ve said that we will prohibit scripts - security nightmare

Next line of attack!

If the publisher adds a package-lock.json, the dependencies should resolve correctly using npm.
package.json:

{
  "name": "fhir-package",
  "version": "1.0.0",
  "dependencies": {
    "hl7.fhir.r3.profiles": "^3.0.1-preview2"
  }
}

package-lock.json:

{
  "name": "fhir-package",
  "version": "1.0.0",
  "lockfileVersion": 1,
  "requires": true,
  "dependencies": {
    "hl7.fhir.r3.profiles": {
      "version": "3.0.1-preview2",
      "resolved": "https://packages.simplifier.net//hl7.fhir.r3.profiles/-/hl7.fhir.r3.profiles-3.0.1-preview2.tgz",
      "integrity": "sha512-zPU5H8lWyM1D/6cWWYG6TXBHJGEPEwVi8GsOkkrh5bbjYSMdVM0ZABOC0Xbdxtxoln3RChZekdFx9Cz1eDL5gQ==",
      "requires": {
        "hl7.fhir.r3.profiles.extensions": "3.0.1-preview1",
        "hl7.fhir.r3.profiles.others": "3.0.1-preview1",
        "hl7.fhir.r3.profiles.resources": "3.0.1-preview1",
        "hl7.fhir.r3.profiles.types": "3.0.1-preview1"
      }
    },
    "hl7.fhir.r3.profiles.extensions": {
      "version": "3.0.1-preview1",
      "resolved": "https://packages.simplifier.net//hl7.fhir.r3.profiles.extensions/-/hl7.fhir.r3.profiles.extensions-3.0.1-preview1.tgz",
      "integrity": "sha512-uzc1Zn6/RF3UBTSJ2m+/Auxd3Ma+PIJGIJksniSWUJtuJ1pN8COwn1IbuMWkTcHDzDkg5HXUvVUlbNz4nWl5sg=="
    },
    "hl7.fhir.r3.profiles.others": {
      "version": "3.0.1-preview1",
      "resolved": "https://packages.simplifier.net//hl7.fhir.r3.profiles.others/-/hl7.fhir.r3.profiles.others-3.0.1-preview1.tgz",
      "integrity": "sha512-GlscITM9wBAL92e4JI2GSnHO0sM6cYIIjMur+gXKeoyh8WE0aahupCq+jNeJ6Buglrq2EYc7v2I05HHzMiA7Uw=="
    },
    "hl7.fhir.r3.profiles.resources": {
      "version": "3.0.1-preview1",
      "resolved": "https://packages.simplifier.net//hl7.fhir.r3.profiles.resources/-/hl7.fhir.r3.profiles.resources-3.0.1-preview1.tgz",
      "integrity": "sha512-HRRUd3T/AdR4B4JJc9YPOxcqFGf8CxwtwEojpBPVnafg4+0KizF3DINmYgNtnYdaJq/JUhrhd9NKqL6jeMBpPQ=="
    },
    "hl7.fhir.r3.profiles.types": {
      "version": "3.0.1-preview1",
      "resolved": "https://packages.simplifier.net//hl7.fhir.r3.profiles.types/-/hl7.fhir.r3.profiles.types-3.0.1-preview1.tgz",
      "integrity": "sha512-Si02FU/olOVGONL6n0Ac2acmaR3Uwlq/ewYf0FjPBzzsbcK4+4OqN8y2bprB8qDG63UGBgrw+ykbpTUE4itZyA=="
    }
  }
}

Hans van Amstel (Nov 25 2019 at 13:14):

The problem I have with the current format is that it does not allow me to use the default NPM tooling like: nmp publish and npm pack. Also having the package.json in the same folder as the resources is not really desirable as it contains different content as the other json files.

Which tools, commands did you use for creating the hl7 packages?

I agree it is not clearly written in the documentation. A package is created from a directory that contains a package.json file and the content. When creating a package of a directory that contains a package.json file a package directory is created by the npm tools that contains all the files and directories including the package.json file. This package directory is then the main entry point of the npm package, as shown below. The directory name is used as the package name.
This is described in: https://docs.npmjs.com/cli/publish, https://docs.npmjs.com/creating-a-package-json-file and https://docs.npmjs.com/misc/developers

I would like to propose the following package structure:
<package-name>
- package
- -json Mandatory (json snapshots)
- - xml optional (xml snapshots)
- -customSubdir optional (n times)
- -package.json mandatory

The in between package folder is added by the tools. I used npm install and npm pack on the following directory structure to create the above package:

<package-name>
- json Mandatory (json snapshots)
- xml optional (xml snapshots)
- customSubdir optional (n times)
- package.json

Dunmail (Nov 25 2019 at 13:34):

The problem I have with the current format is that it does not allow me to use the default NPM tooling like: nmp publish and npm pack.

:+1:

Grahame Grieve (Nov 25 2019 at 16:04):

The problem I have with the current format is that it does not allow me to use the default NPM tooling like: nmp publish and npm pack

@Hans van Amstel @Dunmail do the changes I proposed above resolve this?

Grahame Grieve (Nov 25 2019 at 16:04):

I don't understand the bit about package-lock.json. Why is package.json not sufficient?

Dunmail (Nov 25 2019 at 16:55):

I don't understand the bit about package-lock.json. Why is package.json not sufficient?

npm is configured with a default package repository.
npm can install a package from an alternate repository (e.g. simplifier.net) but will try to resolve any dependencies in package.json from the default package repository (typically npmjs.com).
In this case, npm fails to install dependencies.

package-lock.json has supplementary info including the location from which the dependency was resolved.
npm will use this information to resolve the dependencies from the the location described in package-lock.json (e.g. simplifier.net)
In this case, npm installs the dependencies.

I think it would be relatively easy to add package-lock.json as a step in the simplifier.net publish service.

Happy to put a pair of demo packages onto npmjs.com if that helps!

Hans van Amstel (Nov 26 2019 at 07:39):

@Grahame Grieve Where can I see these proposed changes?

Grahame Grieve (Nov 26 2019 at 07:43):

What I propose:
1. update the FHIR NPM Package spec to also say this, and say that sub-folders of the package spec are not described (instead of siblings)
2. regenerate all the packages on hl7.org/fhir.org to use subfolders instead of sibling folders
3. reload on packages.fhir.org

Grahame Grieve (Nov 26 2019 at 07:44):

turns out that i accidentally uploaded the revised packages overnight - so you can check say http://hl7.org/fhir/us/core/package.tgz to see the change in practice

Hans van Amstel (Nov 26 2019 at 10:38):

The structure that is used in http://hl7.org/fhir/us/core/package.tgz, to have the content under the package folder instead of siblings, works well with npm pack and npm install. :+1:
What is the reason to not have a json sub folder as now the the package.json file is somewhere in the list of json fhir resources. For the tooling it does not matter but tools using the resources must now filter out the package.json file. I have no real preferences other than that it looks a bit cleaner.

@Dunmail Why is a package-lock.json in the package needed. According to the documentation it is recommended not to publish this. If the problem is that the dependencies are not pulled for the fhir repo you can try npm config set registry <FHIRrepoURL> before installing. This will make npm install try to pull all the dependencies from the registry at <FHIRrepoURL>. One of the big drawbacks when using package-lock.json is is that it only works for the top package. This means that when I have a package that depends on a package which contains a package-lock.json this package-lock.json from the dependency is ignored and the original problem remains.

Dunmail (Nov 26 2019 at 10:48):

Dunmail Why is a package-lock.json in the package needed. According to the documentation it is recommended not to publish this. If the problem is that the dependencies are not pulled for the fhir repo you can try npm config set registry <FHIRrepoURL> before installing. This will make npm install try to pull all the dependencies from the registry at <FHIRrepoURL>. One of the big drawbacks when using package-lock.json is is that it only works for the top package. This means that when I have a package that depends on a package which contains a package-lock.json this package-lock.json from the dependency is ignored and the original problem remains.

@Hans van Amstel
We'd like to create a service that includes a mix of FHIR packages and standard packages so resetting the npm registry isn't a practical option.

It's possible that a package-lock.json isn't the solution!

Martijn Harthoorn (Nov 26 2019 at 12:26):

I am
- fine with moving the folders to be subfolders of /package/
- careful to negative about moving resources to /package/json, because it would break all packages that have already been produced.
- "regenerate all the packages on hl7.org/fhir.org" is not really a thing in the world of packes. People depend on it. With some effort, we can make it work once, but I strongly hope this is not going to be a regular practice.

Hans van Amstel (Nov 26 2019 at 13:44):

@Martijn Harthoorn I agree that this should not be regular practice. But the structure as defined was in proposal, as I understood from the DevDays2019 talk.
About moving the resources to /package/json I do not have a real preference but as the packages needs to be regenerated it can be done in one go.

@Dunmail I get the need for this we actually have the same need and use for this a setup with a virtual registry that contains a local registry and a remote one. This has the added advantage that it also functions as a caching proxy and therefore are less depended on packages to not change just before doing a release for example. But I think this is beyond the discussion of the package format.

Grahame Grieve (Nov 26 2019 at 14:51):

Regenerate all the packages

I agree that this is not something we should do regularly. And in fact, I only propose this because I don’t care whether it gets distributed or not. But it would be good to get them into packages.fhir.org

Grahame Grieve (Nov 26 2019 at 19:51):

careful to negative about moving resources to /package/json, because it would break all packages that have already been produced.

I basically agree with this. Sure, we could do it, but it's not hard to separate out package.json and .index.json (and .index.json will still exist)

Last updated: Apr 12 2022 at 19:14 UTC