FHIR Chat · pipeline scripting · IG creation

Stream: IG creation

Topic: pipeline scripting

Jens Villadsen (Mar 08 2021 at 15:28):

@Ward Weistra does the Firely Terminal have the capability of being able to validate a FHIR package. It's a bit of a wierd question, but it is about getting faster from the IG to the demo setup. Currently, the IGPublisher (CI) setup allows a FHIR package to contain invalid FHIR resources (@Jose Costa Teixeira should listen in on this). What I would like to build (or expand) is a pipeline that takes the output from the IG publisher (being the NPM package) - and validates whether the package contains example resources that cannot be parsed and so forth. If the package does contain invalid examples or invalid sdf's the pipeline should probably email someone responsible. If the package does NOT contain invalid resources/examples, the pipeline should proceed and spin up eg. a HAPI FHIR server with the current state of the IG and eg. enable validation of the resources.

If Firely Terminal does support the feature of being able to validate a package, please ping me (in that case I'll wrap it in docker and take it from there). If not, I'll probably just build the tooling myself.

Jens Villadsen (Mar 08 2021 at 15:33):

The whole idea is to make the IG as executable as fast as possible

Ward Weistra (Mar 09 2021 at 09:51):

@Jens Villadsen Good stuff. Firely Terminal has the option to validate resource one-by-one (fhir validate). And with the next Terminal release we're adding Quality Control to that, including the option for bulk validation and checking any other FHIRPath rules (fhir check).

Before validation you'll want to make sure you install the package dependencies (fhir restore which reads the package.json and installs the dependencies listed)

Ward Weistra (Mar 09 2021 at 09:53):

fhir check will be a Firely Terminal option that is customer-only (any Simplifier.net plan), which shouldn't be a problem for you, but perhaps if you want to have others without a license using it.

Jens Villadsen (Mar 09 2021 at 12:56):

Hmmm ... thats good news and not-so-good news at the same time. I have full understanding for your choices of licensing. You choice however probably pulls me in the direction that I'll have to cook it up myself so that it can go in anyones pipeline - but thx for the response

Ward Weistra (Mar 09 2021 at 14:18):

And I understand that @Jens Villadsen . We'll publish a Github Actions pipeline soon on how to use Firely Terminal for bulk validation on every commit/PR. Perhaps as inspiration.

Grahame Grieve (Mar 30 2021 at 19:26):

Currently, the IGPublisher (CI) setup allows a FHIR package to contain invalid FHIR resources

Uhh? @Jens Villadsen it shouldn't allow this without notifying the IG author via qa.html

Jens Villadsen (Mar 30 2021 at 21:01):

@Grahame Grieve It does make some noise in the qa page. The problem I encountered was invalid example resources, specifically, and those are part of the IG eventhough I would recommed to blacklist them in the resulting package. Bear in mind - I'm trying to automate stuff here - not bombarding an IG author. The thing is that (I imagine) while only computable profiles are part of the package, invalid example resources are also part the package, which breaks my build chain currently. Hence the need for a tool that can detect and/or remove invalid example resources as they stop the process of automatically rolling out a FHIR server with the validation capabilities of the IG

Grahame Grieve (Mar 30 2021 at 21:04):

I'm interested in other people's opinions on this. I could either:

• ignore Jens
• restrict what goes in packages to only valid resources for examples
• restrict what goes in packages to only valid resources
• create a special package that only contains valid resources

One of the tricky things about this is that different people will care about different errors. e.g. most people won't care about a wrong code, or a broken link, but others will

Lloyd McKenzie (Mar 30 2021 at 21:17):

What constitutes 'valid' is a hard thing to say. The validator continues to evolve and get smarter - what it catches today and what it caught two years ago aren't the same. Terminologies can also evolve in ways that make instances valid or invalid. Finally, there are some rules that automated validators might never be able to detect - or at least that require human review. For example, having bad codes for an extensible binding.

Also, sometimes an IG might deliberately have invalid examples - to show what not to do, to show ramifications of certain inputs, etc.

If a server wants to have rules that it will only store instances that pass its own validation rules, then I think that's an obligation for the server. If an IG published with instances that don't meet those rules, the server is free to kick them out. It's also free to reconsider what data it stores as its own rules evolve.

Jose Costa Teixeira (Mar 30 2021 at 21:25):

could this be a command line argument?
--ignore-errors (default) / --skip-invalid-examples / --skip-invalid... /--use-qa

Jose Costa Teixeira (Mar 30 2021 at 21:27):

--use-qa: only skip the errors that are in the ignore-warnings (if ignore-warnings can also contain errors to skip

Jose Costa Teixeira (Mar 30 2021 at 21:30):

Or perhaps a simple option /ig parameter "-skip-validation-errors" (default=true). If false, the publisher will return some error code ( not 0) and the pipeline can detect it

Jens Villadsen (Mar 30 2021 at 21:50):

@Grahame Grieve I thought you already did ignore me ;)
I agree that context is everything here (as always), and yes, different people are with different needs. I could actually just (for once) ignore @Grahame Grieve / @Lloyd McKenzie and just hack my way through it by making a PR to HAPI FHIR that parameterizes how the IG loading capacity should treat examples and take it from there. In my case, invalid examples are worthless, as they clutter up the picture by being present on a testing server that should only show what is actually considered valid.

Jens Villadsen (Mar 30 2021 at 21:53):

As a bonus info, the HAPI FHIR starter project may not be the only piece of software that is able to be bootstrapped within seconds with an entire range of IG's. Something tells me that AWS is also looking into this: https://github.com/awslabs/fhir-works-on-aws-deployment/commit/393846b8edd6b9d1a016a8968c8b163c3ba61658

Grahame Grieve (Mar 30 2021 at 21:57):

I thought you already did ignore me ;)

I only pretend to some times.

Jens Villadsen (Mar 30 2021 at 22:00):

Anyways - keep me posted if you actually intend to do something about this. @Jose Costa Teixeira from top of my head, it sounds a bit intimidating if running with a parameter that is not default can change the result - while that may only sound intimidating to me.

Jens Villadsen (Mar 30 2021 at 22:01):

Maybe that would actually be the correct way to do it for this particular task

Jens Villadsen (Mar 30 2021 at 22:01):

Unless I just fix it with adjusting HAPI.

Grahame Grieve (Mar 30 2021 at 22:02):

I suspect that the answer lies in reading the package. What if I add validity information to .index.json?

Jens Villadsen (Mar 30 2021 at 22:02):

That could actually work

Jens Villadsen (Mar 30 2021 at 22:03):

I mean ... it wouldn't hurt to actually have a machine readable format of the QA page

Jens Villadsen (Mar 30 2021 at 22:03):

and that would fit the bill to my setup

Grahame Grieve (Mar 30 2021 at 22:04):

well, there is qa.json, you know. but it's probably not organised right for what you want

Grahame Grieve (Mar 30 2021 at 22:04):

not it doesn't have what you need at all.

Grahame Grieve (Mar 30 2021 at 22:05):

qa.xml does, but not nice

Jens Villadsen (Mar 30 2021 at 22:07):

no .. its 'not nice'

Jens Villadsen (Mar 30 2021 at 22:09):

and qa.json is .... too slim

Lloyd McKenzie (Mar 30 2021 at 23:12):

An OperationOutcome from hell? :) Or perhaps a Bundle of OperationOutcomes - one per resource?

Oliver Egger (Mar 31 2021 at 06:10):

we currently test our validator setup with testing all ig examples against it (and exclude in the test setup explicitly which examples we know fail due to that they already have errors), so if we could get that information from the package in some way that would very nice.

John Moehrke (Mar 31 2021 at 12:29):

I deliberately put in non conforming resources to show the edges of what to test. I don't mind needing some way to tag them as deliberately non conforming.

Martijn Harthoorn (Apr 01 2021 at 07:02):

.index.json is intened to be generated - or even regenerated by multiple tools. If we want to keep .index.json cross tool compatible, we should be very conservative in making changes there.
I would also suggest that reading a resource always has the risk of being not valid. It's something every piece of software should deal with.
However we might add a marker in package.json to at least state that it's a preview/alpha/untested/non-qa package

Brian Postlethwaite (Apr 08 2021 at 08:32):

Or a new json file beside the index, so regeneration doesn't lose the data. The bundle of OperationOutcome almost feels like a good idea.
The could pick what violations you can accept...

Grahame Grieve (Apr 20 2021 at 21:48):

ok so from the next IG publisher release, it will add two new files to the package it produces:

/other/validation-summary.json
/other/validation-oo.json

ValidationSummary is an easy to read JSON file with an entry for each resource like this:

 "{type}/{id}": {
    "errors": 0,
    "warnings": 1
  },

validation-oo.json is a bundle of OperationOutcomes, one for each resource, like this:

    {
      "fullUrl" : "{ig-canonical}/OperationOutcome/{type}-{id}",
      "resource" : {
        "resourceType" : "OperationOutcome",
        "id" : "{type}-{id}",
        "issue" : [
          errors and warnings
        ]
      }
    },

if there's no errors or warnings, then there'll be be no issue property

Lloyd McKenzie (Apr 20 2021 at 22:11):

Is there a reason it needs to go in /other rather than /output or /temp? (I'd prefer to keep directory clutter to a minimum.)

Also, what's the driver for having this in a more computable form?

Grahame Grieve (Apr 20 2021 at 22:16):

it's in the package not the directory structure. And the driver is further back in this thread, people having problems with processing the packages because the IG publisher will produce IG packages with invalid resources

Lloyd McKenzie (Apr 20 2021 at 22:20):

My brain interpreted "to the package" as "to the generated output". Sorry for not paying more attention.

Last updated: Apr 12 2022 at 19:14 UTC