FHIR Chat · SUSHI / IG Publisher Integration Proposal · shorthand

Stream: shorthand

Topic: SUSHI / IG Publisher Integration Proposal

---

Chris Moesel (Aug 05 2020 at 01:06):

We've been thinking about how to improve the SUSHI / IG Publisher integration. Here is a proposal to start some discussion. We'd love your feedback. Full Disclosure: I am going on two weeks vacation tomorrow, but please discuss anyway -- @Mark Kramer and others on the SUSHI/FSH team can engage.

Goal: To allow FHIR Shorthand (FSH) to be more easily integrated into FHIR Implementation Guides using the standard FHIR IG template project structure, avoiding the need for special folders like ig-data.

BLUF:

• Authors maintain .fsh files in input/fsh
• The config.yaml is maintained in the root folder (but is optional for FSH-only users)
  • If config.yaml is provided, SUSHI will generate the ImplementationGuide JSON and other IG files (as indicated in the config.yaml file), in addition to the FHIR definitions
  • If config.yaml is not provided, SUSHI will generate FHIR definitions only and will use the provided ImplementationGuide JSON to determine the canonical.
• SUSHI writes all output to the locations IG Publisher expects. This means that some generated files may be intertwined with author-maintained (non-generated) files.
• SUSHI provides two new capabilities to help authors manage generated files:
  • sushi --clean deletes all generated files, allowing authors to perform "clean builds"
  • sushi --update-ignore updates the .gitignore file to ignore all generated files (so they aren't checked in)

PLEASE FOLLOW THIS LINK FOR EXAMPLES AND DETAILS -----> https://gist.github.com/cmoesel/af6c05e008f726a89dd55505a8190f4b

Jose Costa Teixeira (Aug 05 2020 at 05:35):

1. How are generated files? how does sushi detect them for the --clean?
   For example I was using sushi to generate a first codesystem, but then had to update it separately to create translations.

Jose Costa Teixeira (Aug 05 2020 at 05:45):

2. I would prefer a post-processing step rather , but that is probably because I don't understand the .gitignore. I do want my committed files to look exactly like those I have in my local build

Nick Freiter (Aug 07 2020 at 12:51):

1. If the file is defined as FSH, sushi would consider that a generated file. I think in your case of the CodeSystem, sushi would detect it and try to clean it. The separate updates would be overwritten. But in this context, couldn't you solve that problem by letting sushi generate the CodeSystem once, and then commenting out that section of FSH?
2. I'm not sure what you mean by preferring a "post-processing" step. You prefer this in comparison to what?

Jose Costa Teixeira (Aug 07 2020 at 13:45):

Nick Freiter said:

1. If the file is defined as FSH, sushi would consider that a generated file. I think in your case of the CodeSystem, sushi would detect it and try to clean it. The separate updates would be overwritten. But in this context, couldn't you solve that problem by letting sushi generate the CodeSystem once, and then commenting out that section of FSH?

Thanks. I can comment it out, but how can I use it in a valueset? I think I would have to copy the codesystem.json to my input folder (ig-data) as a BYOJ.

Jose Costa Teixeira (Aug 07 2020 at 13:46):

i guess I just don't see what is the .gitignore doing. Is it just telling github to ignore some local files and not push them to github?

Jose Costa Teixeira (Aug 07 2020 at 13:47):

If so, that will likely get the local build and CI build misaligned more often than not... which I think is a bad idea.

Jose Costa Teixeira (Aug 07 2020 at 13:52):

by post-processing I mean something like:

1. sushi should do its work as a parser in a constant way - always same behaviour - fsh +BYOx in, json out.
2. after that processing, sushi should generate the ig content in sushi's --output folder, (called input folder)
3. finally, what was already defined in the IG (BYOx and ig.ini etc) should just (optionally) be carried over to the sushi's --output folder

Jose Costa Teixeira (Aug 07 2020 at 14:53):

I'm just looking for a way that sushi can be working to create a part of an already existing IG. I don't know exactly how this should work but my concernsare :

• command line parameters cannot be used in the CI build or it's hard to pass them via the IG publisher. all should be autonomous.
• sushi can be run standalone or be part of the IG chain. like most tools, the hard part is to support the latter

Jean Duteau (Aug 07 2020 at 16:01):

Jose Costa Teixeira said:

i guess I just don't see what is the .gitignore doing. Is it just telling github to ignore some local files and not push them to github?

Yes. and most repositories use the .gitignore both FHIR and non-FHIR. As an example, I don't check in any of the temp directories or any of the .DS_Store files that Macs create. I put that in the .gitignore.

Jose Costa Teixeira (Aug 07 2020 at 16:16):

Sorry, I meant I know what it does. I don't know what good will bring 'ignore all generated files'

Jose Costa Teixeira (Aug 07 2020 at 16:18):

Just seems to create a nice setup for people to go asking 'this works on my machine but the ci publisher says it's missing a file. Must be something with the publisher, because my machine works fine' - and those issues will be very hard to help debug, right?

Nick Freiter (Aug 07 2020 at 17:45):

Jose Costa Teixeira said:

Just seems to create a nice setup for people to go asking 'this works on my machine but the ci publisher says it's missing a file. Must be something with the publisher, because my machine works fine' - and those issues will be very hard to help debug, right?

Well, the idea would be that the .gitignore file would only ignore JSON files generated by sushi. When the IG Publisher runs sushi locally and those files are present, they will be overwritten. When the IG Publisher runs sushi on the ci, those files would not be present, since they would be ignored, so they would be generated freshly by sushi.

Jose Costa Teixeira (Aug 07 2020 at 18:00):

Ok, I see it now. I think this is a complicated way and a small in rent - I'd rather se a more robust (redesign)

Nick Freiter (Aug 10 2020 at 12:23):

Jose Costa Teixeira said:

Ok, I see it now. I think this is a complicated way and a small in rent - I'd rather se a more robust (redesign)

Which aspects of this approach do you feel aren't robust? Is it the part with .gitignore excluding generated files from being checked in? To be clear, that is an optional feature. If someone wants their generated files to be checked into source control, they do not have to use the --update-ignore script.

Jose Costa Teixeira (Aug 10 2020 at 13:02):

Nick Freiter said:

Which aspects of this approach do you feel aren't robust? Is it the part with .gitignore excluding generated files from being checked in? To be clear, that is an optional feature. If someone wants their generated files to be checked into source control, they do not have to use the --update-ignore script.

Thanks. First, by "small in rent" i meant " small increment" - I mean that the ig-data feels like it was designed for sushi to be fully autonomous in providing the entirety of the content, and not designed for the fact that sushi is one component in the IG publishing chain. So these seem like local optimisations, but I would rather step back and see would be a way to do fsh parsing and IG preparation - if we did not have that ig-data folder, would we do things the same way?

Jose Costa Teixeira (Aug 10 2020 at 13:03):

I do not know why .gitignore thing - I'm sure there are legitimate use cases, but this may create a big confusion between local build and remote build.

Nick Freiter (Aug 10 2020 at 13:33):

Well to us, this proposal does seem like a way of making SUSHI a part of the IG Publisher chain. If someone does not want to manage IG Publisher files via SUSHI, they would only have to put FSH files in input/fsh, and SUSHI would use the ImplementationGuide JSON file to get the information it needs to process those FSH files into JSON. That seems like it is essentially as minimally invasive as possible.

Jose Costa Teixeira (Aug 10 2020 at 15:09):

I appreciate the direction. What I've been experiencing is that it's not as simple as putting fsh file in the input/fsh. Those are my challenges.
It's ok to have some command line options, but in a CI build the user will not be able to add those.

Jose Costa Teixeira (Aug 10 2020 at 15:10):

the .gitignore I fail to see the purpose, honestly.

Jose Costa Teixeira (Aug 10 2020 at 15:50):

concrete improvement suggestions:

1. in the integration, we could use the -o parameter to let sushi write its output to a "sushi-generated" folder - we don't have to mix them with the existing files in the input/* folder. That additional folder is quite easy to manage by any of the tools.
   1.1. instead of gitignore on file level, we can just -gitignore that folder.

2. Would it be interesting to we add a parameter to sushi for "additional folder for BYOx"?
   2.1 Perhaps that would even have input/ as default, and not need this at all all sushi stuff is in a separate folder

Nick Freiter (Aug 10 2020 at 15:58):

Your second suggestion is a part of this proposal, SUSHI would just look in the surrounding input/ folders for the BYO content, so we agree that something along those lines could be useful.

Jose Costa Teixeira (Aug 10 2020 at 16:07):

The first suggestion would also be simpler than a complicated.gitignore and --clean, I think. But if this starts working I can propose to put this in the template for example

Nick Freiter (Aug 10 2020 at 17:42):

We are also wary of requiring other parts of the IG Publisher toolchain to do more work to support sushi. The proposal above would only require that the IG Publisher runs sushi if it sees an input/fsh folder (as opposed to the current behavior of running sushi if it sees a fsh folder), and then sushi would take care of the rest, and the IG Publisher could run as normal. If we were to make a sushi-generated folder, the IG Publisher would now have to look through that folder for definitions.

Jose Costa Teixeira (Aug 10 2020 at 18:53):

Looking in an additional folder is easy - it's one parameter that can be added to the template.

Jose Costa Teixeira (Aug 10 2020 at 18:56):

I'm looking for the minimal changes to sushi that provide the ease for most users but reserve power to advanced users

Jose Costa Teixeira (Aug 10 2020 at 19:06):

Sushi would only need to implement something like my suggestion 2. #1 we can do in the integration - you don't need to mess with .gitignore or perhaps even the -clean option.

Nick Freiter (Aug 10 2020 at 19:19):

It could be a bit confusing to have a sushi-generated folder, because some people will still use sushi to generate things like ImplementationGuide JSON,ig.ini, and menu.xml based on config.yaml, and those will not be in that folder. That being said, it would provide a more clean separation between author maintained (non-generated) files and generated files, which is an admitted weakness of Chris's proposal above. If it is easy enough for the IG Pub to just look in one more folder, I can't speak for the rest of the team, but to me it seems that is more simple than intermixing definitions among non-generated resource. If any other users have feedback on how this could influence their usage of sushi, we would love to hear it as well.

Jean Duteau (Aug 10 2020 at 19:20):

Chris Moesel said:

We've been thinking about how to improve the SUSHI / IG Publisher integration. Here is a proposal to start some discussion. We'd love your feedback. Full Disclosure: I am going on two weeks vacation tomorrow, but please discuss anyway -- Mark Kramer and others on the SUSHI/FSH team can engage.

I really like this proposal. I have a couple of questions but overall this makes complete sense to me and goes much further towards a decent IGPublisher integration.

1) I don't understand what the line "In addition, the --clean flag can be used when building FSH source to establish a clean build." is referring to in the section on sushi --clean.
2) Would it be possible, if I wanted to segregate my FSH files that way, to put FSH files in the various input directories? i.e. if myfile1.fsh only has profiles, could I put it in input/profiles and have SUSHI generate appropriate JSONs from that? That's not how I currently build my FSH files but I can see someone wanting a one-to-one correspondence between artifacts and FSH files.

Nick Freiter (Aug 10 2020 at 19:35):

1) I am also unsure about this actually. I think what this is meant to mean is that you can do sushi --clean to delete all sushi generated files, or you can do sushi --clean /path/to/project to delete and then rebuild all sushi generated files. But, since sushi overwrites files anyway, I'm not sure why you would need to delete all the files and then rebuild, as opposed to just overwriting.
2) We've considered structures like this in the past. On the sushi end, sushi already works by recursively looking for .fsh files, so it would not be hard for sushi to just search through a set of folders for .fsh files instead of a single folder. But I think this would complicate things for the Publisher. Instead of triggering a sushi run if there is a fsh (or input/fsh) folder, the Publisher would have to search through all folders for .fsh files, and trigger a run of sushi based on that.

Jose Costa Teixeira (Aug 10 2020 at 19:36):

Nick Freiter said:

It could be a bit confusing to have a sushi-generated folder, because some people will still use sushi to generate things like ImplementationGuide JSON,ig.ini, and menu.xml based on config.yaml, and those will not be in that folder.

Depends. If we want to be thorough, we could make sushi-generated contain a input folder for the generated conformance resources only, and the other files in the parent folder.
In my view, those other files don't need to be regenerated every time that sushi runs. For me, creating those files is something that you do once on setup or every time you change the corresponding input. I'd suggest separating that (setup file generation) from the conformance resource generation. (check @Jean Duteau 's suggestion a few weeks ago to have sushi default to FSHOnly or accept it as a command line parameter)

Jose Costa Teixeira (Aug 10 2020 at 19:48):

Nick Freiter said:

But I think this would complicate things for the Publisher. Instead of triggering a sushi run if there is a fsh (or input/fsh) folder, the Publisher would have to search through all folders for .fsh files, and trigger a run of sushi based on that.

If sushi can get all the *.fsh files in the --input folder and subfolders, I don't see why the publisher needs to do something.

Nick Freiter (Aug 10 2020 at 19:57):

Jose Costa Teixeira said:

If sushi can get all the *.fsh files in the --input folder and subfolders, I don't see why the publisher needs to do something.

How would the IG Publisher know to run sushi in the first place? Currently as far as I know, the running of sushi is triggered by the presence of the fsh folder. If .fsh files are all spread out, the Publisher needs to look through all folders to see if any .fsh files are present, so that it can know sushi should even run.

Jose Costa Teixeira (Aug 10 2020 at 19:58):

can we not safe-run sushi and have sushi exit with "0 - nothing to do" ?

Grahame Grieve (Aug 10 2020 at 22:12):

We are also wary of requiring other parts of the IG Publisher toolchain to do more work to support sushi

I don't mind if it makes for a better experience for users

Jose Costa Teixeira (Aug 10 2020 at 22:35):

From what I learned today, a few changes that I think would make for a better integration:

1. Make sushi work by default (or via command line option) in FSHOnly mode.
2. Make the ig-data folder a command line parameter (remains the folder that contains non-fsh stuff like the BYOx, or the ig.ini, ig.xml, or package.json(?)...)
3. Get sushi to exit gracefully if there are no fsh folders in the input folder (so that you can always run sushi without having to detect an fsh folder)
4. Eventually add more input folders? Not sure if needed.
5. Change the command line in the publisher accordingly: sushi [-FSHOnly] [-ig-data=.] ./input [-o sushi-generated]
6. Change the template to add "sushi-generated" as a resource folder

Nick Freiter (Aug 11 2020 at 16:58):

One other thing we considered is trying to rename files that are generated by sushi to indicate as such. Aka a StructureDefinition generated by sushi may be in a file called StructureDefinition-Foo-generated.json instead of StructureDefinition-Foo.json. Is something like that possible with the IG Publisher? In experimenting it seems we couldn't do this, due to the Publisher requiring files to follow the format resource-id.json (or a related format).

Jose Costa Teixeira (Aug 11 2020 at 17:25):

yes, that will be an issue. see
https://chat.fhir.org/#narrow/stream/215610-shorthand/topic/Resource.20filename.20prefix

Jose Costa Teixeira (Aug 11 2020 at 17:27):

(sorry, sent wrong link before)

Jose Costa Teixeira (Aug 11 2020 at 17:27):

using a separate folder would be easy on both sides, I guess. I don't know if it has significant downsides.

Grahame Grieve (Aug 11 2020 at 19:21):

@Nick Freiter I could modify the IG publisher to support that, but I'm not convinced it's the right solution

---

Last updated: Apr 12 2022 at 19:14 UTC