FHIR Chat · sushi - ig publisher integration · IG creation

Stream: IG creation

Topic: sushi - ig publisher integration

---

Jose Costa Teixeira (May 05 2020 at 06:04):

I see that sushi has replaced my ig.ini. That is really blocking for me.
How should this work together? Sushi is facilitating the straightforward use but I don't think it should not take over more advanced use.

Jose Costa Teixeira (May 05 2020 at 06:06):

I don't want sushi to replace my entire ig framework - the ig.ini, the ig.xml/json... I just want sushi to create the content from my shorthand definitions, and create the other files ONLY if they do not exist

Jose Costa Teixeira (May 05 2020 at 06:09):

instead of looking into the ig-data, why doesn't sushi take the existing files? I don't want to have an IG template when using sushi and another when not using sushi.

Grahame Grieve (May 05 2020 at 06:12):

@Chris Moesel but probably a question for #shorthand

Jose Costa Teixeira (May 05 2020 at 06:27):

(anywhere is fine, but this is impacting /impacted by the work on the ig template and sample ig).

Jose Costa Teixeira (May 05 2020 at 06:45):

Jose Costa Teixeira said:

image.png

This was a quick draft but should still apply - if sushi is being called by the publisher, it just needs to look at different folders.

Chris Moesel (May 05 2020 at 13:07):

@Jose Costa Teixeira -- the main trouble is telling if a file exists because the author put it there or if it exists because SUSHI was already run previously. If I understand your diagram correctly, then if we were to implement that, we would run into this potential scenario:

• First run: ../ig.ini does not exist, so SUSHI uses ig-data and outputs to ..
• Author finds issues and makes changes in ig-data that affect generated ig.ini or other config files
• Second run: ../ig.ini now exists (thanks to previous run) so SUSHI ignores ig-data and no changes get incorporated

This is the tricky problem of trying to support generation of files and user-management of files when the user wants to manage the file in the same name and location as the app wants to generate it. We may have to come up w/ creative solutions. Those might include one (or multiple) of the following:

• only overwrite the destination file if it contains a special SUSHI header (or other indicator) indicating it is generated (in which case, authors should remove it if they want to make manual modifications)
• specify in config which files are user-provided (that SUSHI might normally generate) -- and SUSHI won't touch them
• specify in config to run artifact-only mode which only does profiles, extensions, etc and never does any IG-related config (or pages or menus, etc)
• ... other ideas...

Specific to your current issue with ig.ini, is there a reason you don't just put an ig.ini in ig-data? If you do that, SUSHI will merge its inputs with yours and copy over the merged file to ../ig.ini.

Jose Costa Teixeira (May 05 2020 at 13:25):

right. does this presume that sushi will always output the ig.ini file?

Jose Costa Teixeira (May 05 2020 at 13:29):

ig.ini only has 2 parameters..
sushi could generate an ig.ini only if one does not exist (and never touch it again)

John Moehrke (May 05 2020 at 13:44):

I think that if someone is wanting to use pure sushi to produce IG output, the current solution is great. So the problem is when someone wants to use sushi AND also hand edit some stuff... First, we need to identify the reasons one needs to hand edit, all likely good reasons but knowing them would inform how we avoid stomping on that. We need a compromise between the pure sushi to IG output, with the hybrid editing.

Chris Moesel (May 05 2020 at 13:45):

I think this pertains to other files as well -- not just ig.ini -- so we want a more general solution that can handle them all. The current thinking on SUSHI is that having config data spread across multiple files (ig.ini, package.json, package-list.json, menu.xml (arguably config), ImplementationGuide.json) is generally confusing -- and consolidating to a single configuration file can streamline that and reduce redundant and/or possibly conflicting data between those files. So, the config (not yet released) would contain the data that would go in the ig.ini (e.g., "template"). If SUSHI generates ig.ini only once, then if the author changes the template value in the config, that change will not get reflected in the ig.ini. We would, at the least, want to trigger a warning if we found that data in the config didn't match data in the ig.ini.

Chris Moesel (May 05 2020 at 13:45):

@John Moehrke -- agreed, and I think we'll be in a better place to explore that once the new config is released.

Jose Costa Teixeira (May 05 2020 at 14:17):

well this is difficult.

Jose Costa Teixeira (May 05 2020 at 14:18):

I can expect an author to either know how to change the ini (or get that ini provided to them - for example IHE editors), so sushi should not interefere

Jose Costa Teixeira (May 05 2020 at 14:19):

I don't understand what menu.xml has to do with this - i agree a better way should happen, but that is content. ig.ini is config

Jose Costa Teixeira (May 05 2020 at 14:20):

so I don't understand - one config file for all those things?

Jose Costa Teixeira (May 05 2020 at 14:20):

I don't understand why the simplest solution would not work - sushi sees an ig.ini, leaves it alone.

Jose Costa Teixeira (May 05 2020 at 14:24):

I fear this could make sushi collide with the advance uses that IHE will need - @John Moehrke I know some people in IHE are experimenting with sushi but we must be careful (again). At this moment, I experimented with it, and I am learning to be cautious.

Jose Costa Teixeira (May 05 2020 at 14:30):

Chris Moesel said:

I think this pertains to other files as well -- not just ig.ini -- so we want a more general solution that can handle them all.

Perhaps but we should clean up a bit - ig.ini is input for sushi, not output of a complex decision. Sushi can create a default one if needed - but it's 2 parameters...

Lloyd McKenzie (May 05 2020 at 15:36):

What's the rationale for Sushi re-writing the ig.ini? The only circumstance I can think of is if the IG's "id" has changed and thus the filename has changed. If that happens, perhaps Sushi could read the existing file and then re-write the file using the new IG id and leave everything else the same (basically just the template declaration and any commented content)

Chris Moesel (May 05 2020 at 15:40):

I don't understand why the simplest solution would not work - sushi sees an ig.ini, leaves it alone.

I think my main concern is that there are still multiple sources of truth. If the ig-data/ig.ini (or config.yaml in the future) contains a template value and the destination path ig.ini contains a template value -- what do we do when those values disagree? Which is the "truer" value -- the SUSHI source value or the destination path ig.ini value? So I think there are a few potential solutions to choose from:

• If SUSHI config/input value conflicts w/ destination ig.ini value, log warning and prefer destination ig.ini.
• If SUSHI config/input value conflicts w/ destination ig.ini value, log warning and prefer SUSHI generated ig.ini.
• If SUSHI config/input value conflicts w/ destination ig.ini value, log error and require author to deconflict values.
• Don't allow ig-data/ig.ini (or template property in config.yaml) anymore -- reduce single source of truth to ./ig.ini (meaning SUSHI now requires two configs: ig.ini and config.yaml).
• Still allow ig-data/ig.ini (or template property in config.yaml), but make it optional. If it's specified, always create or overwrite ig.ini. If it's not specified, then (a) if ig.ini already exists, leave it alone, or (b) if ig.ini doesn't exist, generate a default one.

After enumerating all those options, I'm guessing that one of the last two is closest to what you're suggesting. Is that right?

Lloyd McKenzie (May 05 2020 at 15:43):

I understand sushi as a short way of defining resources. I'm not sure why it's trying to define an alternate way of defining build processes (which are already defined pretty simply) I don't think our objective is a second way of saying everything?

Jean Duteau (May 05 2020 at 15:44):

Many of us (but not Jose?) are using FSH and SUSHI to handle creating everything that the IGPublisher needs. Not just as a means of defining profiles, extensions, etc., but also the ImplementationGuide-xxx.json, the menus, the web pages, etc.

Lloyd McKenzie (May 05 2020 at 15:47):

Hmm. That seems to be scope creep beyond what the project was originally supposed to do.

Lloyd McKenzie (May 05 2020 at 15:49):

The format for menus is likely to change. And I'm not sure how SUSHI can simplify pages beyond markdown?

Lloyd McKenzie (May 05 2020 at 15:49):

What's the driver for creating additional formats for those other things?

Chris Moesel (May 05 2020 at 15:54):

What's the rationale for Sushi re-writing the ig.ini?

So... TBH, the current state is not a fully intended design with an explicit rationale. The original design was that SUSHI had a dedicated source/input folder and was always writing into a dedicated output folder (i.e., not a folder with user-managed content). In this intended design, the only things that could be overwritten were generated files -- so there was no issue. It's the new IG Publisher integration and use cases where people are generating SUSHI output into existing folders with user-managed files that causes some of these issues to pop up. And we recognize that this requires us to rethink how that all works.

In a perfect world there would be a clean line between source files and output files (ideally in separate folder hierarchies -- like src and build in many software systems). And this is what we tried to support via a source fsh folder that can contain an ig-data subfolder for holding all the IG files -- that ultimately get processed/copied to the destination. But if people don't want to store their ig config in ig-data, and instead want to store it in the destination folder directly, we leave the world of clean separation and get into a world of gold source input files and generated output files all being mixed together. That's where the headaches are. It would be avoided if fsh/ig-data were used, but I guess there are use cases where that approach is not ideal (although I will admit I don't fully understand them yet)...

Chris Moesel (May 05 2020 at 15:58):

Hmm. That seems to be scope creep beyond what the project was originally supposed to do.

So... just to be sure we're all on the same page. FHIR Shorthand (the spec) is focused on generating definitional resources, examples, etc. All of the stuff around IGs and IG config is implementation details of SUSHI -- and outside of the FSH specification. It's not normative; it's just a feature of our SUSHI tool -- which we built based on feedback from users who wanted an easier way to get from FSH files to an IG they can view in a browser (because viewing JSON profiles directly is a little tedious).

John Moehrke (May 05 2020 at 16:02):

agile -- great approach, painful at times.

Jean Duteau (May 05 2020 at 16:07):

I agree with @Chris Moesel 's comments. It seems quite clear to me that SUSHI was built with the assumption that you gave it a set of source files and it generated a set of output files that you then processed with the IGPublisher. In that world, you could migrate exactly what you wanted and leave behind what you didn't want. It seems to me that Grahame surprised the team by adding hooks to SUSHI in the IGPublisher. That plus users wanting to use FSH and the integration but not wanting all of the integration are causing growing pains for the SUSHI team. I have opinions on what I want the SUSHI team to focus on, but I'll leave it to them to solve all of the world's FHIR IG problems. :)

Jose Costa Teixeira (May 05 2020 at 16:07):

Jean Duteau said:

Many of us (but not Jose?) are using FSH and SUSHI to handle creating everything that the IGPublisher needs. Not just as a means of defining profiles, extensions, etc., but also the ImplementationGuide-xxx.json, the menus, the web pages, etc.

I use Sushi for FSH-FHIR. It really saves me a lot of time. But not for everything - there are a few things I cannot do with FSH yet, and my suggestion would be that Sushi focuses on those, not the publication features.

Jose Costa Teixeira (May 05 2020 at 16:13):

I launched the topic not to say sushi is no sufficient - my questions are only about the integration.
One hard(ish) requirement from my side is that I want to keep my Continuous Delivery - if I change something in a FHIR resource or a FSH file, the build should pick up.

Jose Costa Teixeira (May 05 2020 at 16:14):

Chris Moesel said:

Hmm. That seems to be scope creep beyond what the project was originally supposed to do.

So... just to be sure we're all on the same page. FHIR Shorthand (the spec) is focused on generating definitional resources, examples, etc. All of the stuff around IGs and IG config is implementation details of SUSHI -- and outside of the FSH specification. It's not normative; it's just a feature of our SUSHI tool -- which we built based on feedback from users who wanted an easier way to get from FSH files to an IG they can view in a browser (because viewing JSON profiles directly is a little tedious).

And I think feature that is a good thing, but should not impede integration or advanced uses, right?

John Moehrke (May 05 2020 at 16:14):

I vote for sushi team to focus on more simplifying authoriing of defining resources (e.g. OperationDefinition, MessageDefinition, ActivityDefinition, PlanDefinition, then some of the other *Definition.

Jean Duteau (May 05 2020 at 16:16):

it seems to me that your problems would be solved by not using the IGPublisher integration. Have SUSHI take your FSH input files and generate them into a SUSHI output directory and then copy over the FHIR definition stuff (profiles, extensions, terminology) into your IG directories. That would give you exactly what you want.

Jose Costa Teixeira (May 05 2020 at 16:17):

John Moehrke said:

I vote for sushi team to focus on more simplifying authoriing of defining resources (e.g. OperationDefinition, MessageDefinition, ActivityDefinition, PlanDefinition, then some of the other *Definition.

And Logical Models and ExampleScenarios.. :+1:

Jose Costa Teixeira (May 05 2020 at 16:20):

Jean Duteau said:

it seems to me that your problems would be solved by not using the IGPublisher integration.

whose problems? Mine (and the teams that I hope are working in a consistent way?) Or you mean John's?

Jean Duteau (May 05 2020 at 16:21):

yours

Jose Costa Teixeira (May 05 2020 at 16:23):

I can give up on the integration until sushi fixes the issues pointed by chris - the input and output files. I understand the issue, so I just suggested one simple change for now to unblock this - if there is an ini file, leave it alone.

Jean Duteau (May 05 2020 at 16:23):

Jean Duteau said:

it seems to me that your problems would be solved by not using the IGPublisher integration. Have SUSHI take your FSH input files and generate them into a SUSHI output directory and then copy over the FHIR definition stuff (profiles, extensions, terminology) into your IG directories. That would give you exactly what you want.

An alternative would be for SUSHI to have some flags that govern exactly what gets created:
a) just the outputs from the FSH files
b) (a) plus IG web content
c) the whole kit and kaboodle (what is currently done)

edit: (which of course Chris already suggested above)

Jose Costa Teixeira (May 05 2020 at 16:23):

otherwise I cannot have sushi as part of the continuous delivery

Jose Costa Teixeira (May 05 2020 at 16:25):

I think this thread itself is a scope creep - they grow so fast, i know... my suggestion seems compatible with the use of such flag.

Jose Costa Teixeira (May 05 2020 at 16:26):

I think Chris will get it right.

Jean Duteau (May 05 2020 at 16:27):

Jose Costa Teixeira said:

I can give up on the integration until sushi fixes the issues pointed by chris - the input and output files. I understand the issue, so I just suggested one simple change for now to unblock this - if there is an ini file, leave it alone.

But that is in direct opposition to my requirement - overwrite any sushi-generated file as a result of the changes I've made in my FSH folder. I feel for Chris and team as he has to balance these two competing requirements.

Lloyd McKenzie (May 05 2020 at 16:28):

It seems like we should talk about what the actual file structure should be for an integrated environment where input folders are clearly defined and never written to by tools that aren't authoring those input files, where temporary/intermediate files are always generated into the 'temp' folder (where they get wiped every time there's a build) and output is reserved solely for the final published output.

Jose Costa Teixeira (May 05 2020 at 16:30):

Jean Duteau said:

But that is in direct opposition to my requirement - overwrite any sushi-generated file as a result of the changes I've made in my FSH folder. I feel for Chris and team as he has to balance these two competing requirements.

if "sushi-generated files" includes control files like the ig.ini, which is an input file, then yes, there is a problem derived from current implementation.

Lloyd McKenzie (May 05 2020 at 16:30):

I think the 'fsh' files should be able to live in the input folder - ideally alongside .xml and .json files for those resources that the author chooses not to (or can't) maintain appropriately using FSH. The scan process in the IGPublisher should automatically find all relevant files (XML, JSON, spreadsheet or .fsh) and perform whatever processing is needed to convert them into proper resource instances in memory.

Lloyd McKenzie (May 05 2020 at 16:31):

If SUSHI needs control files, we should differentiate between control files that are SUSHI-specific and those that aren't. Ideally, SUSHI would read ig.ini, not write to it?

Jean Duteau (May 05 2020 at 16:31):

Lloyd McKenzie said:

It seems like we should talk about what the actual file structure should be for an integrated environment where input folders are clearly defined and never written to by tools that aren't authoring those input files, where temporary/intermediate files are always generated into the 'temp' folder (where they get wiped every time there's a build) and output is reserved solely for the final published output.

I think a main problem is that SUSHI's design was as Chris stated above:
FSH input folders --> SUSHI --> FSH output folders --> user copied over --> IG Publisher input folders --> IGPublisher --> IG Publisher output folders

When Grahame added in SUSHI integration to the IGPublisher tool, suddenly the FSH output folders became meshed into the IGPublisher input folders and that has caused a lot of the pains that Jose and others are experiencing. If your environment is like mine where I'm willing to let everything in the FSH output be the IGPublisher input, then you are fine. But if you are doing esoteric things and can't work with that, then we get into trouble.

Lloyd McKenzie (May 05 2020 at 16:32):

I think there's good reasons to want the publication process to work without any intermediate steps

Lloyd McKenzie (May 05 2020 at 16:33):

We can create an 'output' folder for SUSHI in the temp folder that can be used as an input folder for the IG-Publisher - no moving stuff around. But it would mean that SUSHI should never be writing ig.ini (or certainly never overwriting it)

Lloyd McKenzie (May 05 2020 at 16:34):

Given that ig.ini has to be there for the CI-build, my leaning would be that SUSHI should be using it as an input.

Jean Duteau (May 05 2020 at 16:34):

Lloyd McKenzie said:

We can create an 'output' folder for SUSHI in the temp folder that can be used as an input folder for the IG-Publisher - no moving stuff around. But it would mean that SUSHI should never be writing ig.ini (or certainly never overwriting it)

What if I want it to? I actually like the fact that SUSHI produces everything I need to generate a proper IG. I now have three projects where my only directory checked into source control is the FSH directory. Everything else is generated by SUSHI and everything just works.

Chris Moesel (May 05 2020 at 16:36):

NOTE: It looks like there are at least half-a-dozen new messages since I started writing this message, so this is in the context of previous conversation... I haven't read the new messages yet!

@Lloyd McKenzie -- FSH doesn't redefine markdown or page content. You write your pages the same way. The difference, in regard to pages, is that SUSHI will read your folder of page files and automatically generate all the page config content in ImplementationGuide.

As for menu, SUSHI currently will create a very standard boilerplate menu if the author doesn't provide one. Otherwise it copies over the author's menu.xml directly. So we haven't invented anything new there (yet) -- but the new config.yaml will provide a simplified structure for defining a menu (e.g., simple name/url pairs) for those who don't want to mess with xhtml.

It will also generate a starter package-list.json file for you if you don't have one. The new config, however, will allow history to be specified there.

SUSHI also uses all the data from generated resources and examples to populate the resources in the ImplementationGuide. And copies over redundant data from package.json into ImplementationGuide. Now that package.json isn't really required anymore, maybe that's less necessary.

So, as it is now, SUSHI doesn't really redefine any formats -- it just generates as much as it can generate (reliably) to lower the barrier of entry for getting to an IG. The proposed new configuration does admittedly redefine some things, but that is mainly in the interest of allowing more control over those generated files and consolidating config to a single file rather than spread across multiple files. We may find that this is a bad idea or that improvements to the IG process in general render this unnecessary or overkill. If so, great!

We'll be releasing the new config approach as beta before the Connectathon and use the Connectathon to test it out. We expect to learn a lot from that process -- and will try to react based on what we learn. As some of you have noted, it's an agile approach, which can be painful at times, and there are certainly growing pains. We also have a diverse set of users with diverse levels of experience (w/ IGs) and varying expectations regarding what SUSHI should provide. And sometimes those expectations are seemingly in conflict with each other. So... we're doing our best to navigate all of this! And, for now, we're trying to focus on what will provide the greatest value to the majority of users -- but we don't want to do it at the expense of users on a narrower path; so the feedback is good!

Lloyd McKenzie (May 05 2020 at 16:36):

Because it the CI build will blow up if that's all you check in? Creating an ig.ini manually isn't an unreasonable ask for consistency across the whole publication process I think?

Lloyd McKenzie (May 05 2020 at 16:37):

How does FSH determine the names of pages or the hierarchy of pages when it sucks them into the IG?

Jean Duteau (May 05 2020 at 16:37):

Lloyd McKenzie said:

Because it the CI build will blow up if that's all you check in? Creating an ig.ini manually isn't an unreasonable ask for consistency across the whole publication process I think?

No it doesn't. That's all I have checked in for the med-list-guidance project and CI works just fine.

Lloyd McKenzie (May 05 2020 at 16:38):

There's a desire to simplify the menu function for everyone - so I'd rather this wasn't something that was FSH-specific

Jean Duteau (May 05 2020 at 16:43):

Chris Moesel said:

...SUSHI will read your folder of page files and automatically generate all the page config content in ImplementationGuide.

...SUSHI also uses all the data from generated resources and examples to populate the resources in the ImplementationGuide.

...So, as it is now, SUSHI doesn't really redefine any formats -- it just generates as much as it can generate (reliably) to lower the barrier of entry for getting to an IG.

The above is why I LOVE SUSHI. As I've said above, I now have one source directory for my FHIR IGuides and all of the other configuration that was tedious to manage is managed by SUSHI. I can now create a new FSH file, put in a profile definition, some extensions, some terminology, and some examples, and SUSHI figures it all out, creates the appropriate json input files, and updates the ImplementationGuide appropriately. Similarly, I can add a new web page and boom, it all just works. This simplicity makes it easier for so many people to create FHIR IGuides without needing to know the esoterics.

Lloyd McKenzie (May 05 2020 at 16:44):

@Chris Moesel Do you think an endpoint where FSH files are maintained in the standard input folders alongside other resources is achievable?

Lloyd McKenzie (May 05 2020 at 16:44):

How does SUSHI know the page name or position in the hierarchy?

Chris Moesel (May 05 2020 at 16:45):

Sorry... in a meeting right now, so can't comment in much detail just yet... But will do so soon...

Jean Duteau (May 05 2020 at 16:45):

Lloyd McKenzie said:

How does SUSHI know the page name or position in the hierarchy?

From the SUSHI Reference Page:
N_pagename.xml|md: If present, these files will be generated as individual pages in the IG. The leading integer (N) determines the order of the pages in the table of contents. These numbers are stripped and do not appear in the actual page URLs.

Chris Moesel (May 05 2020 at 16:46):

Right -- and we also use the file name to try to smartly generate a title. E.g., ImplementationDetailsForClients.md will result in the title "Implementation Details for Clients".

Chris Moesel (May 05 2020 at 16:47):

The numbered prefixes work for top-level. Admittedly, it doesn't address full page hierarchies. The new config.yaml addresses that better for those who need it.

Lloyd McKenzie (May 05 2020 at 16:49):

n gives you a sequence, but not a hierarchy

Lloyd McKenzie (May 05 2020 at 16:49):

Though I guess we could do an n.m.o approach that would

Lloyd McKenzie (May 05 2020 at 16:51):

I'm not thrilled with the notion of page names -> titles. Typically you want the page names to be short and sweet - because that makes short/sweet hyperlinks. Titles, on the other hand, need to be descriptive and sometimes include special characters. I'd much rather impose more work on authors and retain those characteristics than make things easier for authors but have URLs that suck for implementers.

Jean Duteau (May 05 2020 at 16:53):

Lloyd McKenzie said:

I'm not thrilled with the notion of page names -> titles. Typically you want the page names to be short and sweet - because that makes short/sweet hyperlinks. Titles, on the other hand, need to be descriptive and sometimes include special characters. I'd much rather impose more work on authors and retain those characteristics than make things easier for authors but have URLs that suck for implementers.

Why would the URL matter for implementers? When are you going to directly reference a specific page in an IGuide? (and not do that by copying the URL in the first place?)

Lloyd McKenzie (May 05 2020 at 16:58):

I type URLs all the time for IGs I'm familiar with - and the core spec. Also, they are maintained as links in other specs (and you ideally don't want them to change, even if you change the descriptive label for clarification) - thus http://hl7.org/fhir/http.html still having the same name even though a better label might be REST.

Chris Moesel (May 05 2020 at 17:00):

The auto-page-detection approach is a convenience for those who want to use it. Part of the reason we're working on a new config is to also support those users who want more control or for whom the convenience is not desired. I like having a convention that makes simple use cases drop-dead simple while also supporting more complex features where they are needed. I don't think we should force simple use cases into more complex solutions.

Lloyd McKenzie (May 05 2020 at 17:02):

I'm opposed to not forcing authors to define both a page link name (for the URL) and a descriptive name. But I'm fine with the process for that being as simple as possible

Jean Duteau (May 05 2020 at 17:04):

Lloyd McKenzie said:

I'm opposed to not forcing authors to define both a page link name (for the URL) and a descriptive name. But I'm fine with the process for that being as simple as possible

I'm getting to be a broken record, but +1 for what SUSHI currently does. My page names are simple and work just fine as the URL, so I want SUSHI to do the heavy lifting for me.

Chris Moesel (May 05 2020 at 17:07):

@Lloyd McKenzie said:

I'm opposed to not forcing authors to define both a page link name (for the URL) and a descriptive name. But I'm fine with the process for that being as simple as possible

My main concern there is that it can be pretty boilerplate in many situations and it imposes a maintenance burden. If you add a new page but forget to put it into ImplementatonGuide, then it doesn't get rendered. If you remove a page and forget to remove it from ImplementationGuide, it causes issues. If you rename a page and forget to edit the ImplementationGuide, it causes issues. If all the necessary info can be intuited by data that already exists, I prefer to use the data that already exists rather than requiring redundant specification of it.

But I think we just need to agree to disagree there. ;-)

Lloyd McKenzie (May 05 2020 at 17:07):

I'm fine with SUSHI doing the heavy lifting. I'm not ok with things being simplified in a way that impacts what the output is in a negative way - and a page named ImplementationDetailsForClients.html is, to me, a significantly negative thing.

Jean Duteau (May 05 2020 at 17:08):

Lloyd McKenzie said:

I'm fine with SUSHI doing the heavy lifting. I'm not ok with things being simplified in a way that impacts what the output is in a negative way - and a page named ImplementationDetailsForClients.html is, to me, a significantly negative thing.

why? You'd rather have a page named 'http.html' that is actually all about RESTful interfaces? Are you going to make this a quality check - your IG page URLs can't be more than X characters? There is nothing stopping me from creating that page with that filename now.

Chris Moesel (May 05 2020 at 17:09):

a page named ImplementationDetailsForClients.html is, to me, a significantly negative thing.

Would you prefer implementation-details-for-clients.html? That's supported too. ;-)

Lloyd McKenzie (May 05 2020 at 17:09):

I'd prefer clients.html. And if I negative balloted that the URL was too long, finding it non-persuasive because "SUSHI doesn't support that" wouldn't cut it.

Jean Duteau (May 05 2020 at 17:10):

Lloyd McKenzie said:

I'd prefer clients.html. And if I negative balloted that the URL was too long, finding it non-persuasive because "SUSHI doesn't support that" wouldn't cut it.

I'd non-persuasive it because "the URL is too long" is not a valid negative. Stop manually typing the URLs and you'll be fine.

Chris Moesel (May 05 2020 at 17:11):

Lloyd McKenzie said:

Do you think an endpoint where FSH files are maintained in the standard input folders alongside other resources is achievable?

Yes, I think that's definitely something we should explore! And your idea about generating into a temp folder that the IG Publisher reads as input makes sense too. I think something like that starts to solve some of these problems (although, admittedly, not all of them).

Lloyd McKenzie (May 05 2020 at 17:11):

It's a legitimate issue. URLs matter. If they didn't, we'd make them all GUIDs.

Jean Duteau (May 05 2020 at 17:15):

Lloyd McKenzie said:

It's a legitimate issue. URLs matter. If they didn't, we'd make them all GUIDs.

I was going to write something long, but I've decided to go with "we'll have to agree to disagree". :)

John Moehrke (May 05 2020 at 17:16):

OIDS rule

Chris Moesel (May 05 2020 at 17:24):

One thing we can consider is that using the new config approach (again, soon to be released in beta), we could support flags like the following:

• generate-ig-ini
• generate-package-json
• generate-package-list-json
• generate-menu-xml
• generate-implementation-guide-resource

then the user can decide what files they do and don't want generated.

Jean Duteau (May 05 2020 at 17:26):

Chris Moesel said:

One thing we can consider is that using the new config approach (again, soon to be released in beta), we could support flags like the following:

• generate-ig-ini
• generate-package-json
• generate-package-list-json
• generate-menu-xml
• generate-implementation-guide-resource

then the user can decide what files they do and don't want generated.

Yes, I think we've seen that is probably needed and useful. The default could be to generate everything and then advanced users can turn off things that they didn't want generated. I think that solves Jose's needs.

Lloyd McKenzie (May 05 2020 at 17:32):

One of the base premises on which FSH was allowed to proceed was that there would always be an 'escape' mechanism to allow you to express the full complexity of a resource, even if the base/simple syntax didn't. I would expect that to hold true for page URLs vs. titles as for any other ImplementationGuide element

Lloyd McKenzie (May 05 2020 at 17:33):

I.e. If I wanted to express the SDC IG in FSH (which has distinct URL and title names for pages, that needs to be possible.

Chris Moesel (May 05 2020 at 17:35):

So, again, we're not talking about FSH proper now. We're talking about the SUSHI FSH compiler. That's not spec -- it's an implementation. But I think we agree that you should be able to control the page title and URL distinctly if you want. The beta release next week will support that.

Jean Duteau (May 05 2020 at 17:37):

Lloyd McKenzie said:

One of the base premises on which FSH was allowed to proceed was that there would always be an 'escape' mechanism to allow you to express the full complexity of a resource, even if the base/simple syntax didn't. I would expect that to hold true for page URLs vs. titles as for any other ImplementationGuide element

FSH doesn't know anything about page URLs or stuff like that. You are now talking about the SUSHI compiler which is different.

Lloyd McKenzie (May 05 2020 at 17:44):

I think part of the issue is that there's an unclear division of responsibility between SUSHI and the IGPublisher. The IGPublisher already loads resources from the input folders and automatically populates them into the list of IG artifacts (regardless of format authored in). It also adds pages for each of those artifacts into the IG for generation of the table of contents. Therefore, that's not something SUSHI should be doing - because that then gives two sets of code doing the same functionality, but possibly differently and also creates duplication of effort. If we're going to have a convention by which the same sort of thing can happen with pages, I would again prefer that that be handled via the publisher as it's not something that's FSH-specific.

Jose Costa Teixeira (May 05 2020 at 17:57):

I think there is an unclear division between

• sushi as FSH parser,
  and

• publishing:
  ** sushi as a publisher
  ** the IG Publisher

Jose Costa Teixeira (May 05 2020 at 17:59):

I hope that tools remain interoperable.

Jean Duteau (May 05 2020 at 18:05):

Lloyd McKenzie said:

I think part of the issue is that there's an unclear division of responsibility between SUSHI and the IGPublisher. The IGPublisher already loads resources from the input folders and automatically populates them into the list of IG artifacts (regardless of format authored in). It also adds pages for each of those artifacts into the IG for generation of the table of contents. Therefore, that's not something SUSHI should be doing - because that then gives two sets of code doing the same functionality, but possibly differently and also creates duplication of effort. If we're going to have a convention by which the same sort of thing can happen with pages, I would again prefer that that be handled via the publisher as it's not something that's FSH-specific.

You need to go back to before SUSHI was integrated with the IGPublisher to see how it was intended to work:

1) I have a FSH source folder where I create all of my IG source artifacts - profiles, extensions, terminology, examples, page content, etc.
2) SUSHI takes that source folder and auto-magically figures out how what and how to turn that into an input directory that the IGPublisher can then take and run with.

To fulfill that use case, what it does now is very good. It generates the ImplementationGuide.json, the ig.ini, the <insert your IGPublisher source file of choice here> and keeps them all in sync based on what I have in my FSH source folder. As long as I'm willing to live with the constraints that SUSHI places on me, everything works just fine. And I would argue that there is definitely no unclear division of responsibility. SUSHI is a pre-processor that creates an input directory that can be used by the IG Publisher to generate a FHIR IGuide.

I suspect the confusion comes because you want to treat the FSH files as input files to IGPublisher like any other input file when they weren't intended to be that - they were intended to be generated into input files.

Jean Duteau (May 05 2020 at 18:07):

Lloyd McKenzie said:

The IGPublisher already loads resources from the input folders and automatically populates them into the list of IG artifacts (regardless of format authored in). It also adds pages for each of those artifacts into the IG for generation of the table of contents.

Is this true? Because it has definitely not been my experience. I have spent countless hours pre-FSH/SUSHI adding in my list of extensions by hand into the ImplementationGuide.

Lloyd McKenzie (May 05 2020 at 18:11):

It's been true for 3-4 months now - one of the features of the IG templates. The only place where you must mention your resources in the IG is for examples, and that's because the IG pubisher is the only place you can give them a title and a description. (For the other resources, we grab that from the resources themselves.) Other than that, the publisher finds them all, groups them into appropriate package based on type and within each package sorts them alphabetically by title.

Lloyd McKenzie (May 05 2020 at 18:14):

I'm totally fine with SUSHI being used as a pre-processor that converts FSH syntax into non-FSH syntax. To a certain extent, the spreadsheet parser works the same way. What I'm less comfortable with is having tools write to the 'input' folder. I think it makes sense for all input (regardless of syntax) to live in one place and the pre-processing steps that happen to be invoked automatically by the IG-publisher. When we're looking at enhancements that aren't FSH-specific, it would be best if those were in the publisher rather than SUSHI so that IGs that aren't using FSH can still take advantage.

Jean Duteau (May 05 2020 at 18:21):

Lloyd McKenzie said:

I'm totally fine with SUSHI being used as a pre-processor that converts FSH syntax into non-FSH syntax. To a certain extent, the spreadsheet parser works the same way. What I'm less comfortable with is having tools write to the 'input' folder. I think it makes sense for all input (regardless of syntax) to live in one place and the pre-processing steps that happen to be invoked automatically by the IG-publisher. When we're looking at enhancements that aren't FSH-specific, it would be best if those were in the publisher rather than SUSHI so that IGs that aren't using FSH can still take advantage.

SUSHI isn't writing into the 'input folder. The only reason that SUSHI currently has its output go in the 'input' folder is because Grahame configured SUSHI to do that in his integration. There are a multitude of ways that it could have been done that wouldn't have the trouble that we currently have. Think about the pre-integration steps:
1) Create FSH files in FSH input folder
2) Run SUSHI on FSH input folder, creating SUSHI output folder
3) Copy/Merge SUSHI output folder into IG Publisher input folder
4) Run IGPublisher

Right now, in the integrated world, #3 is happening like a blunt tool - we are telling SUSHI to overwrite everything into the IG input directory with its generated files. Your earlier suggestion of having SUSHI generate into a temp directory and then having a publisher "merge" step that copied over the appropriate files from that temp directory into the IG input directory would probably have been safer. All of this configuration that Chris is suggesting for SUSHI would then be configuration for the IGPublisher to know what to merge and what not to.

Lloyd McKenzie (May 05 2020 at 18:25):

I'm not totally clear on the notion of 'merging'. My expectation is that a resource (IG, profile, whatever) will either exist as a .FSH file or it won't. If there are FSH files, SUSHI will be invoked to produce the relevant artifacts in the temp folder. The IG publisher will then suck all artifacts - whether in the temp folder or in the input folders and process normally.

Jean Duteau (May 05 2020 at 18:26):

your expectation is incorrect. SUSHI does more than that because its users asked the team to have it do more than that.

Jean Duteau (May 05 2020 at 18:29):

Go to a look at the fhir-medication-list-guidance source in Github to see what I have as my input that produces a fully fleshed out IG

Jose Costa Teixeira (May 05 2020 at 18:30):

perhaps it's my impression but this isn't getting any simpler.
Sushi does write onto the "input" folder, because that is the folder where the ig publisher picks up things.

Jose Costa Teixeira (May 05 2020 at 18:31):

The merge option is not really needed, I think - we could add a "sushi-generated-resources" in the ig template if that were needed, the IG Publisher would do the rest

Jean Duteau (May 05 2020 at 18:31):

Jose Costa Teixeira said:

perhaps it's my impression but this isn't getting any simpler.
Sushi does write onto the "input" folder, because that is the folder where the ig publisher picks up things.

No, Sushi writes into its output folder. That output folder - in the auto-integration world - is also the IG Publisher input folder but it didn't need to be.

Jose Costa Teixeira (May 05 2020 at 18:32):

that was a choice sushi made when using the publisher framework

Jean Duteau (May 05 2020 at 18:32):

Jose Costa Teixeira said:

that was a choice sushi made when using the publisher framework

No, that was a choice that Grahame made when using the SUSHI framework.

Jose Costa Teixeira (May 05 2020 at 18:37):

IIRC, sushi did just dump everything into input/resources folder

John Moehrke (May 05 2020 at 18:41):

If the sushi conversion was more integrated into the IG build, then it could be used only to convert FSH -->JSON.

Jean Duteau (May 05 2020 at 18:42):

Jose Costa Teixeira said:

IIRC, sushi did just dump everything into input/resources folder

Well, it output things into a build directory that had an input directory inside of it. But it was up to you on how you integrated that into the IG-Publisher. You could have grabbed it all (which is what effectively the IGPublisher is now doing) or you could have just grabbed the 'examples', 'profiles', 'vocabulary', and 'extensions' directories, or whatever you wanted. The way that integration is currently set up, you get it all.

John Moehrke (May 05 2020 at 18:42):

I was also uncomfortable with the additional ig-data things that sushi does... but it does them today, so I use them today.

Jose Costa Teixeira (May 05 2020 at 18:43):

John Moehrke said:

If the sushi conversion was more integrated into the IG build, then it could be used only to convert FSH -->JSON.

that is what we started with - sticking to that, ideally without overwriting configuration content.

Jean Duteau (May 05 2020 at 18:44):

John Moehrke said:

I was also uncomfortable with the additional ig-data things that sushi does... but it does them today, so I use them today.

When I first started using it, I was like you. But then I saw how it simplified the rest of my work so I bought into it.

John Moehrke (May 05 2020 at 18:44):

it did simplify... but as Jose is noting, eventually it gets in the way.

John Moehrke (May 05 2020 at 18:45):

so, for some people the current sushi and how it is automated by the ig builder... works Great.

John Moehrke (May 05 2020 at 18:45):

for some it doesn't

Jean Duteau (May 05 2020 at 18:45):

Sure. but my argument is that the integration with the publisher is causing it to get in the way and not an inherent quality of SUSHI itself

John Moehrke (May 05 2020 at 18:46):

careful wording... the sushi tool is causing these issues... not shorthand (FSH) itself

Jose Costa Teixeira (May 05 2020 at 18:47):

sushi as a fsh parser is great - the greatest fsh parser.

Jose Costa Teixeira (May 05 2020 at 18:48):

sushi as an IG editor may be sufficient for some. for me it isn't

Lloyd McKenzie (May 05 2020 at 18:53):

I understand that SUSHI is doing more than I've described. I'm saying it shouldn't be - it should be focused on converting FSH files. The publisher should be focused on doing the publishing functions. Having two different tools doing publishing functions isn't a good idea. I understand that the publishing functions SUSHI is performing are desired. I also understand that Chris may have been able to move faster at incorporating new functions into his tool than those maintaining the publisher tool have been able to update the IGPublisher. What I would like to see is that functionality moved from SUSHI into the publisher

Jean Duteau (May 05 2020 at 18:54):

BTW, I'm assuming that everyone already knows that SUSHI doesn't automatically create all of those files. Here is the note in the SUSHI User Guide:
Note: If no IG is desired, and you only want to export the FHIR artifacts (e.g., profiles, extensions, etc.), ensure that no ig-data folder is present.

Jean Duteau (May 05 2020 at 18:55):

Lloyd McKenzie said:

Having two different tools doing publishing functions isn't a good idea.

SUSHI isn't doing "publishing" functions. It is generating content that can be used as input into the IG Publisher toolchain.

Jose Costa Teixeira (May 05 2020 at 19:00):

I think the ig.ini discussion is done. The rest of the substance - menus, config files.. I think should be done in the IG framework.

Lloyd McKenzie (May 05 2020 at 19:01):

If it's going through the list of files available and supplementing that information into the IG, that has nothing to do with FSH - and is a publishing function

Jean Duteau (May 05 2020 at 19:02):

Lloyd McKenzie said:

If it's going through the list of files available and supplementing that information into the IG, that has nothing to do with FSH - and is a publishing function

I for one am glad that SUSHI is more than just a FSH compiler. I'm hopeful that there are other users that agree with me.

Jose Costa Teixeira (May 05 2020 at 19:08):

I'd be more glad if sushi did things that no other tool does - parsing logical models, etc. I really don't mind editing the ig for page structure if that is the price to keep things flexible and in a way that I can industrialize the roll-out of several IGs.

Bob Milius (May 05 2020 at 19:09):

I don't have a lot to add, except I've been using FSH/Sushi for learning about IG creation and have been using it to create a couple of local IGs. Typically, I run sushi, and then the igpublisher separately. I've learned (and am still learning) a lot about structure definitions and IG publishing. I do appreciate it's ability for creating a simple IG.

David Hay (May 05 2020 at 19:20):

I'm going to be very interested to try out the configuration options myself - hopefully over connectathon!. Like Bob & Jean I've found it great for generating simple IGs and hope that the issues for the more advanced users can be resolved with options to turn off features that more advanced users don't want...

Lloyd McKenzie (May 05 2020 at 19:22):

I'm not arguing against the features SUSHI has, I'm just arguing about where they ought to be. We don't want two tools doing the same thing when both of them must be used together to get to a published spec.

Nick Freiter (May 05 2020 at 19:29):

When an ig-data folder is specified, SUSHI automatically generates an ImplementationGuide resource based off of the resources defined using FSH, and the configuration information given as input to SUSHI. @Lloyd McKenzie would this be something that you would argue SUSHI should not do? As this is somewhat outside the scope of strictly FSH -> JSON.

Lloyd McKenzie (May 05 2020 at 19:33):

I would presume there is an IG defined using FSH that specifies stuff like the id, title, description, realm and all of the other essential metadata. I'd expect SUSHI to turn that IG FHS file into a proper IG. Populating it with the list of artifacts (and I guess pages if we want to support that) is not something I'd expect SUSHI to do as it has nothing to do with FSH. That should be a general capability of the IGPublisher

Jean Duteau (May 05 2020 at 19:40):

so you want the IGPublisher to be able to auto-generate an ImplementationGuide.json?

Lloyd McKenzie (May 05 2020 at 19:42):

FSH will produce the ImplementationGuide.json (with no listed artifacts and perhaps no listed pages). The IGPublisher will use the template to figure out what folders to check and find the relevant files and produce an updated ImplementationGuide.json containing the artifacts, pages, etc. complete with descriptions and groupings which it then uses for subsequent processing. It already does all of that except for the pages bit.

Bob Milius (May 05 2020 at 19:49):

@Lloyd McKenzie Would parameters (e.g., special-url) be populated by the IGPublisher? or by Sushi? or by hand?

Grahame Grieve (May 05 2020 at 20:00):

ok, several comments on this thread:

• I'm mystified why you would generate a package.json file since that's not input to the IG publisher

• You should not try generating a package-list.json file, since that is edited only once by editor when submitting, and changes to previous entries are not allowed. Several authors have tried to rewrite history this way. In fact, I'm planning to take package-list.json away from the IG authors altogether

• The problem driving this discussion is the conflation of two functions - processing FSH files, and preprocessing an IG. It's a guaranteed outcome of pre-processing an IG that you'll hold yourself to the minimum function being everything people want. I think you shouldn't force yourself to do that, and you'll end up with all the same complexity anyway. So don't do that - let people process just the fsh files, and do the rest themselves, and let people who want the easier and more limited pre-processing thing - it'll be better for them too

Jean Duteau (May 05 2020 at 20:14):

I just created an IGuide from scratch and did not specify a ig-data directory in my fsh directory. I needed to provide an ig.ini and an input/ImplementationGuide-xxx.json file, and some pagecontent. The only things that SUSHI created were the extensions, examples, profiles, and terminology artifacts. My ImplementationGuide only had the meta-data information (id, url, version, etc.), dependsOn, definition with only pages and parameter listed. And I got basically the same IGuide as i got when I was having SUSHI create a full IG resource.

Jean Duteau (May 05 2020 at 20:16):

so I think I might need to backtrack some of my more strident comments from earlier. :) It was my belief that the IGPublisher needed much more of a fleshed-out IG resource than it does.

Grahame Grieve (May 05 2020 at 20:54):

actually, that's never been the case; it's not a recent change. What's recent was actually documenting how to make it work like that

Lloyd McKenzie (May 05 2020 at 21:05):

Well that and the templates set up to take advantage of it automatically so you didn't have to put a bunch of parameters in place to make it work

Chris Moesel (May 05 2020 at 22:10):

I took a 2-hour break from Zulip. 45 new messages while I was gone, but good discussion. ;-) I think we'll have some fun ironing some of this out in the Connectathon! I'm all for clearer separation between resource generation and publishing. At the same time, I'm also all for lowering the bar for people to get from nothing to a working IG. SUSHI made some good early steps toward this goal, but the IG Publisher and template framework have also made some great steps toward that goal. I think we're mainly on the same page regarding where we want to go (minimizing the barrier to entry while also supporting complex use cases). Now we just need to figure out how best to get there.

Jean Duteau (May 05 2020 at 22:11):

Amen!

Chris Moesel (May 06 2020 at 13:14):

I'm not looking to revive this thread too much ( :grimacing: ), but just wanted to address a few loose ends (mostly questions/clarifications for @Grahame Grieve and/or @Lloyd McKenzie)...

If it's going through the list of files available and supplementing that information into the IG, that has nothing to do with FSH - and is a publishing function

I don't think this existed (or apparently may have existed but was not documented) when we first implemented this in SUSHI. But this is definitely a welcome feature. Is this the autoload-resources parameter in the IG? Do authors just set that to true to use it?

I'm mystified why you would generate a package.json file since that's not input to the IG publisher

We don't currently generate package.json in SUSHI; it's actually an input to SUSHI at the moment. That being said, it won't be an input anymore with the new configuration -- so we were actually wondering if there is any need to generate it. It sounds like maybe there is not? What about for FHIR packages that are not IGs (or is that just a theoretical concept at this point)?

You should not try generating a package-list.json file, since that is edited only once by editor when submitting, and changes to previous entries are not allowed.

With the new config, the contents of the package-list are managed by the author in the config -- so we're not generating the file out of thin air; we're just providing a more consolidated location for the author to manage its content. Now, I realize (after this thread) that this approach is still very controversial -- so I'm not suggesting this is the way it should be -- but I just wanted to clarify what it might be (at least temporarily) in the next beta.

In fact, I'm planning to take package-list.json away from the IG authors altogether

I saw some of your proposals regarding Provenance. Sounds good to me!

Lloyd McKenzie (May 06 2020 at 15:14):

Ideally, the template should handle the autoload-resource parameter. My leaning would be to not have SUSHI generate package.json and to use it as ain input instead. I'm not sure we have any notion of a FHIR package that isn't an IG - do you see a use-case?

Chris Moesel (May 06 2020 at 15:34):

I'm not sure we have any notion of a FHIR package that isn't an IG - do you see a use-case?

I think I just recall some conversations where people were very explicit that it is possible to have a package that is not an IG.

As for use cases, I'm not sure -- but I suppose that if a set of profiles is intended to be used only as a dependency of other IGs, then maybe it technically doesn't need all of the things that are typically in an IG (pages, etc); it really just needs the resource definitions themselves and that's it. Still, as someone _using_ that package as a dependency, I'd probably _want_ some user-facing documentation about the resource definitions, so... that still sounds like an IG to me.

I'm fine with every package being an IG. I just thought I remembered people saying it didn't have to be that way.

Nick Freiter (May 06 2020 at 15:42):

I think part of what we have been going off is the package documentation here https://confluence.hl7.org/display/FHIR/NPM+Package+Specification, which seems to indicate that there definitely is at least a notion of a FHIR package that isn't an IG. Not saying I know of any use cases necessarily.

---

Last updated: Apr 12 2022 at 19:14 UTC