Stream: shorthand
Topic: sushi - ig integration
Jose Costa Teixeira (Jul 27 2020 at 12:57):
I have my sushi stuff in a igroot/input/shorthand folder to avoid the fsh folder to trigger sushi automatically. I manually run sushi and then copy whatever I want from sushi output into my IG input. I'm ok with manual work at this moment.
Jose Costa Teixeira (Jul 27 2020 at 12:59):
I can try to figure out a decent folder configuration for having that integration less obtrusive (and less obtuse), but I think sushi can be more interoperable - everything works fine if we use one tool, but the real interoperability starts when the tools really need to share roles
Jose Costa Teixeira (Jul 27 2020 at 13:01):
(we're all experienced with systems that can easily export data in all formats, the funny interoperability issues start when the tools need to actually import data from outside) - that is for me the turning point to interoperability. Applying that to sushi:
Jose Costa Teixeira (Jul 27 2020 at 13:03):
- We need to find a good folder structure for a typical IG that uses sushi for part of the content.
- We need a way for sushi to NOT replace some of the artefacts - for example I have codesystems that I start in sushi, then I have to add language translations in my own tooling, then I don't want sushi to mess with them anymore
Jose Costa Teixeira (Jul 27 2020 at 13:06):
just for brainstorming:
- in YAML we can embed json. in sushi we could embed or point to json/xml...
- we could have an extension "youCanTouchThis" - which is sushi telling itself "ok to replace this artifact"
Jose Costa Teixeira (Jul 27 2020 at 20:22):
Since this is becoming such a popular topic, here's one more:
I think we should:
- more clearly separate the input and the output folders (it's all mixed now) - say a "ig-data-in" and "ig-data-out".
- split the creation of the IG infrastructure (like getting the batch files etc, which you really only do once per ig, and after that you normally don't touch often), and the updating of the fsh-generated resources.
Jose Costa Teixeira (Jul 30 2020 at 19:44):
I am happily surprised to see that the second of my bullet point may be working well with sushi --init (which i tried, doesn't work, but i will wait for next release)
Jose Costa Teixeira (Jul 30 2020 at 19:44):
but the first point shows one thing that I consider a design flaw:
Jose Costa Teixeira (Jul 30 2020 at 19:45):
sushi regenerates everything every time it runs. if sushi has a ig.ini as input, it should not touch it (unless you explicitly tell it to)
Jose Costa Teixeira (Jul 30 2020 at 19:47):
and there should be a way for sushi to know whether to replace some artifacts.
Jose Costa Teixeira (Jul 30 2020 at 19:51):
Suggestion : why not give sushi a parameter -ihsus <foldername> and sushi would then use the stuff in that folder when they'd conflict with what sushi just generated. That folder could contain an ig.ini, a menu.xml, some artifacts... This would be simple.
Are there any donwsides that I can't think of?
Jose Costa Teixeira (Jul 30 2020 at 19:52):
(btw, ihsus means obviously InHeritedStuffthatshallremainUntouchedasiS)
Chris Moesel (Jul 30 2020 at 19:54):
if sushi has a ig.ini as input, it should not touch it (unless you explicitly tell it to)
Can you elaborate on that? If you have {{ig.ini}} outside of the {{ig-data}} folder, SUSHI won't touch it. If you have it inside of the {{ig-data}} folder, I think SUSHI copies it to the output folder, but that's all. So what touching is it that you don't want SUSHI to do?
Chris Moesel (Jul 30 2020 at 19:57):
and sushi would then use the stuff in that folder when they'd conflict with what sushi just generated. . That folder could contain an ig.ini, a menu.xml, some artifacts...
Today, if you don't include template, menu, history properties in the config.yaml, SUSHI won't generate those things. And if you don't have ig.ini, menu.xml, package-list.json in their relative locations in the ig-data folder, it won't copy them either. So if you have ig.ini, menu.xml, and package-list.json anywhere but ig-data, SUSHI won't touch them (as long as you don't specify their contents in config.yaml either).
Chris Moesel (Jul 30 2020 at 19:59):
For resources, if SUSHI finds a resource in one of the supported ig-data/input subfolders, and that resource matches canonicals with a resource SUSHI generated, SUSHI will prefer the one in ig-data/input/* and copy that over instead of overwriting it with the generated content.
Chris Moesel (Jul 30 2020 at 20:00):
In the next few days I hope to put a fuller proposal on the table for how SUSHI / Publisher integration can work in a more seamless way. I hope it will address some of the issues you've brought up, so I will appreciate your feedback!
Jose Costa Teixeira (Jul 30 2020 at 20:02):
suppose if I want my config.yaml to generate all that stuff except an ig.ini.
That is what I'm proposing.
Jose Costa Teixeira (Jul 30 2020 at 20:03):
we can have an option for each of the things that sushi generates, but wouldn't it be easier to just tell sushi :
"hey sushi, when you're done, here's a couple of files to replace anything with the same name even if you may have created it"
Jose Costa Teixeira (Jul 30 2020 at 20:04):
that would be a simple thing, I think. then it's up to me to decide which files I want to have there (and replace those created by sushi)
Jose Costa Teixeira (Jul 30 2020 at 20:05):
in this, I am making a separation between input data and output data (which I hope it's agreeable - it's complicated today with the integration)
Jose Costa Teixeira (Jul 30 2020 at 20:08):
if we have an optional input folder besides an output folder, this should be easy, i think.
Today there is an issue that the output folder is a child of the input folder, unless when run by the ig publisher, which puts the output in the parent folder to the input.. that is complicated.
John Moehrke (Jul 30 2020 at 20:08):
FSHOnly works well for me
Jose Costa Teixeira (Jul 30 2020 at 20:09):
(i use fshonly as well, this is the broader set of problems that I mention above)
Chris Moesel (Jul 30 2020 at 20:11):
I think maybe I'm not fully understanding. With the exception of the IG resource (and I realize that is a big exception), you can control whether or not SUSHI generates each file by excluding its corresponding property in the SUSHI config. If you don't want it to generate a resource (profile, extension, etc) you can comment it out -- or as I noted above, put an alternate non-FSH version into ig-data that will be used instead. But I don't understand why you would want to tell SUSHI to generate something AND then also tell it to ignore what it generated and use something else instead. If you want it to use something else instead, then tell SUSHI not to generate it anymore and supply your own "something else".
Jose Costa Teixeira (Jul 30 2020 at 20:12):
i think I realized it was easier to have sushi replace the file instead of telling it not to generate it.
Jose Costa Teixeira (Jul 30 2020 at 20:12):
easier and more powerful
Chris Moesel (Jul 30 2020 at 20:17):
I guess I find it confusing if you specify multiple input sources, which is what you're doing when you provide info in the config that says to generate the file but also provide a file to replace the generated one. For example, I can imagine people looking at the output menu.xml file and looking at the configured menu content in the config and wondering why they don't match at all (until they realize there is a menu.xml override in some other folder). I really think it's not a great idea to specify two inputs for the same output and rely on rules to determine which one is used and which one is tossed out.
Jean Duteau (Jul 30 2020 at 20:18):
and wasn't that the whole point of the config.yaml - to have one file that ruled it all and told me what was being used for what?
Jose Costa Teixeira (Jul 30 2020 at 20:19):
In this thread i had some random proposals but here is short:
-
allow an "input stuff" that sushi can take as optional parameter, and whatever is in that folder, sushi will copy to the folder where it generates the IG
1.1 use that in the integration e.g. ask Grahame to add a fixed parameter to the way sushi is executed inside the IG publisher -
Make sushi have 2 modules, with options:
2.1 Module 1 IG generation
2.1.1 init a default ig (--init)
2.2.1 Recreate everything about IG (--regen?) - can be by assumed to be default if there is nothing in the input
2.2 Module 2 Shorthand parser
2.2.1 generate everything and ignore input folder
2.2.2 generate everything and use input folder to overwrite
Jose Costa Teixeira (Jul 30 2020 at 20:22):
Chris Moesel said:
I guess I find it confusing if you specify multiple input sources, which is what you're doing when you provide info in the config that says to generate the file but also provide a file to replace the generated one.
yes, it's a downside. I think it is way better than the other options. If a person has an input folder, they know what they are doing. If input folder is empty, they leave their fate to sushi and that is fine.
Jose Costa Teixeira (Jul 30 2020 at 20:23):
what i suggested would also solve the problem with BYOx files.
Jean Duteau (Jul 30 2020 at 20:23):
don't we have most of that already?
1) that is the ig-data/input folder
2.2.1 - FSHOnly = true
2.2.2 - default
Jose Costa Teixeira (Jul 30 2020 at 20:24):
2.2.1 yes.
2.2.2 and 1. - almost, the idea is that it would just replace whatever is in the output.
Jose Costa Teixeira (Jul 30 2020 at 20:24):
this is why I think it is an easy thing
Jose Costa Teixeira (Jul 30 2020 at 20:24):
(above i was thinking of extensions and some more crazy stuff. This is simpler).
Jose Costa Teixeira (Jul 30 2020 at 20:25):
concretely, @Jean Duteau it's not only the ig-data/input folder, it's also the ig.ini
Jean Duteau (Jul 30 2020 at 20:26):
i think I'm with Chris in that I'm missing the point of what you are proposing. Is it "create everything in my FSH files and copy over everything in my ig-data/input folder, even if it overwrites something you just created"?
Jose Costa Teixeira (Jul 30 2020 at 20:28):
almost
Chris Moesel (Jul 30 2020 at 20:29):
The only difference I'm seeing is that you don't want SUSHI to complain if you configure ig.ini content in the config and supply an ig.ini in ig-data (and the same for menu, etc). The thing is, those things are already opt-in in the config. So putting ig.ini or menu info in the config is an explicit (and intentional) action taken by the author to tell SUSHI to generate those files. I don't know why you would so explicitly opt in if you then want SUSHI to overwrite them with some other file anyway.
Chris Moesel (Jul 30 2020 at 20:30):
SUSHI used to generate those things by default, but not anymore. Now you have to opt in by adding specific parameters in your config.yaml.
Jose Costa Teixeira (Jul 30 2020 at 20:30):
"when you're done creating everything (from my FSH files and config), copy over everything in myInputFolder folder to that, overwriting anything you just created"
Jean Duteau (Jul 30 2020 at 20:31):
wow, that seems pretty dangerous
Jose Costa Teixeira (Jul 30 2020 at 20:31):
why?
Jean Duteau (Jul 30 2020 at 20:32):
so I have a bunch of examples that I've created in FSH, but there is an example in myInputFolder that I don't realize. I keep making changes to the FSH file and not seeing the change and I don't know why
Jose Costa Teixeira (Jul 30 2020 at 20:32):
(i'm avoiding mentioning the ig-data folder because that is kind of a Moebius strip.)
Jose Costa Teixeira (Jul 30 2020 at 20:34):
Jean Duteau said:
so I have a bunch of examples that I've created in FSH, but there is an example in myInputFolder that I don't realize. I keep making changes to the FSH file and not seeing the change and I don't know why
yep. that inconvenience is to me a very acceptable downside, if the upside is that you can tell sushi "hey, i've just translated this codesystem in 3 languages, please don't overwrite it"
Chris Moesel (Jul 30 2020 at 20:34):
@Jose Costa Teixeira -- why not comment out or delete the code system in the FSH source? That would eliminate ALL confusion.
Chris Moesel (Jul 30 2020 at 20:35):
But as I said above, SUSHI will actually overwrite a SUSHI-generated resource with one in ig-data if it is a match (sorry, @Jean Duteau!).
Chris Moesel (Jul 30 2020 at 20:36):
(But it doesn't do that with configuration resources)
Jean Duteau (Jul 30 2020 at 20:36):
does it warn you if it is overwriting?
Jose Costa Teixeira (Jul 30 2020 at 20:37):
Ha! you mentioned the ig-data. (we should have a rule that someone pays a beer when they use that in a discussion about redesigning the structure)
Jean Duteau (Jul 30 2020 at 20:37):
i.e. based on the canonical like you said earlier, it warns you that it's overwriting?
Chris Moesel (Jul 30 2020 at 20:38):
I guess that if your requirement is to have a special folder that you use to specify files that should be copied, I don't see why we need to complicate things by insisting that users can call the folder whatever they want. So... let's just call this folder ig-data for now.
Chris Moesel (Jul 30 2020 at 20:39):
@Jean Duteau -- I don't remember, but I can check.
Jose Costa Teixeira (Jul 30 2020 at 20:39):
if you don't assume that sushi is the beginning of all the IGs (you got something already working , you just want to add some stuff, maybe even a because that is easier), isn't that much simpler?
Jose Costa Teixeira (Jul 30 2020 at 20:39):
2 beers.
Jose Costa Teixeira (Jul 30 2020 at 20:41):
I can call it ig-data, but to me that presumes it is a folder under the fsh folder, which is something I'd challenge. But sure let's call it that.
Jean Duteau (Jul 30 2020 at 20:41):
Jose Costa Teixeira said:
if you don't assume that sushi is the beginning of all the IGs (you got something already working , you just want to add some stuff, maybe even a because that is easier), isn't that much simpler?
But that isn't sushi's happy path, so I don't think so. Sushi's happy path is where a user is using FSH and the ig-data folder to define all of their IG content. That is what many of us are doing and it's how I'm preaching about SUSHI and FSH to projects in Canada.
Jose Costa Teixeira (Jul 30 2020 at 20:41):
(see? I just gave you a belgian beer :smile: )
Chris Moesel (Jul 30 2020 at 20:45):
I have to switch over to my other project now... but again, I'll be revisiting all this very soon... so I'm sure there will be further discussion. Feel free to talk amongst yourselves, but I'm signing out for now!
Jose Costa Teixeira (Jul 30 2020 at 20:45):
Jean Duteau said:
Sushi's happy path is where a user is using FSH and the ig-data folder to define all of their IG content. That is what many of us are doing and it's how I'm preaching about SUSHI and FSH to projects in Canada.
this should be rephrased, right? Sushi cannot be the only source of IG content- logical models, retouching the conformance resources- this is not in sushi.
Jose Costa Teixeira (Jul 30 2020 at 20:47):
and it's not that this way of working has been really there for a long time and without any issues and complaints...
Jose Costa Teixeira (Jul 30 2020 at 20:48):
I know it is a change. I am not saying my proposal is bullet proof. I'm proposing and definding it and I like the feedback. But it seems a simple change and I don't see why it would not be better.
Jose Costa Teixeira (Aug 03 2020 at 17:48):
@Chris Moesel does this proposal help?
In short, it's about using an input folder (with the standard IG folder structure) and letting sushi use that for parsing fsh and also to copy that content over to the final output. Simpler than specific flags, and more powerful for those cases where the user must see what they want Sushi to replace or not.
Chris Moesel (Aug 03 2020 at 17:56):
@Jose Costa Teixeira -- did you intend to link to a specific proposal? Re: the general gist, the proposal I was going to build on sounds somewhat similar -- in short, you put your .fsh files in input/* just like you do any formal FHIR artifact -- and SUSHI builds it to a special temp folder that the IG Publisher knows to pickup (keeping SUSHI src separate from output). I was thinking that temp folder would be outside of input, but I guess it could also be something like input/sushi-out. Anyway, I was going to draft something up for internal review today and probably post it here tomorrow. But that's basically the long and short of it.
Jose Costa Teixeira (Aug 03 2020 at 18:00):
I was just feeding the general thoughts to you - basically having 3 folders (configurable as command line parameters): ig-input , fsh-input, and output
But I will prepare something more detailed
Last updated: Apr 12 2022 at 19:14 UTC
