Stream: IG creation
Topic: moving away from ig.json
Grahame Grieve (Sep 01 2018 at 02:12):
So we've previously agreed that we're going to move away from ig.json to use the IG resource directly, and I'm getting ready to do that. (it was always scheduled for this gap between ballot opening and baltimore, though I lost a week :-( )
Grahame Grieve (Sep 01 2018 at 02:14):
one useful function that the ig.json file performs is that it's at a known location. In terms of replacing it, we can do one of the following things:
- leave a minimal json file at the same place that instructs where to find the IG resource
- define a known path where the IG resource must be found (e.g. /Resources/ig.[xml|json])
- say that the IG resource must be located at ig.[xml|json]
- say that there's an ini file at ig.ini that says where to find the IG resource
- something else I haven't thought of
Lloyd McKenzie (Sep 01 2018 at 02:48):
ini file is easier than a json file. If we're using certain tools to author content, the filename will likely have to include the id of the IG, which makes a consistent IG filename difficult. And there could be more than one IG file present, so just searching all files for IG files wouldn't be safe.
Lloyd McKenzie (Sep 01 2018 at 02:48):
I would love if we could make the output directory name configurable too... :)
Grahame Grieve (Sep 01 2018 at 02:57):
we would we bother with that?
Grahame Grieve (Sep 01 2018 at 02:57):
ini file is not meaningfully easier than a json file if there's 2 or 2 items in it
Eric Haas (Sep 01 2018 at 05:11):
I think it would be useful to keep the ig.json as minimal file.
And the old ig.json already makes the output directory name configurable too - should be an option in the new ig.json too, I add a whole bunch of my own stuff in there too.
Lloyd McKenzie (Sep 01 2018 at 19:37):
Output folder isn't configurable if you're using the CI Build.
Lloyd McKenzie (Sep 01 2018 at 19:39):
My concern with an ig.json file is it'll be tempting to add new stuff into it. A hard switch seems cleaner. And ini's innate limitations make it (slightly) less likely to be abused. That said, this isn't a hill I'd die on.
Eric Haas (Sep 04 2018 at 00:39):
"My concern with an ig.json file is it'll be tempting to add new stuff into it" Which I do for my preprocessing. And there are other options to the autobuilder thus the need for the output dir option.
Lloyd McKenzie (Sep 04 2018 at 00:42):
The point is to move it so that the IGPublisher does any necssary pre-procssing. The objective is to ditch ig.json. The fact you're hoping to keep information in it actually reenforces my point that it needs to go away...
Eric Haas (Sep 04 2018 at 00:52):
Its not going to prevent anything and its a convient place to stick custom stuff.
Richard Townley-O'Neill (Sep 04 2018 at 01:11):
(deleted)
Lloyd McKenzie (Sep 04 2018 at 02:02):
Custom stuff can be stuck as extensions in the ImplementationGuide. If we move from ig.json to ig.ini, it forces a change. If the direction is "change to something in the IG", it's more likely that people will do that than if we stick with a .json file.
Eric Haas (Sep 04 2018 at 02:32):
OK, its not something I'll get too worked up about.
Chris Moesel (Sep 04 2018 at 18:08):
If you want a minimally structured file, but also want to be sure it's not confused for the old ig.json file, you could also make it a YAML file (ig.yml). Using YAML for tool config is a fairly common practice.
Grahame Grieve (Sep 04 2018 at 22:06):
for me that would YAP (yet another parser)
Sean McIlvenna (Sep 06 2018 at 14:11):
why use an ini or json file at all?? Seems to me that we're talking about a very SMALL number of things that needs to be conveyed to the tool. I would think those could simply be parameters to the JAR's execution... ex:
java -jar publisher.jar -workingDirectory /home/sean/someIgDir -ig /home/sean/someIgDir/resources/ImplementationGuide/some-ig.xml -etc .. -etc .. -etc ..
Grahame Grieve (Sep 06 2018 at 21:57):
because there's no way to feed those parameters to several contexts in which it runs. I, for instance, often am running 20-30 different IGs in parallel. I'm not interested in having to read human documentation to figure out a special parameter. if we're going to have one, it's going to be a parameter that the IG publisher understands. and for the auto build
Grahame Grieve (Sep 07 2018 at 05:14):
ok. ig.ini it will be. Shortly (possibly before the freeze) I'm going to commit a version of the IG publisher that generates an ig.ini and a new IG that will be the alternative to ig.json.
Grahame Grieve (Sep 07 2018 at 05:14):
here's my notes for what I'm doing with the things in ig.json:
Grahame Grieve (Sep 07 2018 at 05:14):
"source" - moved to [ig.ini] "version" - moved to ig.version "fixed-business-version" - moved to ig.parameter "license" - moved to ig.license "logging" - moved to command line "html-template" - moved to parameter "suppress-qa" - moved to command line (and default to suppress) "paths" "resources" - extension igpublish-folder-resources on ig "pages" - extension igpublish-folder-pages on ig "temp" - dropped: hard coded to temp "output" - dropped: hard code to output "txCache" - dropped: hard coded to txCache "qa" - dropped to command line "history" - dropped. hard code to history.html "specification" - dropped (implicit in version) "jurisdiction" - ig.jurisdiction and parameter to populate "npm-name" - moved to ig.packageName "npm-template" - dropped (never used) "language" - dropped for now "languages" - dropped for now "pre-process" - dropped - move stuff to template "canonicalBase" - dropped; implicit in URL "dependencyList"- move to ig.dependsOn. "source" dropped (use package resolution instead) "sct-edition" - move to ig-expansion-parameters.json "no-inactive-codes" - move to ig-expansion-parameters.json "broken-links" - move to ig-validation-parameters.json "check-aggregation" - move to ig-validation-parameters.json "check-mustSupport" - move to ig-validation-parameters.json "anyExtensionsAllowed" - move to ig-validation-parameters.json "hintAboutNonMustSupport" - move to ig-validation-parameters.json "suppressedWarningFile" - move to ig-validation-parameters.json "extension-domains" - move to ig-validation-parameters.json "special-urls" - move to ig-validation-parameters.json "extraTemplates" - dropped for now "gen-examples" - dropped for now "do-transforms" - dropped for now "defaults" - dropped; xml/json/ttl controlled by parameters. everything else generated "spreadsheets" - moved to extension http://hl7.org/fhir/StructureDefinition/igpublisher-spreadsheet "bundles" : - moved to extension http://hl7.org/fhir/StructureDefinition/igpublisher-bundle "resources": - moved to extensions on ig.definition.resource "swagger" - dropped (for now) "tool" - dropped
Grahame Grieve (Sep 07 2018 at 05:41):
ok added. if you run your IG now, you should get ig.ini, and ig-new.json/xml created for your review.
Lloyd McKenzie (Sep 07 2018 at 06:44):
So the things that are dropped for now - are they coming back? Because some of those things are pretty important for some IGs. Also, have you documented how the templating stuff works now?
Grahame Grieve (Sep 07 2018 at 06:53):
which dropped things do you identify as important?
Grahame Grieve (Sep 07 2018 at 06:55):
templates: http://wiki.hl7.org/index.php?title=FHIR_IG_publisher_templates
Eric Haas (Sep 07 2018 at 07:19):
re: ig.ini can we create our own instead as well? or do we still need ig.json. ( Since I start from a csv file - I'd rather go straight to the ig.ini or Ig.json ( resource)
"ig-new.json/xml created for your review". is this the IG Resource?
Grahame Grieve (Sep 07 2018 at 07:26):
yes that's the IG resource. that'll be the master in the future. We won't be having ig.json. or supporting pre-processing of the IG. The only pre-processing we'll be supporting is in the templates
Lloyd McKenzie (Sep 07 2018 at 18:53):
1. temp being hard-coded doesn't work - I need 2 temp folders when I'm generating multiple versions
2. ditto for output. (And if I'm running multiple IGs from a common source, I need to specify different output folders for them)
3. txCache cache on the root is annoying. I put all of the interim stuff that is deletable but commitable for performance regions under a single folder - and I want to be able to specify where that is.
4. extraTemplates I need
5. gen-examples looks like an Eric thing
6. do-transforms - so long as templates handle this, could be ok
Grahame Grieve (Sep 07 2018 at 20:41):
#1 - still can be hardcoded
#2 - "running multiple IGs from a common source" - not supported anyway
#3 - what other interim stuff is there? and so what?
#4 - what do you need extraTemplates for
#5 - gen-examples is my thing, and I haven't decided what to do about it
#6 - the intent is for templates to do it, yes
Lloyd McKenzie (Sep 07 2018 at 21:16):
1 - Don't understand - I specify two different temp folders when I run crd. How will that work
2 - It is supported - I've been doing it for almost 3 years with a client and we'd better not be taking it away...
3 - files generated by the pre-processing that don't necessarily need to be updated every time, current copy of the schemas for the appropriate version(s) for validation when editing artifacts, current copy of the igpublisher
4. I hve templates for mappings, examples, profile-json and profile-xml
Lloyd McKenzie (Sep 07 2018 at 21:18):
Note on #2 - I'm not expecting it to be supported for the CI Build environment - at least not yet. But it's a completely legitimate thing to do otherwise
Sean McIlvenna (Sep 07 2018 at 21:28):
How will this affect building stu3 implementation guides? Many of the ig.json properties are moved to ImplementationGuide in R4, but those properties aren't going to be available in STU3. So... are we defining extensions for each of these properties for STU3 implementation guides? I assume the ig publisher is still planned to be backwards compatible...
Lloyd McKenzie (Sep 07 2018 at 21:31):
At one point we'd said that everyone would have to move to authoring using R4 IGs, but I think the roundtrippable extension approach is probably better...
Eric Haas (Sep 07 2018 at 22:13):
@ Lloyd can you translate what that means? I understood is all going to use the latest IG Resource (e.g. R4) and I already have coded for that.
Eric Haas (Sep 07 2018 at 22:16):
@Grahame? re: "The only pre-processing we'll be supporting is in the templates" I preprocess before starting the IG-pub jar. I'm generating the IG resource and I guess the ig.ini files and other goodies.
Grahame Grieve (Sep 07 2018 at 22:18):
@Eric Haas I have no intention of allowing for generating the IG resource. nor the ig.ini file. the ig.ini file is only edited once. And the IG resource is the master
Grahame Grieve (Sep 07 2018 at 22:19):
I am not interested in pre-processing outside templates at all
Eric Haas (Sep 07 2018 at 22:19):
how does one create an IG Resource for a new guide then?
Grahame Grieve (Sep 07 2018 at 22:20):
I'm not round tripping all the IG publisher stuff with earlier versions. here's how it's going to work:
- author the IG resource in R4
- Ig publisher will publish the IG in the specified version
Grahame Grieve (Sep 07 2018 at 22:20):
you edit it
Grahame Grieve (Sep 07 2018 at 22:21):
when forge goes to R4, then you can edit it in forge. But that won't support all the extensions anyway, so you'll just end up editing it in text.
Eric Haas (Sep 07 2018 at 22:21):
author the IG resource in R4 - +1
Eric Haas (Sep 07 2018 at 22:24):
Perhaps I am using the wrong words I'm not preprocessng templates - they are static. I create the IG resource instance on the fly before running the ig-pub.
Grahame Grieve (Sep 07 2018 at 22:25):
I don't know you do that, or why we should support that
Eric Haas (Sep 07 2018 at 22:30):
You don't need to support it. My question is do I need to support this :
"Grahame Grieve: ok. ig.ini it will be. Shortly (possibly before the freeze) I'm going to commit a version of the IG publisher that generates an ig.ini and a new IG that will be the alternative to ig.json."?
...or .... can I just create a R4 IG resource myself and ig,ini myself ???
Grahame Grieve (Sep 07 2018 at 22:31):
that's a transitional arrangement - it takes the content from ig.json and the ig resource, merges them, and then sets up the source directory for how it will need to look once we make the transition.
Grahame Grieve (Sep 07 2018 at 22:32):
once we make the transition, then ig.json will not be allowed to present, and ig.ini and the ig resource will be the master
Grahame Grieve (Sep 07 2018 at 22:32):
for now, I'm looking for the editors following this discussion to run the latest code on their various igs and check that it looks ok
Eric Haas (Sep 07 2018 at 22:33):
OK got it
Lloyd McKenzie (Sep 07 2018 at 22:36):
I start with an IGResource - but I invoke the IGPublisher after I've tweaked it - or in some cases split it into two. Will that still work?
Grahame Grieve (Sep 07 2018 at 22:36):
split it into 2? what does that mean?
Lloyd McKenzie (Sep 07 2018 at 22:37):
I start with a single IG with some embedded extensions that say "this stuff is only for the R3 version". Then I generate a version of the IG for R4 only and a different one for R3 only.
Grahame Grieve (Sep 07 2018 at 22:45):
I don't see why that's a good idea. Just have separate IGs. the content will be very different for them now
Lloyd McKenzie (Sep 07 2018 at 22:46):
The metadata needs to be the same - and they get combined into a single publication. The heading numbers, etc. need to be continguous across the content in each.
Grahame Grieve (Sep 07 2018 at 22:47):
in general, I'm simply not interested in all this complexity. It makes everything sooooo much harder, and we can't deliver on what we have. I'm sick of inventing complexity and then gtting hammered because the tools are constantly broken
Lloyd McKenzie (Sep 07 2018 at 22:47):
Separate files is just begging for inconsistencies and other problems.
Lloyd McKenzie (Sep 07 2018 at 22:47):
The complexity comes from real requirements
Grahame Grieve (Sep 07 2018 at 22:48):
you can just keep inventing real requirements, sure. But I have to actually deliver on this stuff. You make contributions, but it's me that maintains, and supports, and keeps things going.
Grahame Grieve (Sep 07 2018 at 22:48):
and I can't keep doing this. We have to simplify
Lloyd McKenzie (Sep 07 2018 at 22:50):
In my mind, simplifying comes from the templates. If we're driving off standard templates, most people will be isolated from the complexity.
Lloyd McKenzie (Sep 07 2018 at 22:50):
Multi-version IGs are going to become more common, not less common.
Lloyd McKenzie (Sep 07 2018 at 22:51):
And it's pretty hard to go to a customer that's had their content and publishing process organized around a single repository for 3 IGs to suddenly say "sorry, you can't do that anymore because we decided it was too complicated"
Grahame Grieve (Sep 07 2018 at 22:52):
customers that depend on critical infrastructure... at some stage they need to actually pay.
Lloyd McKenzie (Sep 07 2018 at 22:53):
They have, in part. They've paid for a huge amount of my time to address issues, maintain stuff. I recognize that hasn't flowed to you for the parts you do though.
Grahame Grieve (Sep 07 2018 at 22:58):
no, and that's a problem.
Lloyd McKenzie (Sep 07 2018 at 22:59):
The base issue is: you don't have the bandwidth to provide support (for free) for more than a certain minimal set of functionality. But the functionality needed by the community is more than that.
Grahame Grieve (Sep 07 2018 at 22:59):
if we're going to support multi-version IGs, we need to figure out how we're actually going to support them. The current arrangement feels like a dreadful hack. I only tolerated it because we had ballot deadlines. though I'm seriously burnt by the publishing process from the last round, and guidance from exec committee is: we won't do that again
Eric Haas (Sep 07 2018 at 23:02):
I going to have to side with Lloyd on the need for multi version IG but I don't think we should stick it to GG to solve. Maybe we ask our client to provide the funding to support developing a maintainable solution or stick to our guns and say we can't do it.
Lloyd McKenzie (Sep 07 2018 at 23:03):
I'm open to us discussing alternatives to how to publish multi-version IGs.
Lloyd McKenzie (Sep 07 2018 at 23:03):
It's hard to say "we can't do it" - when we can. And DaVinci (through me) spent money to make it happen.
Lloyd McKenzie (Sep 07 2018 at 23:04):
We can however say that more resources are needed to move the solution to something more sustainable.
Brian Reinhold (Sep 07 2018 at 23:04):
I'm open to us discussing alternatives to how to publish multi-version IGs.
what's a multi-version IG in this context? Different ways to generate IGs? Different appearances?
Eric Haas (Sep 07 2018 at 23:04):
STU3 +R4 concurrently
Grahame Grieve (Sep 07 2018 at 23:04):
I guess I should've hit Da Vinci up for the multi-version support stuff I did. The problem is that it all just merges into one great big ball of stuff for me. Things are not easily separated out
Grahame Grieve (Sep 07 2018 at 23:05):
really, we want deeply integrated IGs, where we can have different version for different profiles.
Brian Reinhold (Sep 07 2018 at 23:05):
STU3 +R4 concurrently
OH, Not what I was thinking. But thanks for the clarification
Grahame Grieve (Sep 07 2018 at 23:05):
and if we extend into R2 support, different terminology resources
Grahame Grieve (Sep 07 2018 at 23:05):
the problem is actually in the package spec
Grahame Grieve (Sep 07 2018 at 23:06):
and all the tools, which all use the package spec
Lloyd McKenzie (Sep 07 2018 at 23:13):
I'm not sure. I think it's reasonable to have separate packages for one IG "publication". Multi-version packages seem like they're going to cause problems for more than just the the build process. In any case, DaVinci is currently figuring out plans and budgets for next quarter - if this is something they need to plan for/deal with, need to let Bob & Viet know.
Sean McIlvenna (Sep 07 2018 at 23:15):
... catching up on the thread.
I am strongly against the notion of having to create an R4 ImplementationGuide for an STU3 package.
It makes far more sense to me, to use extensions to represent the properties in R4 ImplementationGuide that are not implemented in STU3.
Grahame Grieve (Sep 07 2018 at 23:19):
these are really big extensions and really lots of work for me. WHy are you strongly against it? the IG will be a mess. We made a lot of changes
Brian Reinhold (Sep 07 2018 at 23:22):
... catching up on the thread.
I am strongly against the notion of having to create an R4 ImplementationGuide for an STU3 package.
It makes far more sense to me, to use extensions to represent the properties in R4 ImplementationGuide that are not implemented in STU3.
In my trivial little 3-FHIR resource case an R4 IG wouldnt be anything like an STU3. All the content would be completely different.
Grahame Grieve (Sep 07 2018 at 23:24):
why would it be completely different? your profiles would be different. What else would be different?
Brian Reinhold (Sep 07 2018 at 23:27):
why would it be completely different? your profiles would be different. What else would be different?
Okay, that's what I meant. The content in the resources/pages directories would be completely different. What otherwise lies under the hood in generating an IG is beyond me. But simple (as much as possible) is nice. It does seem like things would be cleaner if there were separate guides for the different versions but I am really not qualified to comment on that. I'm still at the stage of learning how to use the publisher and generate the resources/pages. - making the move from Simpplifier
Grahame Grieve (Sep 07 2018 at 23:28):
your documentation should not that different.your value sets should be the same. you do have value sets, right?
Brian Reinhold (Sep 07 2018 at 23:29):
your documentation should not that different.your value sets should be the same. you do have value sets, right?
Not too many and they are simple. IN most cases I use already existing ones - I try as much as possible to use what it defined in the base standard.
Grahame Grieve (Sep 07 2018 at 23:38):
smells like you should be investing in value sets more then.
Grahame Grieve (Sep 07 2018 at 23:39):
but irrespective, most people will.
Grahame Grieve (Sep 07 2018 at 23:39):
I think it's reasonable to have separate packages for one IG "publication". Multi-version packages seem like they're going to cause problems for more than just the the build process
Grahame Grieve (Sep 07 2018 at 23:40):
not sure about this. when a tool hits a package, it knows which version it wants.
Grahame Grieve (Sep 07 2018 at 23:41):
except the IG publisher. which wants all versions.
Brian Reinhold (Sep 07 2018 at 23:43):
@Grahame Grieve All these details are beyond me at this time. I'm still too much of a noob to know how it all works. I am amazed at what you guys are doing and are able to do.
Grahame Grieve (Sep 07 2018 at 23:47):
well, we should sit down with you and work through the value set stuff in baltimore. telling people what codes can be used in what contexts is mostly very important
Brian Reinhold (Sep 08 2018 at 00:01):
well, we should sit down with you and work through the value set stuff in baltimore. telling people what codes can be used in what contexts is mostly very important
I will take any help I can get!
Eric Haas (Sep 08 2018 at 00:36):
I don't think using R4 IG instead of ig.json + Ig.xml that we are now is any more complex.. The data is all the same. Its just a bit of repackaging
Lloyd McKenzie (Sep 08 2018 at 00:44):
Agree with that. The change of syntax is about having a "standard" way to share all of the publishing information to allow other tools to produce/consume the information.
Lloyd McKenzie (Sep 08 2018 at 02:39):
In reading through the wiki page on templates - am I correct in understanding that templates will still use config.json?
Grahame Grieve (Sep 08 2018 at 08:42):
I think so. there's a bunch of things that you want to say there that are outside the scope of the IG
Grahame Grieve (Sep 08 2018 at 08:44):
umm no. I think that this wants to be an IG resource. But I'm not sure. that's one of the thing we have to experiment with
Grahame Grieve (Sep 08 2018 at 08:47):
what is it we want to say in a template?
Brian Reinhold (Sep 08 2018 at 11:28):
I think so. there's a bunch of things that you want to say there that are outside the scope of the IG
What role does the template play? Is this the part that gives the content/appearance of the headers and footers on a given page?
What happens with the IG publisher when I create pages that have no (what I believe are) Jekyll commands like {% include header.html %}?
I created a massive amount of pages in markdown format in Simplifier. They have no special 'commands'. Can I use these pages in the IG publisher without change if I populate the ig.json correctly?
Can I use these Jekyll commands in a markdown document? If so, how?
Lloyd McKenzie (Sep 08 2018 at 15:37):
The template sets expectations for how artifacts will be rendered, what the headers and footers will be on all pages, how the table of contents and/or artifact indexes are generated, what the styles are, helps with th e creation of section numbers, etc. If the framework is well designed, there is very little liquid you should need to add to your stuff. The main thing will be replacing any explicit references to the URL for FHIR (e.g. http://hl7.org/fhir) with {{site.data.fhir.path}}. That allows the publication process to "lock" your IG to a particular ballot release of FHIR or specific official release of FHIR (e.g. http://hl7.org/fhir/R4).
Lloyd McKenzie (Sep 08 2018 at 15:45):
@Grahame Grieve Ideally, I'd like an IG template to cover everything I currently put in the "framework" folder in my IGs - so defining all of the styles, javascript, Jekyll config, page templates and standard includes as well as the transforms and scripts involved. I set most of my config stuff using a transform. So if we can allow generation of the actual "build" IG as part of the launch stuff, I won't need a file to contain it - it'll be embedded in an XSLT. Though this approach does mean that the source IG needs to be expressed in XML (not RDF or JSON). If we're not comfortable with that as a limitation, I'm not thrilled with the idea of an IG. Because it really isn't one. It has no resources. It doesn't have any pages, per se. It doesn't make sense to publish it or use it as an IG. It's just trying to set lots of expectations around IG publishing. And, to be honest, I'm not sure how we do some of that genericly - my transform will often list out a bunch of the artifacts with specific templating rules for each defined because I need to differentiate between "example" StructureDefinitions and "profiling" StructureDefinitions - each gets rendered differently.
Lloyd McKenzie (Sep 08 2018 at 15:48):
An IG template package that just let me point to and bring in a non-editable version of the framework folder to use in the build plus an ability to pre-process the IG would be ideal. If we had an ini file that pointed to the IG and the template and the IGPublisher launched from that, everything else would be pretty straight-forward.
Grahame Grieve (Sep 08 2018 at 20:55):
this sentence:
So if we can allow generation of the actual "build" IG as part of the launch stuff, I won't need a file to contain it - it'll be embedded in an XSLT.
you'll need to explain this
Grahame Grieve (Sep 08 2018 at 20:56):
I need to differentiate between "example" StructureDefinitions and "profiling" StructureDefinitions - each gets rendered differently.
These are differentiated in the IG
Grahame Grieve (Sep 08 2018 at 20:57):
an ability to pre-process the IG would be ideal
Grahame Grieve (Sep 08 2018 at 20:57):
what do you think you need to do here?
Grahame Grieve (Sep 08 2018 at 20:57):
Remember: what we agreed is that the IG will be the master file, that everything else derives from
Lloyd McKenzie (Sep 08 2018 at 22:24):
I want to tweak the source IG before the IGPublisher runs with it - to insert pages, manipulate build parameters (and maybe even split it in two).
Yes, they're differentiated in the IG. But the build templates drive off artifact type, not whether something is flagged as an example or not
Lloyd McKenzie (Sep 08 2018 at 22:26):
The IG is the master file, except that there's a whole bunch of stuff that's template driven. So the IG is the master only insofar as it points to a template. And the problem with it pointing to the template is that you load the IG, find the template, then change the IG, then kick off the build with the revised IG. That seems a little flakey. But you absolutely want the scripts to be able to drive changes to the IG before it gets build (e.g. inserting the pages for each artifact - maintaining that in the source IG is a huge pain and source of errors)
Grahame Grieve (Sep 08 2018 at 22:55):
our discussion was very clear: we will work the IG resource so it is suitable to be the master
Grahame Grieve (Sep 08 2018 at 22:56):
sounds like you're not on board with that
Grahame Grieve (Sep 08 2018 at 22:57):
manipulate build parameters
why? Why not just say what you want?
maybe even split it in two
why?
inserting the pages for each artifact - maintaining that in the source IG is a huge pain and source of errors
why? Got to maintain it somewhere
Lloyd McKenzie (Sep 08 2018 at 23:36):
Build parameters are mainly driven by framework. Most IG authors care about resources and "extra" pages. The framework takes care of everything else
Lloyd McKenzie (Sep 08 2018 at 23:36):
Split in 2 = for multiple versions
Lloyd McKenzie (Sep 08 2018 at 23:37):
We maintain the artifacts in the IG. Creating pages for each artifact too is redundant. Much better for the framework to add them automatically
Lloyd McKenzie (Sep 08 2018 at 23:38):
That way if someone wants to change the name of something or add something or delete something, they only have to do it in one place, not two.
Grahame Grieve (Sep 09 2018 at 10:55):
Creating pages for each artifact too is redundant. Much better for the framework to add them automatically
Grahame Grieve (Sep 09 2018 at 10:55):
that already happens in the IG. So what are you talking about?
Lloyd McKenzie (Sep 09 2018 at 15:19):
Where (and how) does the current IG tooling add "page" entries for each artifact to the IG? (I'm not talking about pages being generated, I'm talking about pages being listed in the IG artifact to use in generating table of contents, section numbers, breadcrumbs, etc.
Grahame Grieve (Sep 09 2018 at 20:51):
if you ask it to, it will automatically add pages to the ig. but you're once again talking about manual decisions. Which we said we wanted mastered in the IG. But you're not on board with that....
Lloyd McKenzie (Sep 09 2018 at 20:54):
I'm on board with eliminating an ig.json maintained by IG authors. Everything that the author maintains about how an IG should publish should be in the IG instance. However, a lot of the guidance about how an IG should publish won't be maintained in the authored IG - it'll be in the framework/template that the IG publishes with. Maintaining that information in an IG instance seems wierd to me.
Grahame Grieve (Sep 09 2018 at 20:55):
how can the template know what pages exist?
Lloyd McKenzie (Sep 09 2018 at 21:02):
When I author an IG, I create a page for the index and a few other descriptive things and a placer-holder "generated" page for artifacts. The pre-processing step then generates pages for each artifact as "children" of the artifacts page, strips out declared artifacts that are being grabbed from spreadsheets, strips out artifacts from other versions and spits out the resulting IG. A separate process spits out the config.json file which hard-codes most of the rendering stuff and goes through each resource and decides what templates it's going to use based on whether it's a profile, extension or example, true value set, example value set, etc. In the new model, those processes would merge because all of that information would be in the resulting IG. The publication process then gets kicked off for the generated IG.
Lloyd McKenzie (Sep 09 2018 at 21:03):
It's not ok for IG authors to have to create a page for each resource in the right place or to specify configuration stuff for each resource. That has to be driven by the framework - both to get consistent publications and so the IG authors don't hate us.
Grahame Grieve (Sep 09 2018 at 21:08):
what about this pre-processing step changes the IG resource?
Lloyd McKenzie (Sep 09 2018 at 21:19):
It adds pages beneath the "artifacts" page, it removes resources defined by the spreadsheets referenced by extension (because including both artifact reference and spreadsheet makes the build blow up) and strips "other version" stuff plus (in the new world) it will inject a bunch of configuration parameter stuff.
Grahame Grieve (Sep 09 2018 at 21:20):
It adds pages beneath the "artifacts" page
Why? if that's not the master, why do that?
Grahame Grieve (Sep 09 2018 at 21:20):
it removes resources defined by the spreadsheets referenced by extension
I don't even understand what this is
Grahame Grieve (Sep 09 2018 at 21:21):
strips "other version" stuff
I don't know what this is either
Grahame Grieve (Sep 09 2018 at 21:21):
a bunch of configuration parameter stuff.
What parameters?
Lloyd McKenzie (Sep 09 2018 at 21:21):
Let's work with an example
Lloyd McKenzie (Sep 09 2018 at 21:21):
Here's my "source" IG for CRD:
Lloyd McKenzie (Sep 09 2018 at 21:22):
Lloyd McKenzie (Sep 09 2018 at 21:23):
Here's the "generated" R4 davinci-crd.xml and R3 davinci-crd.xml IGs and R4 davinci-crd.json and R3 davinci-crdSTU3.json config files
Lloyd McKenzie (Sep 09 2018 at 21:25):
You can see all of the pages inserted into the IG. You can see the resources that got stripped. You can see all of the configuration information that is automatically determined by the framework. The only parts of that the IG author has any control over are specified in an ini file: properties.txt
Lloyd McKenzie (Sep 09 2018 at 21:25):
Everything else is driven by the framework.
Lloyd McKenzie (Sep 09 2018 at 21:26):
My hope/expectation for the new process is that the framework comes to be maintained by a separate project so all you need to do is point to it. That will make it easier to maintain and keep consistent across projects.
Grahame Grieve (Sep 10 2018 at 06:02):
so what I can see is that it's simply no use trying to talk about this in any productive way until we can figure out the version issue
Grahame Grieve (Sep 10 2018 at 06:04):
comparing the2 IGs, it doesn't really make sense to source them from one file - they have very little overlap. but since we can't have any rational discussion without you obsessing about single sourcing them, and being able to pre-process them, ok:
Grahame Grieve (Sep 10 2018 at 06:04):
I think the current arrangement for supporting 2 different versions is a dreadful hack. It would be a lot simpler simply to publish a single implementation guide.
Grahame Grieve (Sep 10 2018 at 06:06):
the existing IG publisher could relatively straightforwardly be adapted to be able to produce conformance resources that have a specific FHIR version. So you could have r2, r3 profiles and r4 profiles.
Grahame Grieve (Sep 10 2018 at 06:06):
there'd be some trickery to making that happen, but I can't think of anything that would actually be broken.
Grahame Grieve (Sep 10 2018 at 06:08):
but it would require some specification support:
- changes to the IG resource so you could say that the IG covers more than one versions
- and also so that you could say, for each conformance resource, which version(s) it applies to (value set etc can cross versions)
- and we'd have to change the package spec so that a single package could handle multiple versions
Grahame Grieve (Sep 10 2018 at 06:09):
I guess, for me, the question is: do we want to make multiple version IGs a core feature, or do we want to push it off to the side, and complicate all the build tools
Sean McIlvenna (Sep 10 2018 at 06:51):
@Grahame Grieve
"these are really big extensions and really lots of work for me. WHy are you strongly against it? the IG will be a mess. We made a lot of changes"
I don't disagree that these are big extensions. I'm not trivializing the work. But, having STU3 implementers use R4 instead of STU3 due to core FHIR tooling doesn't seem right either. If an implementer is using tooling that only supports STU3, there is no way for them to make an ImplementationGuide that will work with the IG publisher. I'd almost rather preserve the ig.json file for R3 and use ImplementationGuide for R4. Although, I don't like that solution, either. Extensions are the "right" approach in my mind... albeit not the easier approach.
Lloyd McKenzie (Sep 10 2018 at 14:52):
Multi-version IGs seem like a likely outcome of implementers supporting multiple versions or of the community migrating versions at different speeds. The complexity needs to live somewhere, so the question is "where should it live". I'd prefer it in the authorship rather than for the consumers and I'd prefer it in the tooling rather than for the authors.
Lloyd McKenzie (Sep 10 2018 at 14:54):
Supporting R3 IG + extensions doesn't seem like a super-huge deal. Is there something that would be involved other than updating the R3->R4/R4->R3 conversion code to use standard extensions? In principle, we're committed to creating the extensions for all resources anyhow. And under the covers, the tool always runs with latest draft...
Eric Haas (Sep 10 2018 at 22:40):
(I inadvertantly clicked on the thumbs up. so I am negating it)
Eric Haas (Sep 10 2018 at 22:41):
The R3 IG + extension is extra work.. I'd rather keep ig.json than do that.
Grahame Grieve (Sep 10 2018 at 22:41):
I would expect that this would be 3-4 days work for me over time
Bryn Rhodes (Sep 10 2018 at 23:10):
So does the tooling right now recreate the newig.json and xml files every time?
Grahame Grieve (Sep 10 2018 at 23:39):
yes for now
Eric Haas (Sep 11 2018 at 01:22):
where are newig.json and xml files I don't see them in the output or my root directory?
Grahame Grieve (Sep 11 2018 at 01:27):
created in the first nominated resource directory in the source when you run the current IG publisher
Eric Haas (Sep 11 2018 at 01:32):
not there? Is this for all FHIR versions? or before 3.6.0?
Grahame Grieve (Sep 11 2018 at 01:34):
all versions
Grahame Grieve (Sep 11 2018 at 01:34):
what version are you running?
Grahame Grieve (Sep 11 2018 at 01:35):
so you have a ig.ini in the root directory next to the ig.json?
Lloyd McKenzie (Sep 11 2018 at 02:59):
What happens if your root ig has a redirect - are you generating for both?
Grahame Grieve (Sep 11 2018 at 05:31):
I don't know what that means
Lloyd McKenzie (Sep 11 2018 at 13:13):
I have an ig.json that redirects to a different ig.json - and I'm not getting the generated files.
Lloyd McKenzie (Sep 11 2018 at 13:13):
I.e. no "new" ig.json
Eric Haas (Sep 11 2018 at 16:27):
@Grahame Grieve I'm running 3.6.0 on my test IG using Mac OSx. There is no fhir.ini in the root. see screenshot of my directory after running the build. Screen-Shot-2018-09-11-at-8.45.05-AM.png
Grahame Grieve (Sep 11 2018 at 21:50):
do you see this in your log: exception generating new IG
Eric Haas (Sep 11 2018 at 23:19):
yes ( I looked for something like this before but searched on 'error')
Contacting Build Server... (01.0500sec)
... done (04.0612sec)
Checking hl7.fhir.core-current currency (04.0650sec)
... ok: is 1, must be 1 (04.0650sec)
Load hl7.fhir.core-current package from /Users/ehaas/.fhir/packages/hl7.fhir.core#current (04.0650sec)
exception generating new IG
java.lang.Exception: Unsupported version 3.6.0
at org.hl7.fhir.igtools.publisher.ConfigFileConverter.parse(ConfigFileConverter.java:277)
at org.hl7.fhir.igtools.publisher.ConfigFileConverter.convert(ConfigFileConverter.java:92)
at org.hl7.fhir.igtools.publisher.Publisher.initialize(Publisher.java:1046)
at org.hl7.fhir.igtools.publisher.Publisher.execute(Publisher.java:500)
at org.hl7.fhir.igtools.publisher.Publisher.main(Publisher.java:4865)
Grahame Grieve (Sep 11 2018 at 23:35):
you're not using the very latest IG publisher then
Eric Haas (Sep 12 2018 at 00:50):
I am now and same log file message. Does it work on Mac?
Jens Villadsen (Jan 16 2019 at 07:38):
1) I'm about to deep dive into making an IG, and I've got some samples running. Should I consider not using the control file 'ig.json'? If that's the case, how is the publisher then controlled? command line arguments?
2) I probably need to have a look at the source code for the publisher. Where is that located?
3) The documentation mentions swagger (http://wiki.hl7.org/index.php?title=IG_Publisher_Documentation#Swagger) but adding the swagger conf to my ig control file seems to have very little impact. Where's the output of the swagger configuration intended to be found running with 'single' option?
@Grahame Grieve
Oliver Egger (Jan 16 2019 at 13:41):
@Jens Villadsen for 2: https://github.com/HL7/fhir/tree/master/tools/java/org.hl7.fhir.igtools
Eric Haas (Jan 16 2019 at 13:59):
3: in output as [capability statement].openapi.json
Lloyd McKenzie (Jan 16 2019 at 14:01):
1. The content is moving into ImplementationGuide.definition.parameter. I think that's supposed to work now, but I don't know that anyone other than Grahame's tests have tried to use it
Eric Haas (Jan 16 2019 at 14:01):
1: still uses ig.json config file but changing over to ImplementationGuide resource + ig.ini very soon
so docs here are still valid...
http://wiki.hl7.org/index.php?title=IG_Publisher_Documentation
Eric Haas (Jan 16 2019 at 14:02):
@GG can we switch to the IG resource now, I have had my tooling in place for a while and ready to test it out
Lloyd McKenzie (Jan 16 2019 at 14:06):
I don't think it's going to be a hard-switch. I think we're going to work at creating a test IG and test template thingy. Once that works, we'll give projects time to migrate, then we'll disable the old way.
Jens Villadsen (Jan 16 2019 at 14:06):
I don't think it's going to be a hard-switch. I think we're going to work at creating a test IG and test template thingy. Once that works, we'll give projects time to migrate, then we'll disable the old way.
whats the timeline we're talking here ...?
Eric Haas (Jan 16 2019 at 14:07):
month(s)??
Jens Villadsen (Jan 16 2019 at 14:08):
3: in output as [capability statement].openapi.json
Right ... as stated in the code ... ;)
Lloyd McKenzie (Jan 16 2019 at 14:10):
May ballot will use the current frameworks. Projects will be able to switch sometime after ballot opens and will be expected to switch for Sept. ballot
Jens Villadsen (Jan 17 2019 at 10:50):
1: still uses ig.json config file but changing over to ImplementationGuide resource + ig.ini very soon
so docs here are still valid...
http://wiki.hl7.org/index.php?title=IG_Publisher_Documentation
So what if resources are specified in both the control file and the ImplementationGuide resource ... - what can I expect from that?
Lloyd McKenzie (Jan 17 2019 at 13:39):
That's normal - and is one of the reasons to get rid of the IG.json. There's lots of redundancy
Grahame Grieve (Jan 18 2019 at 11:44):
the build doesn't work without ig.json right now. it's work in progress
Last updated: Apr 12 2022 at 19:14 UTC
