Stream: IG creation
Topic: IG Templating to Solve All Problems
Gino Canessa (Sep 22 2021 at 15:10):
Hi all, I was reflecting on the discussions from the last FHIR-I WGM Quarter about IG Templating... I feel like there's a lot of contention around the IG template because it's trying to be everything to everyone. There were a lot of times that people spoke of "how they want to consume data" vs. "how they want to present data". It made me think about how differently even a single person would consume IGs as their experience grows (e.g., from first IG they read to 'oh, another version', etc.).
Is it realistic to try and make a single template that will be suitable for every use and every experience level across every discipline (e.g., expert clinical and novice tech, novice clinical and expert tech, etc.)? Especially given constraints like 'browsers without JavaScript', I feel like we're chasing an impossible target.
Is there any way we can change that? For example, could we make a 'basic' template that serves as the source of truth, works universally, etc., but then have other views into the same data? Can we build tools that have things like dynamic tables, filters, etc. that are useful across many IGs, regardless of who published them?
To be clear, I don't have any proposals or answers right now, but I wanted to ask the question =).
Grahame Grieve (Sep 22 2021 at 15:25):
we have one publication, so one template. That doesn't mean we shouldn't have good views that are focused on particular users. There are some IGs that do that
Lloyd McKenzie (Sep 22 2021 at 15:38):
Other organizations can have vastly different templates, but our intention is to have a consistent look & feel for at least HL7 IGs. (And many organizations find that having consistency with the way HL7 publishes is helpful for their own readers.
Gino Canessa (Sep 22 2021 at 15:47):
Yes, having one template for one publication makes sense. But, the fact that we can't do something like add toggleable components makes a lot of UI stories hard. On top of that, if we do add a lot of usability, etc., it will not apply to anything published earlier in time.
So if Jan 1 we have a great new format, only IGs that are published after that be able to use it. Given that publication is a lot of work, IGs published in December would just be out of luck, potentially for a long time.
I'm wondering if we can separate the publication aspect from the view aspect to allow for more development. I don't know if it's possible / realistic / desired / etc.
Yes, other orgs have different templates now too. I don't see any way of changing that, but it does work in direct opposition to the desire for consistency. If I need to use 5 IGs, I likely don't care who developed or published them. I need the same info from each and would like to be able to view them in a consistent way. Obviously that only applies to something that uses the HL7 tooling, but I think that's a venn diagram with a lot of overlap.
Lloyd McKenzie (Sep 22 2021 at 15:55):
If someone wants to put out a technical correction that just moves to the newest templates/publisher, that's a relatively low lift. (Though you may run into new validation issues you might want to address, FMG probably wouldn't force you to.)
Lloyd McKenzie (Sep 22 2021 at 15:56):
I'm expecting many of the Da Vinci IGs to do technical corrections to take advantage of the next set of IG enhancements once they're done and tested (unless they already have a new release coming up for other reasons).
Grahame Grieve (Sep 22 2021 at 16:00):
specs are a standard, not a UI. That imposes some real - important - constraints
John Moehrke (Sep 22 2021 at 16:03):
creative UI may feel satisfying to the author, but it is very frustrating to the reader.
John Moehrke (Sep 22 2021 at 16:04):
also. as a standard we need to meet readability criteria of the whole community.
Gino Canessa (Sep 22 2021 at 16:17):
All fair points. I see the specs / standard as data and want to think about how to separate concerns.
For example (something random that popped into my head writing this), but have the templates gone though things like accessibility review? I don't see a way to change the UI color / contrast / etc. Those types of issues don't exist on a 'publication', since the output is a single 'printed' version (e.g., the PDFs that are generated from the specs). But for people consuming data on the web, those are table stakes.
I don't think we want a system that forces everyone to 'republish' each time we address something like that, but I don't think we should be ignoring those tasks either.
David Pyke (Sep 22 2021 at 16:22):
I would be afraid to see the output from a US section 508 review of a spec. I think we'd have a lot of work to do to make this accessible to screen readers and such
Gino Canessa (Sep 22 2021 at 16:22):
Grahame Grieve said:
specs are a standard, not a UI. That imposes some real - important - constraints
A lot of requests I see for the IG template are because it is a UI (or rather, the output is).
Gino Canessa (Sep 22 2021 at 16:23):
David Pyke said:
I would be afraid to see the output from a US section 508 review of a spec. I think we'd have a lot of work to do to make this accessible to screen readers and such
Yep - I was starting with lower-hanging fruit as an example =)
Lloyd McKenzie (Sep 22 2021 at 16:24):
We're waiting for someone who needs US section 508 adherence to fund that initial pass-through. We absolutely recognize it's needed, but it's a significant enough effort, it's very hard to do on a volunteer basis. (Once the work is done, checking new changes to make sure they don't break our compliance would be added to the set of requirements.)
Grahame Grieve (Sep 22 2021 at 16:28):
there is work around accessibility. but some of it's crazy... every page has to have a H1, for instance. Not going to happen under my watch
Grahame Grieve (Sep 22 2021 at 16:29):
other things, totally reasonable
Gino Canessa (Sep 22 2021 at 16:30):
So, to bring it full circle - is it worthwhile to look at a process where someone who wants an H1 on every page can build that without re-inventing the entire publishing pipeline and running it all themselves?
Lloyd McKenzie (Sep 22 2021 at 16:49):
They can create their own templates. SHould be able to run off the same publisher.
Gino Canessa (Sep 22 2021 at 17:21):
Fair - so to get an accessible version of <random IG I didn't author>, even if there's an accessible template already, I need to: find the source for the IG, run IG publisher (which hopefully doesn't introduce errors moving between templates), and use that site.
I'd like to explore if we could separate the data from presentation instead (e.g., use this app/site/SPA/etc. to view an IG).
John Moehrke (Sep 22 2021 at 17:29):
@Elliot Silver did do a review using tools that evaluate accessibility. Unfortunately tools are not humans doing the evaluaton
Elliot Silver (Sep 22 2021 at 17:31):
“Review” is a bit of an exaggeration. I ran a page from a couple of IGs though a couple of online tools to see what they would spit out.
Gino Canessa (Sep 22 2021 at 17:38):
I'm using accessibility as an example... I think that there are things that can be done in the presentation layer that we don't do right now, in part, because we're using a combined data and presentation.
As another example, I've heard a few times recently about using spreadsheets to consume IGs. I don't think the people are [necessarily] using excel files in their own workflows, just that Excel provides a useful view into the data. It'd be great if someone could build a tool that gives the desired view by navigating to the IGs.
Lloyd McKenzie (Sep 22 2021 at 18:12):
The ability to render IG content differently is certainly more possible than it has been in the past with v2 or CDA, but it's certainly going to require some effort and isn't guaranteed to work, depending on what you're trying to do.
Jose Costa Teixeira (Sep 22 2021 at 18:26):
My understanding is that we have
a) computable content that can be UI'ed.
b) one reference IG UI flavour.
We are never get the infinite variety of flavours that would make everyone happy all the time (and user preferences do change with time and experience).
Jose Costa Teixeira (Sep 22 2021 at 18:29):
The things I like to give more attention to are in those lines.
Jose Costa Teixeira (Sep 22 2021 at 18:31):
It is possible to drastically change the look and feel of an IG (the content part will mostly remain the same though)
Jose Costa Teixeira (Sep 22 2021 at 18:36):
But we should not chase all those flavours.
Gino Canessa (Sep 22 2021 at 18:36):
Lloyd: yes, I'm wondering if it worth exploring or not.
Jose: yes and no.. Today, people can build their own templates, but it requires using IG publisher from source. If I wanted to create (for example) a fully accessible template, I would need to rebuild all of the IGs from source in order to use it. Especially for IGs that are "old"*, this isn't feasible.
*"old" meaning published under some prior version of IG publisher that includes breaking changes backwards from current.. could be 1 or 10.
Jose Costa Teixeira (Sep 22 2021 at 18:38):
@Gino Canessa
It'd be great if someone could build a tool that gives the desired view by navigating to the IGs.
What do you mean with this?
Gino Canessa (Sep 22 2021 at 18:38):
I specifically don't want to chase all those flavors (because it's not achievable). I'm wondering if there's something IG publisher can output (state before Jekyll?) that could be 'published' and available for other presentation layers to consume.
Jose Costa Teixeira (Sep 22 2021 at 18:38):
All the pre-jekyll content is there
Jose Costa Teixeira (Sep 22 2021 at 18:40):
(I'm not saying we should not use other presentation layers - just that it will be an infinite and always changing set of features - some of which we may onboard, of course)
Jose Costa Teixeira (Sep 22 2021 at 18:42):
For example it seems reasonable that a PDF version could be produced not from Jekyll but in parallel with Jekyll
Jose Costa Teixeira (Sep 22 2021 at 18:42):
Is that similar to what you are thinking, Gino?
Jens Villadsen (Sep 22 2021 at 18:43):
Gino Canessa - that would also open up for having a bit more control over the process meaning applications not calling other applications - having that call embedded that is
Jens Villadsen (Sep 22 2021 at 18:43):
(I mean the publisher invoking jekyll and all)
Gino Canessa (Sep 22 2021 at 18:45):
Jose Costa Teixeira said:
Gino Canessa
It'd be great if someone could build a tool that gives the desired view by navigating to the IGs.
What do you mean with this?
I think this came up specifically in Q1 today - someone (might have been you? I can't remember) wanted a filterable table view of the elements that supports reordering of columns, sorting, etc.
Right now, the answer is download spreadsheets.
If I publish an IG and want that, the answer is build a template, which I cannot use if my IG is official.
Or, I could build a UI* that offered that functionality. Today, I could either download the spreadsheets myself (which do not contain all the content of an IG), or try to scrape the info off the published sites. Those publications will have differences even with a single template over time, much more if you want to support 'non-HL7' templates, so scraping is ...frustrating? to get right and maintain.
Is there a space that allows HL7 to meet their concerns, hopefully reduce burden / workload, and open things up for others? :shrug:
*UI: perhaps doing too much work by saying 'UI' here, I'm thinking of presentation layer in general.
Gino Canessa (Sep 22 2021 at 18:48):
For clarification, if I were doing something like this my hope would be that I could build a tool that does not need to rebuild the IG to consume it.
Jens Villadsen (Sep 22 2021 at 18:49):
you're advocating for a "pipe-and-filter" design?
Gino Canessa (Sep 22 2021 at 18:53):
Well, I started this thread specifically without a proposal or answer in mind, so I'm not advocating anything beyond having the discussion =).
In reality, that's one approach of many possibilities. Some number of people would need to look at this seriously and spend time weighing the pros and cons before I'd actually push for something.
Right now I'm curious if something like this is desirable and achievable.
Jose Costa Teixeira (Sep 22 2021 at 18:53):
When the IG publisher runs it generates a bunch of jsons and xmls in the temp folder
Jose Costa Teixeira (Sep 22 2021 at 18:54):
So it is possible to do something with the template (template is not only presentation but pre- and post-processing) to reuse that content
Jose Costa Teixeira (Sep 22 2021 at 18:56):
I have gone through the steep-ish curve of learning some of those mechanisms, and if we want to have a workshop on that I'm in - for me to learn more or share what I can
Jose Costa Teixeira (Sep 22 2021 at 18:58):
For example, to talk about adding a JQuery datatable to Artifacts page
Jose Costa Teixeira (Sep 22 2021 at 18:58):
:)
Lloyd McKenzie (Sep 22 2021 at 19:01):
Templates primarily drive outputs, but they also drive inputs. It's easy for someone to swap templates if they expect the same inputs, but it doesn't work well if the inputs differ. So what you really want is a set of templates that are intended to work together. We've talked about having a template for PDF generation. We've held off because we think PDFs suck and don't want to encourage their use, but at some point someone will probably force us to because of some regulatory requirement or other. If someone wants to define their own template for their own specific rendering requirements and align it with our current inputs and re-run the publisher pointing to the new template, they can. We don't need to do anything to enable that. (But I also don't think we want to advertise it as desirable as we'd prefer that almost everyone look at specs from a consistent rendering and we also don't have a ton of bandwidth to support those who author custom templates, so we'd prefer if the volume of those was smaller rather than larger.)
Gino Canessa (Sep 22 2021 at 19:39):
Yes, that's why I want to discuss. If something like this were to happen, it would need to be defined in such a way that it was generally stable. I'm fairly confident it's possible since we have definitional resources that are normative, but there's a lot of content that falls outside of those.
If the process starts by pulling files from a temp processing directory, the result will always be a fragile workflow that arbitrarily breaks. There are times when that's the best option, but I don't think this is one, and I would be very hesitant to go that route.
One option would be trying to split apart IG publisher (as mentioned earlier) into a part that does all of the required validation, assembly, etc. and a part that translates that into presentation. Another would be to build a separate process that starts from the same source, skips all the validation, etc. (since that's done by IG publisher), and then outputs the desired files*.
I don't know much about the internals of IG publisher and the publication pipeline, so I'm not in a position to suggest the best approach... I'm just the idea guy ;-).
In all seriousness though, I don't see something like this working unless it's generally wanted and supported. It's why I brought it up for discussion instead of just adding it to my 'fun projects to work on' list.
*desired files: no idea what this looks like yet either, just spit-balling.
Lloyd McKenzie (Sep 22 2021 at 19:58):
The IG templates and publisher interact at several points. In the first pass, the templates update the IG to include parameters that affect IG publication, including where to look for input files. In the second pass, all resources have been identified, but information about them can be supplemented, the IG can be tweaked to do things like insert additional pages, and some data files that subsequently drive generation are produced. Then the publisher validates everything and generates whatever fragments were requested with the naming conventions requested. Templates have a chance to do another bit of messing around (though I'm not sure anyone does anything at this step), then Jekyll is run, then the templates can do a bit more stuff (generally QA), then final QA and packaging is done. It's going to be pretty hard to disentangle all of this, and I'm not sure we'd want to. I think the best approach is to simply create 'aligned' templates and monitor the evolution of the 'base' template to maintain that alignment (in terms of expecting source files to exist in certain places, take advantage of publisher-generated fragments, etc.).
Gino Canessa (Sep 22 2021 at 20:30):
Sure. Based on what you're saying (and in a non-binding way =), would it be better to create a separate template to run in a separate pass, or add functionality to it to export files (e.g., how it exports spreadsheets, zip files, etc.)?
I hate the idea of two passes, but since there are other templates in production already I don't know that tacking on there would be better.
Jose Costa Teixeira (Sep 22 2021 at 22:16):
It is possible to add functionality to the template
Jose Costa Teixeira (Sep 22 2021 at 22:27):
So it would be one pass, I guess. But it's hard to say without a specific topic or feature
Grahame Grieve (Sep 22 2021 at 23:55):
We’ve already had to generate some PDFs for regulatory purposes. We used a website -> PDF tool. I investigated automatically producing PDFs but it was difficult tooling wise, super messy template wise, and not worth the iWork when there’s website to PDF tools
Grahame Grieve (Sep 22 2021 at 23:57):
As for the wider issues Gino is dreaming about: Gino, the entire knowledge that the IG publisher has is embedded in processible form in the output, and the eco-system of published packages is entirely open. There’s nothing it could do that it hasn’t already done to make it possible to write your own IG viewer. Also, the IG publisher can be fed completed html, and it just qa’s it. So if you think there’s mileage in writing a viewer, I don’t need to do anything to make it possible for you to do it
Grahame Grieve (Sep 23 2021 at 00:07):
So knock yourself out.
Eirik Myklebost (Sep 23 2021 at 00:40):
Gino Canessa said:
One option would be trying to split apart IG publisher (as mentioned earlier) into a part that does all of the required validation, assembly, etc. and a part that translates that into presentation. Another would be to build a separate process that starts from the same source, skips all the validation, etc. (since that's done by IG publisher), and then outputs the desired files*.
I'm sure there are good reasons behind this but I've also felt that the IG Publisher is kind of doing too much. From my observations the IG Publisher is:
- Compiling FSH files using SUSHI.
- Validating resources
- Generating the ImplementationGuide resource.
- Generating examples ++ ?
- Generating snapshots and narratives?
- Creating FHIR Package (NPM)
- Generating HTML page using templating.
There are times while developing locally or during a CI\CD pipeline that I would like more control over what is done, primarily to speed things up.
Some use case examples:
- Generate the HTML while working locally without caring about the validation or FHIR package.
- Only validate the IG.
- Building the FHIR Package without the presentation.
- Skipping the html generating and instead use some other tool\framework to create the presentation: hugo, next.js, readthedocs or whatever is in fashion.
Jose Costa Teixeira (Sep 23 2021 at 07:29):
The publisher is not generating the IG resource or examples. And it's sushi that compiles the fsh
Jose Costa Teixeira (Sep 23 2021 at 07:46):
As for radical changes, they should be based on a need and consider the entire ecosystem.
Jose Costa Teixeira (Sep 23 2021 at 07:52):
(and I don't see a need for a big radical change, given what we can do. I may be biased based on my work with the templates)
Grahame Grieve (Sep 23 2021 at 09:40):
Grahame Grieve said:
here are times while developing locally or during a CI\CD pipeline that I would like more control over what is done, primarily to speed things up.
Some use case examples:Generate the HTML while working locally without caring about the validation or FHIR package.
Only validate the IG.
Building the FHIR Package without the presentation.
Skipping the html generating and instead use some other tool\framework to create the presentation:the functions are not that separated - the IG is only really validated when the HTML is generated, though 98% of the validation happens in the validation step. But lots of meta is lined up in the validation step that would make the generated html not quite perfectly correct if it didn't happen. But it might be useful to do one and not the other for editorial workflow convenience.
Producing the package... a fraction of the time - seconds max.
As for generating using some other tool... whatever you generate with, you'll need the fragments that the IG publisher generates for Jekyll. Actually, Jekyll is pluggable in the IG publisher - we expected people to want other things, but no one ever has. - it's kind of a swiss army knife thing, and people just keep asking for more gems to be plugged into it. But the IG publisher isn't tied to Jekyll. You just need the fragments
Eirik Myklebost (Sep 23 2021 at 10:28):
Grahame Grieve said:
Producing the package... a fraction of the time - seconds max.
Would it be possible to instruct the IG Publisher to produce the package without the HTML?
Grahame Grieve said:
Actually, Jekyll is pluggable in the IG publisher - we expected people to want other things, but no one ever has. - it's kind of a swiss army knife thing, and people just keep asking for more gems to be plugged into it. But the IG publisher isn't tied to Jekyll. You just need the fragments
This is pretty cool. Is it possible to make the IG publisher only generate the fragments/artifacts and skip the Jekyll step? The artifacts can then be used as input in some other tool and the IG Publisher can be used without installing Ruby+Jekyll.
Eric Haas (Sep 23 2021 at 14:19):
delete
Gino Canessa (Sep 23 2021 at 16:49):
Grahame Grieve said:
So knock yourself out.
With an offer like that? I'm good.
I started this thread because from what I can see, IG authors and consumers want features that HL7 either can't (e.g., publication process/restrictions) or won't (e.g., not an area of interest/focus) provide - for example, functionality that needs JavaScript. Are there ways around it? Sure:
- new site that wraps published HTML
- build pipeline that uses published content to make a new site
- browser extension for advanced formatting / UI
- a separate application that handles advanced viewing
- etc.
But packages don't include narrative content, and building a UI on top of presentation-formatted HTML with no contract isn't a good use of time (or at least my time, other people's mileage may vary). Any change made in the template or lower could break it, and if people working in those layers aren't even aware of what is a breaking change :shrug:
As of right now, my understanding is that narrative is only accessible via a single presentation layer that also needs to be PDF printable and cannot include any JavaScript. Additionally, that presentation layer is fixed at the time of publication and only updated if a new version is published, which requires a decent (if not significant) effort from both authors and HL7. So if someone did the work to update the template for accessibility (leaving aside if it's possible within the constraints), it would be available on 0% of all published specs when the work was done.
What is the open space here? Can we make a template that has JS and modern features, but falls back to a simpler rendering? Can we add in 'advanced' features that only work if you do support JS? Can we have alternative templates on official HL7 IGs? Is it restricted to what we build on artifacts and have to exclude narrative? I have no idea, hence this thread.
To be clear, I don't think this is trivial, nor do I think it's something that can be addressed quickly or in isolation. So my "dream", if that's what we're calling it, was to see if my I was correctly identifying the issue, and if so open discussion. Apologies if I wasn't clear enough on that up front. I don't know what are hard constraints (e.g., publication requirements) vs. what different parties would like to see. I'm not even all that familiar with IG Publisher and the templates (maybe I'd be willing to label myself a novice on publisher, but not even that on the templates =).
For example, and to catch up on the thread, it looks like it might be possible to have the publisher export some form of 'data' files that can be consumed by generic UIs. Great! Is it something that could be agreed on? Would HL7 consider adding those files and making them available alongside the current pages? Could HL7 even do that? If so, could we start including them so that they're available when the other layers exist? If not, what kind of options are open that don't violate the rules (e.g., copyright)? What if the UI was under HL7 purview? What if it wasn't? (I'm not interested in building a site that will just be taken down)
I have no idea how much work is needed here, other than my gut saying: a lot*. However, I'm also not asking you (HL7 in general, Grahame in particular) to do any right now. But to have any hope of success, there would need to be agreement on what's allowed, what things are stable and what can change, etc. If there's no interest in exploring that, then (to me) question answered and move on.
*really, my gut is saying "a decent but not insurmountable amount of up front work, plus a long-term maintenance commitment", but that doesn't flow inline with the text =)
Grahame Grieve (Sep 23 2021 at 16:58):
so you have a number of false assumptions here. The pages do include JS, though I do use it very carefully, to ensure that that presented pages meet ANSI rules, which have a lot to say about not hiding things from people. So we can use JS to build things, but they have to somehow different so that they don't create problems for ANSI. A non-normative rendering would be fine, for instance, but you're right, I'm not doing it.
The packages do include narrative content. Not all the narrative that goes into an IG, but that is residually available in the temp folder if someone wanted to build something new with it.
If we fix the template for accessibility, then that will become de facto in all newly published IGs
Gino Canessa (Sep 23 2021 at 19:05):
re: false assumptions: Fair. I assumed those as a moderately informed participant, so thank you for clarifying =).
re: JS: Good to know where the rules are coming from (again, I assumed as much, but didn't know which organization was imposing them). I don't know what is possible here, since I many of the usability requests I've heard are specifically around hiding things, but it's discoverable.
Re: narrative: Sure, but I prefer not to build software based on files I find leftover in a temp folder. If that has the necessary content (I haven't looked), I would prefer it gets documented and labeled so that it wouldn't arbitrarily break. Also, while I'll grant that 'narrative' includes content inside resources (e.g., definitions), I was trying to refer to narrative that lives outside of resource definitions (e.g., page content).
re: publication time: Yes - which means that sometime between that day and never, a version of an IG would be available with the new template. For a personal example, once the Subscription R4 Backport is published I hope to never publish another version of it. Any new work will be done on R5, which already includes (or will when I merge it) that same content, and stability of the implementation is one of the top concerns. At the same time, I expect R4 will be around for many years and would prefer to have the view 'look' current. Maybe that's just a penalty/tax I need to pay as an IG author, but it does assume that every IG has a long-term custodian (which is another topic anyway).
Grahame Grieve (Sep 23 2021 at 19:38):
well, it's a balloted standard. I can't change it later. Even just changing the look and feel
Gino Canessa (Sep 23 2021 at 20:14):
Blah. This is quite useful, but blah. So it sounds like most of the 'interesting' requests would have to be contained in a non-normative view into the content.
- Is that something that would be interesting / possible for HL7 (again, not a request - just feeling out the space)?
- If not, is there conceivably a space that wouldn't go against HL7 policy / desires (e.g., I don't think HL7 wants the main way people consume their content to be something external)?
Lloyd McKenzie (Sep 23 2021 at 23:26):
There's nothing to stop anyone from taking any FHIR specification and re-publishing it using whatever views they like. The base standard could even be updated with a technical correction to point to those alternative views.
Grahame Grieve (Sep 28 2021 at 04:21):
well, the next release of the IG publisher will support two new parameters for editorial convenience:
- -validation-off - don't do any validation
- -generation-off - don't generate the HTML output
in both cases, the file qa.html is still produced but some kind of errors will be omitted from it because resources aren't being validated (obviously) or html isn't being generated, which causes other errors to be picked up. but they may be useful for editors.
Note, though: the package output is unreliable in these cases as an input to some other tool chain. If you're going to do something more with the output, you really need the full generation. I know in theory the HTML shouldn't matter, but, well, I only generate some things I need during the HTML generation. Sure, I could spend some time chasing all of them down, but that would slow down the run if you're not generating HTML... so it seems like something not worth doing to me
-
Lloyd McKenzie (Sep 28 2021 at 14:23):
Does the QA contain a warning indicating that those parameters have been used - so it's clear that the output might be suspect and so FMG can ensure that the build has been run properly?
Last updated: Apr 12 2022 at 19:14 UTC
