Stream: ig publishing requirements
Topic: bring back front matter!
Eric Haas (Nov 09 2021 at 21:25):
On future call, I would like to discuss the topic of bringing back front matter to source pages. It is useful for defining local page variables and all kinds of stuff for Jekyll, and for direct editing in external markdown editors like Hackmd which use front matter too -- very useful.
Grahame Grieve (Nov 09 2021 at 23:01):
what's front matter?
Vassil Peytchev (Nov 10 2021 at 04:16):
Apparently this:
https://jekyllrb.com/docs/front-matter/
Grahame Grieve (Nov 10 2021 at 06:19):
what happened to it? I certainly didn't take it away
Eric Haas (Nov 10 2021 at 20:22):
LLoyd did since it messes with xhtml validation.
Lloyd McKenzie (Nov 10 2021 at 20:28):
There were a bunch of issues. Front matter was being used to define things like page titles - which needs to live in the ImplementationGuide. Also, markdown needed to be injected in the XHTML to get the headers/footers, etc. to work properly. Also, the whole point of markdown was to have something super simple for authors to write. As soon as we start requiring a bunch of complex stuff up front, then the value proposition for supporting markdown diminishes. Also, we don't want a ton of custom hacking that makes IGs appear different - our objective is to get IGs to appear more consistent.
None of that means that we can't re-evaluate if/how frontmatter (or at least certain types of it) could be supported. Happy for us to discuss on our weekly calls.
Jose Costa Teixeira (Nov 14 2021 at 09:31):
My requirements for front matter:
We are using the IG publisher ecosystem in a bunch of different specifications, not only because of the FHIR content, but because it is automated and self-validating documentation system. As such, we want the IG Publisher to be open to other structured content.
Types of content we'd add - Actor definitions, transaction definitions, requirements/criteria,...
Jose Costa Teixeira (Nov 14 2021 at 09:33):
All those are structured data objects that we could add to an IG. We can pre-process them and add them as XHTML, but we need something more than just narrative.
Jose Costa Teixeira (Nov 14 2021 at 09:34):
There seem to be 2 possible enablers for that:
- Front matter
- the newly introduced logical model instances (we can define our custom types)
Josh Mandel (Nov 14 2021 at 14:31):
(ooh, where can I learn more about the newly introduced logical models instances?)
Jose Costa Teixeira (Nov 14 2021 at 18:43):
Josh Mandel said:
(ooh, where can I learn more about the newly introduced logical models instances?)
https://chat.fhir.org/#narrow/stream/215610-shorthand/topic/InstanceOf.20a.20logical.20model.3F
https://chat.fhir.org/#narrow/stream/179252-IG-creation/topic/Logical.20Model.20Examples
Grahame Grieve (Nov 14 2021 at 18:43):
I'm totally confused by linking front-matter and logical models
Jose Costa Teixeira (Nov 14 2021 at 18:46):
they're not linked. they are the only ways i know that structured (not narrative) content can be added to an IG besides the resources
Grahame Grieve (Nov 14 2021 at 18:46):
how would you 'add structured content' using front matter?
Jose Costa Teixeira (Nov 14 2021 at 18:48):
using the jekyll templates
Jose Costa Teixeira (Nov 14 2021 at 18:51):
here's an example of the structured front matter for defining a "transaction" (don't mind the dummy content, the only important thing is that it is structured according to a model)
https://raw.githubusercontent.com/costateixeira/gendocs/main/_transactions/transaction1.md
and the rendered result
https://costateixeira.github.io/gendocs/transactions/transaction1
Grahame Grieve (Nov 14 2021 at 18:52):
my first response is wtf? Surely you'd put all that in a json file in _data?
Josh Mandel (Nov 14 2021 at 18:52):
https://github.com/costateixeira/gendocs/blob/main/_layouts/transaction.html is the logic making this happen?
Jose Costa Teixeira (Nov 14 2021 at 18:53):
yes
Jose Costa Teixeira (Nov 14 2021 at 18:54):
Grahame Grieve said:
my first response is wtf? Surely you'd put all that in a json file in _data?
yes we could, but the _data folder is not accessible to an IG author (is it?)
Jose Costa Teixeira (Nov 14 2021 at 18:55):
Unless of course, the file in the _data folder is an instance of a logical model.
Grahame Grieve (Nov 14 2021 at 18:56):
what you put in /input/pagecontent/_data will got to /temp/pages/_data
Jose Costa Teixeira (Nov 14 2021 at 19:01):
Grahame Grieve said:
what you put in /input/pagecontent/_data will got to /temp/pages/_data
ok this bit of information is very handy.
Jose Costa Teixeira (Nov 14 2021 at 19:02):
Still, I'd like to enforce some structure. For example if my data object is a transaction, i should be able to say that the transaction.id is 1..1 (and fail if it's not there)
Jose Costa Teixeira (Nov 14 2021 at 19:03):
From my side, all of this is to explore the options in preparation for a discussion.
I personally experimented with front matter, there were some challenges, and I'm seeing if the new dot connects in the picture
Lloyd McKenzie (Nov 14 2021 at 19:38):
I agree w/ Grahame that we'd be better off sticking an JSON file in _data than sticking this stuff in markdown frontmatter where it can't be properly validated at all. We could also explore using an actual logical model instance and somehow indicating that a distinct rendering template is to be applied.
Jose Costa Teixeira (Nov 14 2021 at 19:58):
Yes. Files in _data would be structured but not validated, and I presume would all be rendered with the same template
Jose Costa Teixeira (Nov 14 2021 at 19:59):
Instances could be validated and may have specific templates
Jose Costa Teixeira (Nov 14 2021 at 20:00):
My experience with front matter predates both of these alternatives. For me the logical model instance may meet the requirements
Lloyd McKenzie (Nov 14 2021 at 20:23):
You can define a template that renders things as you wish, I think. It would reference whatever _data file you've asked it to reference. If you want to use the same template with a bunch of different _data files, then we should look at logical model instances and some means of determining the appropriate template to use.
Jose Costa Teixeira (Nov 14 2021 at 21:23):
I think it may be the winning approach (for my purposes at least). I would want the template to take each of those files, figure out what template to use (based on the type), and render it. That would also allow managing those objects in a much better way than front matter buried in a file before the narrative
Eric Haas (Nov 15 2021 at 16:15):
Jose Costa Teixeira said:
Grahame Grieve said:
my first response is wtf? Surely you'd put all that in a json file in _data?
yes we could, but the _data folder is not accessible to an IG author (is it?)
Indeed it and I routinely copy the ig resource ig package list in there it just stick in it there:
├── FHIR-us-core.xml
├── README.md
├── SUSHI-GENERATED-FILES.md
├── fsh-generated
│ ├── includes
│ └── resources
├── ig.ini
├── input
│ ├── Archives
├── Drafts
├── data
│ ├── README.md
│ ├── ig.yml
│ └── package-list.yml
├── examples
│ ├── Device-defib.json
...
Eric Haas (Nov 15 2021 at 16:27):
I want the to tag and name use variables:
- I can use some other tools beside the ig publishing. for example HackMd, The front matter tags and title lets me sort and find them when pushing and pulling from the repro.
- can use page variables
Eric Haas (Dec 08 2021 at 23:38):
when can we talk about this?
Lloyd McKenzie (Dec 09 2021 at 04:05):
We could on Tuesday. I'm happy for the templates to stick extra stuff in data (possibly from a defined folder). I'm not in favor of allowing more complexity in markdown.
Eric Haas (Dec 09 2021 at 08:17):
I would like or reuse the markdown documents and the prohibition of front matter is preventing that. I don't understand what the templates have to do with it.
Lloyd McKenzie (Dec 09 2021 at 14:36):
Can you provide a concrete example of what you’re trying to accomplish?
Eric Haas (Dec 09 2021 at 20:07):
sigh.... well ...
first
Front matter is metadata written in yaml, located at the top of a file wrapped in ---'s.
Many static-site generators like Hugo, Jekyll, and Hackmd support it. I want to be to port my documents from one to the other without having to remove it and use the front matter capabilities baked right in jekyl when using the ig pub.
Lloyd McKenzie (Dec 09 2021 at 20:26):
I know what front matter is. I want to know what you're trying to accomplish with it - to see if there's a simpler mechanism that can be consistent across both XML and markdown and that also minimizes the learning curve for authors.
Lloyd McKenzie (Dec 09 2021 at 20:26):
If the issue is just that you have tools that automatically plug it in and you don't want to have to remove it to get the publisher to not bomb, we can certainly figure out a way for the templates to do that.
Grahame Grieve (Dec 09 2021 at 20:26):
why would the publisher bomb?
Lloyd McKenzie (Dec 09 2021 at 20:28):
Because the markdown gets embedded in an XHTML page inside a "process markdown" command - and I believe front matter isn't allowed in that context.
Grahame Grieve (Dec 09 2021 at 20:35):
that's a template thing then. I don't understand why a template would do that and not let jekyll process the content
Lloyd McKenzie (Dec 09 2021 at 20:52):
Because the headers and footers weren't working properly otherwise.
Grahame Grieve (Dec 09 2021 at 21:00):
how so? And is there an alternative solution?
Lloyd McKenzie (Dec 09 2021 at 21:02):
Well, the first question is to find out what's needed. Because several of the things typically done in front matter should be done in other ways in the publishing framework (defining data, specifying titles, controlling page table-of-contents, etc.)
Grahame Grieve (Dec 09 2021 at 21:23):
can we be specific?
Vassil Peytchev (Dec 09 2021 at 21:25):
And is there an alternative solution?
Adopt Astro? This is mostly tongue-in-cheek, since I haven't used it, and only glanced over the docs, but if we are looking for both flexibility, and output in HTML+CSS, having a single tool that depends on Node/npm only, might make things easier (and it has frontmatter everywhere).
Could be something to look at post R5....
Grahame Grieve (Dec 09 2021 at 21:28):
wow. in principle, we could support something other than Jekyll, but we have a huge infrastructure built on it
Vassil Peytchev (Dec 09 2021 at 21:43):
Yes, Astro would be a replacement of Jeckyll. The only reason to even look into that might be to reduce the complexity of the Java side of building the specification. If the Java part of the IG builder creates (reusable?) components, and the site builder stitches them together, that may make things easier to manage.
On the other hand, that might be possible just with Jeckyll and a reorg of the publishing process?
Sorry for the distraction, keep carrying on...
Grahame Grieve (Dec 09 2021 at 21:45):
If the Java part of the IG builder creates (reusable?) components, and the site builder stitches them together, that may make things easier to manage.
That's exactly how it works.
Vassil Peytchev (Dec 09 2021 at 22:16):
It seems to me that Jeckyll's assumption that it is building a blog might be too constraining. Are the renderers in the core library aware of the Jeckyl structure - i.e. can they build XHTML partials for the Jeckyll _include directory?
Grahame Grieve (Dec 09 2021 at 22:24):
the iG publisher produces 1000s of fragments in _include. And it produces some non-processible files of it's own in the Jekyll root. And then files as driven by the templates, which control how everything is stitched together.
Grahame Grieve (Dec 09 2021 at 22:25):
at issue here is that Lloyd has built an extensive pre-processing process into the templates that is XML orientated, and removes the ability to do some convenient things in the Jekyll layer that is usually used to drive templating in Jekyll. It's not really a limitation of Jekyll here
Lloyd McKenzie (Dec 09 2021 at 23:51):
Right. And one of my objectives is to minimize the amount of Jekyll any IG author needs to use to create IGs. The reason to support markdown is it's "simple and easy to learn". If we start requiring understanding of a whole bunch of extra stuff in markdown files, then it becomes harder to learn. It also makes it harder to move content from markdown to XHTML or vice versa if the author wants to do that. So if we're going to leverage front matter, there needs to be a really good reason.
Eric Haas (Dec 12 2021 at 19:44):
Promoting a Jekyll replacement without having used it before is preposterous. I am not proposing replacing Jekyll just unlocking a core feature.
Jose Costa Teixeira (Dec 12 2021 at 21:01):
I think the idea of Jekyll replacement is a non-starter.
I also think that we should discuss the front matter. I started experimenting with it, I have a first impression of what I'd do with it, but my front matter is not just data that I want structured, it is that I want structured AND managed.
Jose Costa Teixeira (Dec 12 2021 at 21:03):
So my current purposes, logical instances + templates is a superior replacement to front matter. I think Eric's requirements will be different but if there's some commonalities, we can discuss it and I can work on it (whatever "it" turns out to be).
Jose Costa Teixeira (Dec 14 2021 at 19:00):
I had another call today where this is becoming an important thing to do - for me, not front matter, but the ability to add templates to render the logical model instances.
Jose Costa Teixeira (Dec 14 2021 at 19:00):
I can volunteer to make some templates and do the config.json changes in a branch to get this going.
Jose Costa Teixeira (Dec 14 2021 at 19:02):
Another thing that is is high demand is the ability to publish inherited artifacts. Possibly an ig parameter, or even no change, just a change to the code to allow listing artifacts that are not in the IG but in the dependencies
Grahame Grieve (Dec 14 2021 at 19:03):
to get what going? I don't really see the link between logical models and front matter. What's the issue with logical models?
Jose Costa Teixeira (Dec 14 2021 at 19:47):
logical model instances
Jose Costa Teixeira (Dec 14 2021 at 19:47):
or examples
Jose Costa Teixeira (Dec 14 2021 at 19:48):
I need to add structured & managed content to the IGs. For example an "actor definition"
Jose Costa Teixeira (Dec 14 2021 at 19:48):
I can do that as a logical model instance,
Jose Costa Teixeira (Dec 14 2021 at 19:48):
and then have a template that knows how to render that thing
Jose Costa Teixeira (Dec 14 2021 at 19:50):
I could do that with front matter, but with front matter I cannot define a structure upfront (and the structured content is not really manageable, it's just a bit of yaml in a narrative)
Jose Costa Teixeira (Dec 14 2021 at 19:56):
now that there's a way to define instances of a logical model, it should not be hard to PoC the publication of these "custom artifacts".
For this, I think the publisher needs to tell the template (via config.json) what "type" of model is this an instance of.
Jose Costa Teixeira (Dec 14 2021 at 19:56):
(or is there another way to tell which template to use for each type of model?)
Frank Oemig (Dec 15 2021 at 14:27):
@Jose Costa Teixeira I am interested in that..
Jose Costa Teixeira (Dec 15 2021 at 14:34):
@Frank Oemig the discussion on whether/how we're going to do that hasn't happened yet, but it's good to know that you're also interested.
Frank Oemig (Dec 15 2021 at 14:40):
I have to generate instances of LMs and currently do not know how to do that...
So, if it goes with front matter - great. Although I haven't understood that enough.
Jose Costa Teixeira (Dec 15 2021 at 16:25):
it should go without front matter but that is the other discussion.
Lloyd McKenzie (Dec 15 2021 at 18:28):
Outcome of the discussion was that the issue is with Jekyll, not with the template. Jekyll doesn't support front matter in imports and, because the page content must be imported into the templates to get the standard header, footer and other stuff, we run into a problem. It's an issue that's been raised with Jekyll by others and their response has been "don't do that". So we've agreed to modify the publisher so that it strips front matter from imports as part of the migration process at the beginning of running the publisher so that if the markdown tool you're using injects front matter, you don't have to worry about stripping it yourself. However, you won't be able to do anything useful within the frontmatter such as defining variables as they'll just get stripped.
Jose Costa Teixeira (Dec 15 2021 at 18:45):
Yes. So just for info @Frank Oemig I'll be looking at the other topic (logical model instances) for the "doing anything useful such as variables"
Jose Costa Teixeira (Dec 15 2021 at 20:09):
for the instances of LMs, check this https://chat.fhir.org/#narrow/stream/179252-IG-creation/topic/Logical.20Model.20Examples
Grahame Grieve (Dec 20 2021 at 08:56):
I'll come back to the LM issue later when I have time to think. The front matter problem should be sorted in the next release
Last updated: Apr 12 2022 at 19:14 UTC
