Stream: IG creation
Topic: Impossible to understand
Mark Kramer (Sep 09 2019 at 20:44):
There must be a better way to present a profile than this: https://lightmyfhir.org/2019/09/09/fhir-implementation-guide-presentation/
Some food for thought going into our IG discussions later this week.
Grahame Grieve (Sep 09 2019 at 21:26):
that does look sensationally awful. My first question was, what does the differential look like, and I think that you did yourself a disservice by asking "Can you summarize the purpose of this profile" without showing the differential
Grahame Grieve (Sep 09 2019 at 21:27):
and you didn't even provide a link....
Josh Mandel (Sep 09 2019 at 21:32):
Differential looks like this
From http://hl7.org/fhir/us/breast-radiology/2019Sep/StructureDefinition-breastrad-BreastAbnormality.html
Grahame Grieve (Sep 10 2019 at 00:07):
well... it's better. Not great. We can certainly search for ideas to improve that
Mark Kramer (Sep 10 2019 at 02:05):
I added the link to the post, thanks for catching that.
Mark Kramer (Sep 10 2019 at 18:13):
Well, @Grahame Grieve , this issue got me a bit inspired, and together with @Chris Moesel, we put together a proposal on something that might be useful in this context. We would like to get your feedback on this idea. FHIR-Profiling-Language-draft-v1.docx
Grahame Grieve (Sep 10 2019 at 19:44):
starting by criticising excel will get me on board ;-)
Grahame Grieve (Sep 10 2019 at 19:45):
we've kind of toyed with this idea on and off. I could certainly build support for something like this straight into the IG publisher - for me, it would be a text format like the one we defined for structure map (and I would argue it should be similar and consistent where they overlap).
Grahame Grieve (Sep 10 2019 at 19:47):
but.... there's a problem, which is that if you look at real world profiles, taken from actual IGs, the documentation details make a mess of the simplicity of the format. It turns ugly. And I firmly believe that documentation is important. So I think it would be good to extend this with some real world documentation
Grahame Grieve (Sep 10 2019 at 19:48):
"5. Many years of experience has proven that creating and maintaining complex software projects is best approached with textual languages. As a DSL designed for the job of profiling, FPL is concise, understandable, and aligned to user intentions."
That I certainly agree with
Grahame Grieve (Sep 10 2019 at 19:51):
but, let's take value sets.... the value set text format is going to collapse to some YAML variant of a value set. Nicola has long argued that we should use YAML for this. And why doesn't the same apply for StructureDefinitions? what higher level language could be defined that doesn't break down in the presence of messy documentation and real world mixed constraints from actual cases?
Grahame Grieve (Sep 10 2019 at 19:51):
maybe there's some, so I'm interested...
Grahame Grieve (Sep 10 2019 at 19:53):
btw, with regard to the original, I had some specific questions I haven't had time to pursue yet.
- why are these component observations, not independent observations - is that a right design (or just hack)
- if they are not really components, then this comes back to: how to visualise graphs of observations (already an open issue for me)
- if they aren't.... is it time to get to breaking up the reprsentation of a large profile into chunks to make it more manageable for consumers of the documentation?
Grahame Grieve (Sep 10 2019 at 19:56):
one more comment. One proposed benefit:
• Provides meaningful differentials
But my experience here looking at the IGs that we have: the more pre-processing before the differential, the less meaningful your differential (the IGs you guys produce through thins like CIMPL tend to jam all sorts of non-essential things into the differential), where as shallow design processes are more sparing. I don't see why introducing a new text representation would change that (though it depends how much this is about language as opposed to syntax)
Eric Haas (Sep 10 2019 at 20:49):
is the proposal to use this natively and drive the publisher or it just a language to create these artifacts that will drive the publisher, and why couldn't it be YAML ? ( I'm on the YML bandwagon too since Python is all about indents :-) )
Grahame Grieve (Sep 10 2019 at 22:13):
i would take it as a possible input to the publisher. certainly we have a running thread for a text format like this. The suggested format has some value add over a straight YAML format, but it's not clear to me that what will survive the iteration process will be a big deal difference
Mark Kramer (Sep 10 2019 at 23:04):
Agreed, Excel is a catastrophe ;-)
Now that we've agreed that, we can discuss the best form for FPL. One keenly important aspect, as the leader of multiple profiling projects, is that it must be readable and understandable for everyone across the FHIR community, not just the most technically skilled folks. The second thing is that is should be fast to build and revise. We went through literally 100's of revisions on the way to mCODE, and I'm convinced we never would have made it without very agile profiling tools.
Another place we agree is that we should use existing piece-parts. There's no sense inventing new things if existing things are meet the design criteria.
On the "provides meaningful diffs" comment, what I was trying to say is that two versions of the FPL could be compared to each other,.meaningfully. Comparing FPL across versions reveals exactly what was changed in the profiles. I wasn't saying diff the resulting SDs.
As far as the example profile that drove this conversation, I can't answer for it. It isn't mine. This discussion came about because I got frustrated trying to review it for the ballot, and couldn't make heads or tails of it.
I'll defer comments on YAML or Structure Map for a later date.
Grahame Grieve (Sep 10 2019 at 23:08):
sure I'll wait on that. Slicing matters too....
Lloyd McKenzie (Sep 10 2019 at 23:56):
Excel for IG authoring (and publication) is useful in several ways:
- there are times when a columnar view of what you're editing (including the ability to quickly and arbitrarily filter the rows) is extremely helpful in the authoring process.
- The excel column headings give you prompts for what needs to exist and a visual overview of what's going on that a text format doesn't
- from a publication perspective, most people know how to use Excel and can easily use it to create filtered views, add notes, document mappings, etc.
Eric Haas (Sep 10 2019 at 23:56):
I don't think we have agreed to anything yet . I think excel is an excellent UI and easier to manipulate than text file and familar and accessible by 90% of the population, but also agree is not good for collaboration and repos. My biggest concern is that a new text based language that you have to learn will be a barrier to adoption.
Eric Haas (Sep 10 2019 at 23:57):
Lloyd beat me to it :-)
Lloyd McKenzie (Sep 10 2019 at 23:58):
I'm not arguing that Excel is a desirable persistence format - it's horrible for that. Nor am I arguing that Excel should be the only or even preferred tool for authoring profiles, just that it should be available. My personal desire is to have an ability to easily do a save-as/load from Excel such that the persistence layer can be regular XML/JSON (the same as all of our other editing tools)
Grahame Grieve (Sep 11 2019 at 04:43):
feel free to write an import macro....
Grahame Grieve (Sep 11 2019 at 04:43):
but the workflow around saving is horrible. Excel should die
Lloyd McKenzie (Sep 11 2019 at 12:40):
It can die when we have other tools that offer equivalent functionality (and also run the latest & greatest version of FHIR and are responsive enough to add new support for new StructureDefinition extensions in a timely manner)
Eric Haas (Sep 11 2019 at 14:10):
The import thingy is on my wish list - but pretty far down there
Lloyd McKenzie (Sep 11 2019 at 14:12):
It's high on my wish list. But it's a long ways down my priority list :)
Mark Kramer (Sep 11 2019 at 15:23):
The discussion isn't whether Excel should die. It will continue to live on because it has some usefulness to some users. The idea of FPL is to provide something very concise and readable that scales over large projects. I believe that FPL will be easy to learn, and turn out to be _one_ of the preferred ways to approach profiling.
Our experience with CIMPL has been that we often run the CIMPL complier without running the IG publisher, because it checks all sorts of things about the syntactic and logical consistency of the specification, in a few seconds. Once the model is sound, we may run the IG publisher if we want to see how the final output will look. But often, we just keep going and building profiles without running the IG publisher. I suggest these steps should be separate.
Rute Martins (Sep 11 2019 at 20:55):
@Mark Kramer I agree these two steps should be separate - i.e., but regression testing is important, so I think most times I want to see the output for a particular FHIR profile (or set of profiles) and related extensions and value set, without having to generate the entire IG. I think the fact that we have to drag the entire IG to see the output for a single profile is the reason why we don't run the IG publisher everytime; it's just too cumbersome. But I guess that's a different requirement entirely :)
Mark Kramer (Sep 12 2019 at 15:38):
@Grahame Grieve I would like to offer a PSS to the FMG on this topic. Would you be agreeable to that discussion?
Grahame Grieve (Sep 12 2019 at 15:41):
it seems premature to jump to that step right now
Mark Kramer (Sep 12 2019 at 15:43):
What kind of collateral would be needed to make case that FPL is ready to become a project?
Grahame Grieve (Sep 12 2019 at 15:44):
I think FHIR-I would need to own it, so you'd need to get some sense that FHIR-I would have consensus that this was worth while. I certainly haven't seen any sense of that.
Mark Kramer (Sep 12 2019 at 15:45):
Perfect, thanks. FHIR-I....
Josh Mandel (Sep 12 2019 at 16:09):
You'll have support from me :)
Grahame Grieve (Sep 12 2019 at 16:10):
well, I'm yet to be convinced that this is anything better than a YAML format for the resources we have
Josh Mandel (Sep 12 2019 at 16:30):
You mean, like hand authoring StructureDefinitions in YAML?
Josh Mandel (Sep 12 2019 at 16:30):
That'd be approximately no different than hand authoring them in JSON.
Grahame Grieve (Sep 12 2019 at 16:30):
yes. What would "FPL" offer as an advantage over that?
Grahame Grieve (Sep 12 2019 at 16:31):
well, json is tricky stupid with commas but you could go to loose json, yes
Josh Mandel (Sep 12 2019 at 16:31):
If it didn't offer an advantage, it'd be a pretty poor DSL.
Grahame Grieve (Sep 12 2019 at 16:31):
which is how I do my authoring
Grahame Grieve (Sep 12 2019 at 16:32):
well, what advantage can it offer? So far, I see 3 apparent advantages
- it's simpler - because it omits actual real requirements (oops)
- it allows you to define common canonicals (nice, but not the only way)
- it groups a set of related things into a single file (nice, but that big a deal?)
Grahame Grieve (Sep 12 2019 at 16:32):
I'm open to learning that I'm wrong....
Michel Rutten (Sep 12 2019 at 16:49):
@Brian Kaney
Andy Stechishin (Sep 12 2019 at 17:05):
@Grahame Grieve I do agree with your statements about YAML and I have fairly extensive experience with YAML (I can generate the entire V3 Vocabulary into YAML files). I don't think writing in YAML provides large advantages over writing in JSON. And likely if you could hand code the YAML, you are adept enough to write a YAML->JSON converter (if not contact me and we can come to an arrangement)
Eric Haas (Sep 12 2019 at 17:20):
can FPL be imported to and exported from spreadsheet format too :-)
Chris Moesel (Sep 12 2019 at 17:27):
@Eric Haas -- sure! It's just a matter of writing some code. ;-) But seriously, we talked about translating SD to FPL, so I suspect it could also go for Excel. The question is how to prioritize all this stuff. (And determining what is the subset of SD that is supported; it might be difficult to translate every possible amalgamation of SD into FPL -- but if Excel spreadsheets are a little more constrained, that might actually be easier).
Grahame Grieve (Sep 12 2019 at 17:27):
please no can spreadsheet tails not wag the dog
Lloyd McKenzie (Sep 12 2019 at 18:18):
Spreadsheets can (in principle) be converted to and from any format. The question is why we need to introduce another format.
Mark Kramer (Sep 12 2019 at 22:41):
Because a DSL could be very useful. Right now, spreadsheets are the only format outside of editing the SD itself (Forge and Trifolia are essentially SD editors). A DSL is 10x more productive.
Lloyd McKenzie (Sep 12 2019 at 22:47):
You can edit in JSON
Lloyd McKenzie (Sep 12 2019 at 22:48):
So the question is whether a version of DSL that supports everything that's legal in StructureDefinition would be significantly easier to edit than JSON.
Josh Mandel (Sep 12 2019 at 23:44):
Does the spreadsheet support "everything that's legal in StructureDefinition," btw?
Eric Haas (Sep 12 2019 at 23:55):
So the question is whether a version of DSL that supports everything that's legal in StructureDefinition would be significantly easier to edit than JSON.
or YML
But what is the downside of yet another way to create SDs (YAWTCS)
if we all agree the official exchange layer is the fhir artifact. Realistically we will use what works best for each of us since changing ones personal workflow is hard.
Josh Mandel (Sep 12 2019 at 23:58):
One important issue for me is: what's the official format checked into our revision control systems including the FHIR core spec. Excel spreadsheets are very hard to read diffs on; are prone to merge conflicts; and require a fair degree of sophistication to parse. (Though I'll note that @Gino Canessa still often edits these in a text editor when making tweaks, rather than in the Excel GUI, which to me shows just how valuable an actual editor-friendly format would be.)
Lloyd McKenzie (Sep 13 2019 at 00:29):
For those that use spreadsheets, spreadsheet XML is what's checked in. We've talked w/ Grahame about running the same normalization mechanism on IG Excels as he does on the 'core' ones, but it's not high up his priority list. (I may do it at some point, though it's not at the top of my list either). We agreed that, in principle, a better solution is to use Excel as an editor but save XML or JSON structure definitions, but that's a lot more work.
Chris Moesel (Sep 13 2019 at 00:43):
So the question is whether a version of DSL that supports everything that's legal in StructureDefinition would be significantly easier to edit than JSON.
SD provides multiple ways to do the same thing (as evidenced in the recent thread about the best way to fix a code). Ideally FPL will provide a single way to do each thing an author might need to do. So that's easier. I'd also like to see FPL simplify common patterns to make them easier (particularly common approaches to slicing).
I'd also question whether FPL really needs to support every possible thing SD can support. I'd suggest we go through the top 10 or so IGs and take inventory of the SD features they use. It will be a subset for sure. That's what we optimize for. And then, if necessary, we provide a way to drop into a mode that's nearly like editing SD itself -- because someone will probably say it's a no-go if we can't do everything SD does. But 98% of people will never use that part. After all, we're about interoperability -- so we should probably be discouraging people from doing the really wacky one-off stuff anyway.
Lloyd McKenzie (Sep 13 2019 at 01:58):
In answer to your question Josh about "do spreadsheets support everything?" the answer is "mostly, but probably not entirely". That's not so much from intention as that the spreadsheet tends to catch up to new features when people complain/need them. If we're going to look at a new syntax, it should certainly at least support all of the weird stuff that people use now.
Lloyd McKenzie (Sep 13 2019 at 02:00):
Everything that appears in an IG is because the use-case requires it. No-one does "wacky" things for fun and entertainment. They only figured out how to do (and often badgered tool-makers to support doing it) because business requirements demanded it. "Discouraging" isn't necessary or appropriate. The fact that it's hard/esoteric is discouragement enough.
Lloyd McKenzie (Sep 13 2019 at 02:02):
Anyone who's going to express profiles in a custom language is, pretty much by definition, going to be a power user. We're unlikely to get non-technical users to author in free-hand text using a custom language when there's no tools that support guiding or validating. So that makes the user of "power features" of profiles even more likely.
Chris Moesel (Sep 13 2019 at 02:58):
I think we're in agreement on some of these things, Lloyd.
"Discouraging" isn't necessary or appropriate. The fact that it's hard/esoteric is discouragement enough.
Right, that's actually the kind of discouragement I meant. Every weird edge case that an author struggles to make work in their IG is a weird edge case that every single implementing system will also have to struggle with to be conformant. So keep it hard to do one-off stuff so that people are darned sure they really need whatever it is before putting it into their spec.
We're unlikely to get non-technical users to author in free-hand text using a custom language when there's no tools that support guiding or validating.
I'm not trying to suggest that non-technical authors will be writing IGs, but I would like for the bar to be lower than it is today -- and we will 100% want tools to support guiding and validating. The good news is that products like VS Code, Atom, and JetBrains have great infrastructure for developing tools to support DSLs.
Lloyd McKenzie (Sep 13 2019 at 03:13):
What I'm suggesting is that FPL is more likely to be used by techie folks. The non-technically sophisticated users will migrate to Forge, Trifolia and other tools with more limited capabilities but nicer UIs that hand-hold and prevent users from doing anything that's not allowed. So if we're introducing something like FPL, it should be aimed at the stuff the more sophisticated users will want to do.
Mark Kramer (Sep 13 2019 at 14:28):
I don't disagree that novice users will prefer the visual tools, and "techie folks" will prefer the FPL. It's productivity, maintainability, and transparency that drives FPL.
Chris Moesel (Sep 13 2019 at 15:04):
Thanks for the clarification, @Lloyd McKenzie. I get it now. :-)
Bob Milius (Sep 13 2019 at 15:27):
As one who represents someone between "non-technically sophisticated" and a "techie folk" (I use Lloyd's spreadsheets to create my IGs), I'd like a nice GUI to get started, and then be able to import those artifacts into a more powerful tool (but possibly less user friendly ) for possibly tricky things. I'm not shy about hand-editing JSON or what ever FPL might become.
Eric Haas (Sep 13 2019 at 17:05):
Developer tools in atom for this would nice. Do they exist today? So I’ve never done profiling using either FPL or YML and it would be nice try each approach if we are committed to a change.
Mark Kramer (Sep 13 2019 at 17:15):
Just to clarify -- FPL doesn't yet exist :) But we are exploring what the DSL might look like. Inputs are very welcome. Any interest in a Birds of a Feather meeting on FHIR Profiling Language this week at HL7?
Last updated: Apr 12 2022 at 19:14 UTC
