FHIR Chat · Feature request - data lookup · shorthand

Stream: shorthand

Topic: Feature request - data lookup

Jose Costa Teixeira (Feb 23 2021 at 18:15):

I'd really like to have a feature in sushi that would allow me to loop / lookup data from a data file.
I'd have a .fsh file with a parametrized rule set, and this parametrized rule set would do something like (abused pseudo-syntax):

with i in file(mydata.csv)
* code = {i.code}
* display = i.display
* i.designation[+].language = "en"
* i.designation[=].language = {i.en}
* i.designation[+].language = "fr-BE"
* i.designation[=].language = {i.fr}
* i.designation[+].language = "nl-BE"
* i.designation[=].language = {i.nl}
end

This would take this csv:

code; display;en;fr;nl;
10;Unmarried;Unmarried;Célibataire;Ongehuwd
20;Married;Married;Marié;Gehuwd
25;Annulled;Annulled;Annulation de marriage;Nietigverklaring van het huwelijk

and create for each of these lines something like this:

    {
      "code" : "25",
      "display" : "Annuled",
      "designation" : [
        {
          "language" : "en",
          "value" : "Annuled"
        },
        {
          "language" : "nl-BE",
          "value" : "Nietigverklaring van het huwelijk"
        },
        {
          "language" : "fr-BE",
          "value" : "Annulation de marriage"
        }
      ]
    },

Jose Costa Teixeira (Feb 23 2021 at 18:16):

This would
a) reduce typing to some degree
b) allow us to generate test data oh-so-more-easily

Jose Costa Teixeira (Feb 23 2021 at 18:16):

There's a few more things to it, but I'd like to ask for feedback

Chris Moesel (Feb 23 2021 at 19:16):

I guess my first question would be if there is significant advantage to building this into the FSH language vs. writing a script (in your favorite language) to generate FSH files based on a CSV? One of the primary advantages of FSH is its simplicity, so I (personally) am wary of complicating the FSH spec when non-FSH solutions exist -- especially if those non-FSH solutions are not that much more difficult than the FSH-based solution would be.

In my opinion, the FSH approach you are suggesting is close enough to an equivalent Python or Node -based approach that I'd just prefer to educate people on how this can be done using Python or Node instead -- which would provide a lot more flexibility without any added complication to the FSH spec.

Jose Costa Teixeira (Feb 23 2021 at 19:22):

Yes, that is my point as well - should we have a external script that does this, or does it make sense in the language?

Jose Costa Teixeira (Feb 23 2021 at 19:23):

Test data generation is something we discussed some time ago and AFAIK there is no good approach to it.

Jose Costa Teixeira (Feb 23 2021 at 19:24):

the idea was triggered by an excellent article by @David Hay who explained about node-red - so for all purposes, it's his fault, really.

David Pyke (Feb 23 2021 at 19:25):

I found a simple excel script all I needed to create 50 fsh examples

Elliot Silver (Feb 23 2021 at 19:36):

David Pyke said:

a simple excel script

Right. Those words don't all belong in the same sentence.

To paraphrase an old saying, "When faced with a problem, some people say 'Let's use excel scripting.' Now, they have two problems."

Jose Costa Teixeira (Feb 23 2021 at 19:36):

yes, that is what I did as well. I have an excel file that creates fsh stuff that generates my data (before the excel file created the json directy)

Jose Costa Teixeira (Feb 23 2021 at 19:36):

and then I thought i'm surely not the only one with this issue

David Pyke (Feb 23 2021 at 19:36):

Hey, I use what I know. In the old days, that's all I had access to...

Elliot Silver (Feb 23 2021 at 19:37):

I'd rather use a rock and chisel. I'd be more productive.

David Pyke (Feb 23 2021 at 19:37):

I'd use bash scripts but they keep telling me to use my windows laptop for work stuff.

Elliot Silver (Feb 23 2021 at 19:38):

What's wrong with cygwin?

Elliot Silver (Feb 23 2021 at 19:38):

Or docker?

David Pyke (Feb 23 2021 at 19:38):

YOu want the full list or the top ten?

Jose Costa Teixeira (Feb 23 2021 at 19:43):

So, data lookup...

Jose Costa Teixeira (Feb 23 2021 at 19:43):

is this something others would want?

Jose Costa Teixeira (Feb 23 2021 at 19:43):

and if so, external script, or does it make sense as part of the language

David Pyke (Feb 23 2021 at 19:44):

I'd much prefer keeping FSH/Sushi simple and using an external script

Chris Moesel (Feb 23 2021 at 19:58):

I think it would definitely be worth exploring the use of external scripts first. We could maybe even provide a few examples or even a small library to help with it.

David Hay (Feb 23 2021 at 20:05):

I'm into nodejs scripts these days.. (The csv post was more to explore node-red in truth...)

Lloyd McKenzie (Feb 23 2021 at 20:27):

My concern is having stuff in SUSHI that can't be round-tripped through resources. We shouldn't have features in SUSHI that aren't expressible (at least as extensions) in one of the standard resource types. That way if someone wants to move from using SUSHI to author content to something else, they can.

Jean Duteau (Feb 23 2021 at 20:41):

Lloyd McKenzie said:

My concern is having stuff in SUSHI that can't be round-tripped through resources. We shouldn't have features in SUSHI that aren't expressible (at least as extensions) in one of the standard resource types. That way if someone wants to move from using SUSHI to author content to something else, they can.

I'm not sure that applies here. SUSHI does a lot of time-saving things and this, if it were to be done via SUSHI, would just be another one of those. We already have rule sets to simplify repetition and this would be like that. I think Jose was just proposing a way that SUSHI could create umpteen example instances. If you moved from SUSHI to something else, you'd still have the underlying instances that were created. I might have misunderstood your concern though. :)

Jose Costa Teixeira (Feb 23 2021 at 20:47):

I think parametrized rule sets are not in the roundtrip zone, but they are definitely helpful.

Jose Costa Teixeira (Feb 23 2021 at 20:47):

We could create an external toolbox for such things - We can call it Paella.

Jose Costa Teixeira (Feb 23 2021 at 20:48):

The goal in my question is that these tools should themselves be interoperable and hopefully useful for more than a few iterations.

Jose Costa Teixeira (Feb 23 2021 at 20:50):

so, better in sushi than yet another script

Jose Costa Teixeira (Feb 23 2021 at 20:52):

@Lloyd McKenzie do i understand correctly that the notion of "create M resources and populate this field, with each value of column N" should be expressable in FHIR?

Lloyd McKenzie (Feb 23 2021 at 20:56):

@Jose Costa Teixeira - yes. That shouldn't be something that's specific to SUSHI. I should be able to author examples in JSON or XML or even RDF and embed extensions or something that would drive that same behavior.

Jose Costa Teixeira (Feb 23 2021 at 21:01):

Ok let me see if I can improve the concept

Jean Duteau (Feb 23 2021 at 21:25):

Lloyd McKenzie said:

Jose Costa Teixeira - yes. That shouldn't be something that's specific to SUSHI. I should be able to author examples in JSON or XML or even RDF and embed extensions or something that would drive that same behavior.

i don't understand what you are proposing Lloyd. You are saying that we should have the ability to just author a parameterized example in JSON with extensions that say "put parameterized content here"?

Chris Moesel (Feb 23 2021 at 21:30):

I think I disagree w/ the notion that every feature in SUSHI should have an exact parity in the IG Publisher and/or other FHIR authoring tools. That kills innovation. At the end of the day, SUSHI spits out valid FHIR resources using the standard FHIR specification and serialization. Any tool that can load and edit FHIR resources (SDs, etc) can "round trip" anything that SUSHI created. So what is the problem? Or am I misunderstanding?

Lloyd McKenzie (Feb 23 2021 at 22:04):

The concern is that you lock people into a single authoring approach if the other authoring approaches can't capture or represent the same content. Innovation is fine and good. With extensions it should be possible to reflect whatever is being done in SUSHI into 'regular' resources.

Chris Moesel (Feb 23 2021 at 22:32):

I guess I am saying you're not locked in since the resources at that point (after export) are complete. You can load them into whatever other solution you want. You might author it a different way thereafter (which is kind of the point of having different authoring tools anyway), but you definitely can load it and edit it in any tool that supports FHIR definitions. In a way, FSH and SUSHI are really just a UI for how you interact w/ FHIR definitions. Some tools have graphical UIs; we have text-based UIs that can be managed in a source control system.

Jose Costa Teixeira (Feb 23 2021 at 23:03):

Some ideas:

1. Both in Sushi and in other implementations, the "master" resource would remain untouched - it is the other 500 resources that would be created.
2. If we define a few extensions for this, then a sushi editor can populate those extensions (and we can have a rule set to facilitate that further)
3. The procedure for expanding this extension can be coded in a standard library - or in sushi. Sushi would still generate the corresponding resource and then generate the data files (or could do it in one pass if there's a way)
4. The extensions could look like:

{
    "resourceType": "Patient",
    "extension" : [{
        "url" : "http://hl7.org/fhir/StructureDefinition/data_generation_config",
        "extension" : [{
            "url" : "source_csv_file",
            "valueString" : "mydata.csv"
        }, {
            "url" : "total_rows",
            "valueInteger" : 500
        }]
    }],
    "id": "000",
    "_id": {
        "extension" : [ {
            "url" : "http://example.org/fhir/StructureDefinition/data_generation_column",
            "valueString" : "id"
        }]
    }
    "birthDate": "1970-01-01"
    "_birthDate": {
        "extension" : [ {
            "url" : "http://example.org/fhir/StructureDefinition/data_generation_random_date",
            "valuePeriod" : {
                "start": "2020-01-01",
                "end": "2020-01-31"
            }
        }]
    }
}...

Lloyd McKenzie (Feb 24 2021 at 03:35):

If the only way you can reasonably maintain the resources (using the generation process) is only SUSHI, then the notion "you're free to take the 500 generated resources and maintain them in something else" isn't terribly realistic.

Chris Moesel (Feb 24 2021 at 13:42):

OK. I understand what you're saying then, @Lloyd McKenzie. I'm still not sure I agree w/ the overall premise, but I understand it. Let me ask this: are you talking about proprietary features that SUSHI might implement outside of the spec; or are you saying that you don't think FSH should even specify features like this unless it also specifies extensions that can capture all of the detail of whatever that feature provides (like @Jose Costa Teixeira demonstrated above)?

Chris Moesel (Feb 24 2021 at 13:43):

Because if FSH is a standard, then the feature is already captured in a standard-based way that any tool can choose to implement or not implement.

Lloyd McKenzie (Feb 24 2021 at 14:27):

I think the base premise is that most features that are used as input to the FHIR IG process should be expressible and shareable as FHIR resources. We try to keep the set of content that's not resources relatively limited - batch scripts, a very limited ini file, etc. The biggest exception so far has been the files that define narrative pages and intros and usage notes, and even with those, there's discussion about whether that content should move into resources.

I'm not setting an absolute, but I am saying that when new features are introduced in the IG publication stack, the community (not just the FSH community) should have a discussion about whether and how those features should manifest as resources.

Lloyd McKenzie (Feb 24 2021 at 14:28):

My criteria in that conversation would be about how strongly does that feature lock a designer into a single approach. The greater the lock-in, the more I think we'd need to have a mechanism to expose the capability in a 'standard' resource syntax.

Jose Costa Teixeira (Feb 24 2021 at 15:13):

question on this:
If we have a "master" resource from which the tool generates the 500 resources, we don't expect those 500 resources to be committed, right?

Lloyd McKenzie (Feb 24 2021 at 15:32):

Also true

Jean Duteau (Feb 24 2021 at 16:16):

Lloyd McKenzie said:

My criteria in that conversation would be about how strongly does that feature lock a designer into a single approach. The greater the lock-in, the more I think we'd need to have a mechanism to expose the capability in a 'standard' resource syntax.

I'm sorry but this is pure hogwash. If I create an Excel script that takes a list of parameters and generates 500 resources from it, and I check that in to my Guide so that other committers of my guide can use it, you're now telling me that is verboten because I've "locked a designer into a single approach". That's just not a reasonable position to take. You're telling me that Grahame is going to add some code to the publisher and we're going to add some extensions that are purely for pre-assembly of resource instances? That is so not in the purview of what the publisher should be doing. It should be incumbent on the spec writers to provide those resource instances in some means. And if that is using a homebaked Excel script or using a feature that SUSHI might incorporate or if it means hand-writing them, that should be up to the spec writer and definitely not something required of the publishing tooling stack.

Now, if SUSHI required the publishing tooling stack to do something specific so that one of its features could be used, that is when I would agree with you. But as long as SUSHI or Forge or whatever tool you use, ends up producing the proper inputs, it should not and can not matter how it does it.

Elliot Silver (Feb 24 2021 at 17:09):

I tend to agree with @Jean Duteau. What do we say about all of those interactive questionnaire designers that don't have any input files, and output Questionnaire resources heavily annotated with private extensions? Should they be verboten too? Is FSH the design tool, or an input format? If it's an input format, why isn't there pressure to support it in Forge? How do we decide which features a tool is allowed to differentiate and show value with, and which features are the common base level? Should we say Sushi should be banned until goFSH is capable of ingesting any IG and allowing you to move from/away the Sushi/goFSH combo with full fidelity? Can I develop a process that uses Adobe Illustrator or Microsoft Visio to generate images instead of plantUML? As long as the output is SVG should it matter?

As a user, I'd like the ability to move my content from one toolchain to another, and I'd be ticked to find out that I'm stuck in a tool because of a choice I made. But portability should be value that the tool author adds, not a requirement of HL7 on anyone who builds a tool.

I feel strange arguing against interoperability, but we already have an interoperable layer: the IG resources. I don't know that we need--or more correctly, that HL7 should mandate---to move interoperability up the chain.

Lloyd McKenzie (Feb 24 2021 at 23:12):

I'm not saying that's strictly verboten, but it is problematic - at least for HL7-maintained IGs. You've now introduced a custom tool into the maintenance process for that IG. If you quit working for the committee or get hit by a bus, HL7 is now on the hook for figuring out what the heck you did and how to maintain it and make necessary changes to put out a new release. Granted, there are going to be new requirements and tight timelines that lead to short-term cobbled-together solutions that do whatever's needed for the initial release. However, over time, there needs to be some thought about how to incorporate that into a general purpose solution that's part of the framework, understood by most authors and that we don't have 10 different ways of accomplishing the same thing across most IGs. I don't think that "coming together timeline" is necessarily in the weeks or even months - i.e. going forth and trying stuff is cool. But as an organization we do have to be sensitive to long-term maintainability, durability of tooling stack and consistency of published artifacts.

It may be that the outcome of that discussion is "spreadsheets are fine, this is the macro we're going to use, here's the Git project we're going to maintain it on, and these are the minimum two people who will take on responsibility for keeping it current. And FSH could fall in the same boat. "For certain functions, the only way to make it happen is to use FSH", meaning no need for extensions, no support from other authoring tools. But that's not a choice that should be made just by the FSH community or just by a particular IG developer. For short-term experimentation, sure. But for long-term maintenance, we need to ensure the organization is comfortable with the dependencies we have and that we're also creating an environment that's conducive to multiple authoring environments that can share inputs and produce consistent outputs. All our eggs in one basket has proved to be problematic in the past. Maybe for some functions the number of eggs is small enough that one basket is fine. Being able to generate large volumes of example instances seems like something that will involve a large number of eggs (i.e. it'll be a requirement for lots of IGs.)

Elliot Silver (Feb 25 2021 at 18:22):

Lloyd McKenzie said:

I'm not saying that's strictly verboten, but it is problematic - at least for HL7-maintained IGs.

I'm fine with HL7 saying "here is the accepted toolset for HL7 IGs" or even "here are the accepted features of the accepted tools for HL7 IGs" but the way you were framing it was "tools shall not have features unless all tools have those features."

Jean Duteau (Feb 25 2021 at 18:32):

i'm still missing your point @Lloyd McKenzie. Let me go back to the original request - I want to create a multitude of example instances and I don't want to create them all by hand. It seems like you are saying that unless I come up with some generalized way that everyone can do this, I can't do this.

Lloyd McKenzie (Feb 25 2021 at 18:39):

What I'm saying is "this is a capability that is going to be of interest to a bunch of IGs, and therefore probably shouldn't be a solution that's tied to FSH only - so think about that as part of your initial experimentation - "How could we surface this back into standard FHIR instances".

Lloyd McKenzie (Feb 25 2021 at 18:40):

Not saying you can't experiment without making it work everywhere first. But don't just say "we're going to do this in FSH and not worry about the ability to interoperate with other authoring approaches"

Last updated: Apr 12 2022 at 19:14 UTC