FHIR Chat · Test Data Generation · workflow

Stream: workflow

Topic: Test Data Generation

Jose Costa Teixeira (Mar 22 2019 at 21:02):

not sure if right stream but we mentioned this in the calls.

Jose Costa Teixeira (Mar 22 2019 at 21:03):

while looking at node-red (not red-node as I typed before), i saw one example of data generation (could be useful to introduce or may trigger better ideas).

Jose Costa Teixeira (Mar 22 2019 at 21:04):

there is a component that generates data based on syntax shown in this example:

{
    "ID": "{{guid}}",
    "value": {{float 20 80 '0.00'}},
    "manufacturerID": "{{randomItem 'IBM' 'SONY' 'APPLE'}}",
    "tags":[
        {{#repeat 0 3}}
           "{{tag}}"
        {{/repeat}}
    ],
    "location":{
        "city": "{{city}}",
        "zone": "{{randomItem 'A' 'B' 'C'}}",
        "building": "{{randomItem '1' '2'}}",
        "floor": "{{randomItem '1F' '2F' '3F'}}",
        "room": "{{randomItem '01' '02' '03'}}"
    }
}

Lloyd McKenzie (Mar 22 2019 at 23:08):

Yup, that's definitely something we should talk about/explore. Is it based on extending a Liquid engine?

Jose Costa Teixeira (Mar 23 2019 at 06:27):

not sure why you ask, but there's no Liquid there

Jose Costa Teixeira (Mar 23 2019 at 06:27):

(AFAIK)

Grahame Grieve (Mar 23 2019 at 06:29):

you should use Liquid then - see http://wiki.hl7.org/index.php?title=FHIR_Liquid_Profile

Jose Costa Teixeira (Mar 23 2019 at 06:30):

ah, just realized now the double brackets

Jose Costa Teixeira (Mar 23 2019 at 06:36):

we still need to define the functions for generating data, right? Can we define functions in liquid like "randomItem(a,b, c)"?

Lloyd McKenzie (Mar 23 2019 at 15:30):

That's what I'm wondering. If t's just extending a liquid engine, that has lots of possibilities

Grahame Grieve (Mar 23 2019 at 21:45):

you don't define them 'in liquid' but you expose an api to liquid that has those behaviors. it's effectively the same outcome. In this case, you really want to conceive of it as iterating over a synthesized set of values

Lloyd McKenzie (Mar 23 2019 at 21:51):

I'm actually more interested in this for generating 'defined' examples that have specific values - but using a templating approach so that it's a heck of a lot easier to maintain them. For ExampleScenario, having 50+ instances for a single scenario will be average to even on the low side, so we're looking for an authoring/tooling solution to make that process lighter-weight. But we very much want the example instances that get generated to be consistent.

Grahame Grieve (Mar 24 2019 at 09:43):

check out how \source\patient\patient-examples-cypress-template.xml works

Jose Costa Teixeira (Mar 24 2019 at 11:02):

my previous solution was:
1. an excel file where I updated the values of the relevant attributes. I was in full control of consistency, and while automatically generating value from a template would be nicer, I was ok with excel fill down and copy-paste.
2. any dumb way to transform that into the resources (my way was embarassingly dumb)

Jose Costa Teixeira (Mar 24 2019 at 11:05):

cypress template seems the same approach.

Jose Costa Teixeira (Mar 24 2019 at 11:11):

could we combine these approaches and have a template that can contain both
{{randomItem 'A' 'B' 'C'}}
and
{{loopVar(patientId)}}
{{loopVar(medicationCode)}}

( loopVar is if you have several loops e.g. patient1..patient5 but also medication1..medication20)

Jose Costa Teixeira (Mar 24 2019 at 11:11):

don't mind the syntax

Grahame Grieve (Mar 24 2019 at 19:48):

I'll rework the cypress example to use the liquid engine and we'll see what it looks like

Eric Haas (Mar 24 2019 at 21:18):

I think starting with excel is the way to go, I've use both templates and class libraries. I am not sure what the inherit advantage of liquid is for this is. ( supports fhir path, works in build? ) versus creating examples outside of build tooling using your language template combination of chioice.(e.g., Python-Jinja2) I would probably create a python dictionary using its standard library and validate against a test server. Templates are clones and give you less flexibility across example scenarios. I think would need a template per resource per example scenario.

Lloyd McKenzie (Mar 24 2019 at 21:27):

I want examples to be authorable by non-programming people. Liquid templates they might be able to manage. Full on Python, not so much...

Lloyd McKenzie (Mar 24 2019 at 21:28):

Also, I want all examples across all IGs published by HL7 to be authored using the same approach - don't want 3 or 4 different programming languages sprinkled around based on the author's preferences.

Eric Haas (Mar 25 2019 at 00:14):

I think you misunderstand. The way to create an example is the spreadsheet. The code is just a way to transform it into the proper json file.

Eric Haas (Mar 25 2019 at 00:14):

Everybody can use a spreadsheet to author an example

Eric Haas (Mar 25 2019 at 00:15):

Liquid is no easier than python to a non programner

Eric Haas (Mar 25 2019 at 00:21):

I really don’t understand the use case. You want valid examples in a wire format for the build source right? Or do you want them in some other format as the source. That would be a bad look ( something about eating own dogfood. )

Lloyd McKenzie (Mar 25 2019 at 01:11):

Non-programmers are going to have to create the templates, not just the data. And I think stubbing in liquid variables and perhaps learning a couple of iterators inside a 'standard' example (XML or JSON) is going to be much easier than righting code. My targets are the people who create instance examples now in the pharmacy, patient care and similar work groups. Few, if any of them are programmers.

Lloyd McKenzie (Mar 25 2019 at 01:12):

I definitely expect that the source is going to have to be templates + data in the source when we're looking to create 50+ instances to show a particular scenario.

Eric Haas (Mar 25 2019 at 02:06):

I'm still not clear what you want. It is not clear to me how you plan to use a template without a data source. A tool external to the build to convert excel to json/xml using liquid? or examples in xml+ liquid? why not just use clinfhir's ui?

Lloyd McKenzie (Mar 25 2019 at 03:15):

I'm not sure if the source data will be in Excel or not. If it is, then the new IGPublisher template framework will have to turn the Excel into JSON for Liquid to use. The work group (or other non-technical individuals) would author and maintain both the templates and the data. If we can extend ClinFHIR to allow authoring templated instances, that might be ok too, but I'm not presuming that all instance authors will necessarily want to use ClinFHIR.

Jose Costa Teixeira (Jul 13 2019 at 08:40):

I see two possible approches in the above discussion:
1. Data set based, sets are in an excel or otherwise-easy-to-enter format
2. Data rules based.

Jose Costa Teixeira (Jul 13 2019 at 08:41):

1: pasted image

Jose Costa Teixeira (Jul 13 2019 at 08:41):

2: pasted image

Jose Costa Teixeira (Jul 13 2019 at 08:43):

The mock syntax above is more about option 2, but option 1 may be easier to maintain consistency.

Jose Costa Teixeira (Jul 13 2019 at 08:44):

This can be used not only to generate many instances of one resource, but many instances of many resources.
Not sure how that could be with option 2, but with option 1 this should be easy.

Jose Costa Teixeira (Jul 13 2019 at 08:45):

and one question: "patient.json", where the expressions are, should be:
1. a resource instance populated with expressions
2. a profile populated with expressions
?

Last updated: Apr 12 2022 at 19:14 UTC