FHIR Chat · Decimal Syntax question · fhirpath

Stream: fhirpath

Topic: Decimal Syntax question

Grahame Grieve (Jul 30 2019 at 02:49):

@Bryn Rhodes while I'm noodling about decimals.... there's a problem in FHIRPath. FHIR allows the exponent representation, but FHIRPath doesn't. So what should convertsToDecimal() / toDecimal() do? In my implementation, they accept exponents, since that might come from the resource....

Grahame Grieve (Jul 30 2019 at 02:49):

but that is not conformant.

Bryn Rhodes (Jul 30 2019 at 02:58):

Yeah, that's a good question. There's an implication that not all systems will return the same result for the same values already (given variable support for decimal precision), and it would be a backwards compatible change to say that in some future version, sytems could support exponent representation.

Bryn Rhodes (Jul 30 2019 at 02:58):

What do other implementations do? @Ewout Kramer , @Brian Postlethwaite ?

Grahame Grieve (Jul 30 2019 at 03:00):

note that I still only accept non-exponent form in the FHIRPath expressions themselves

Bryn Rhodes (Jul 30 2019 at 03:00):

Funny enough, the Java-based CQL engine behaves exactly the same way :)

Bryn Rhodes (Jul 30 2019 at 03:01):

@Chris Moesel , what does the JavaScript CQL engine do with define Test: ToDecimal('1.00E+00')?

Bryn Rhodes (Jul 30 2019 at 03:01):

Yes, the literal still doesn't support exponents.

Chris Moesel (Jul 30 2019 at 03:13):

@Bryn Rhodes -- the JavaScript CQL engine returns back the Number 1 for that. We pass the string directly into JavaScript's parseFloat function then intentionally limit precision to 8 decimal places.

Ewout Kramer (Aug 01 2019 at 13:37):

Yes, I noticed this too. The FhirPath spec says there are so many issues around exponential notation that it's forbidden. This is only one of the subtle differences between these languages (and CQL). I recently discussed the subtle differences between time/date/dateTime too (allow timezone on time?). So, just like Grahame I have now decided to accept everything "internally" in the part of the API that is model independent, and then there can be language-specific rules. E.g. I can forbid exponential notation in the lexer for FP, even though the engine internally could perfectly well handle it, and so is able to work with decimals WITH exponentials coming in from the FHIR model. On the other hand, the parsers for FHIR forbid timezones on times, which I can still allow in parsers for CQL. (though Bryn told me work is underway to unify this).

Ewout Kramer (Aug 01 2019 at 13:40):

Many of these questions have surfaced while working on the mapping engine. We've fully integrated FP into the mapping language engine, but if FHIR and FP have subtle different rules for say literals, you run the risk of allowing exponential notation in, say, the arguments to function calls in the mapping language, while 10 characters to the left, in a 'where' statement you are not allowed to use it. Not very user-friendly.

Ewout Kramer (Aug 01 2019 at 13:44):

I checked my implementation. It will accept exponential notation while parsing FHIR, but now allow exponential notation when using convertsToDecimal().

Ewout Kramer (Aug 01 2019 at 13:49):

While we're on the subject, is the 'T' part of the DateTime part of the literal, or part of the actual value? Is @2013-04TZ.toString() = '2013-04Z' or '2013-04TZ' ?

Ewout Kramer (Aug 01 2019 at 13:53):

(we know the '@' is just a part of the literal notation in the lexer, is the same the case with 'T'. E.g. compare to Java/C#: (1.2f).ToString() = '1.2', not '1.2f'.)

Paul Lynch (Aug 01 2019 at 13:56):

While we're on the subject, is the 'T' part of the DateTime part of the literal, or part of the actual value? Is @2013-04TZ.toString() = '2013-04Z' or '2013-04TZ' ?

T is a part of the ISO-8601 format e.g. , "2019-08-01T11:33:42Z". I don't know what ISO-8601 would do with a timezone that didn't have a time. (Does FHIRPath allow that?)

Ewout Kramer (Aug 01 2019 at 13:56):

Does toDateTime() require the 'T'? So is '2013-04'.toDateTime() valid?

Ewout Kramer (Aug 01 2019 at 13:59):

 I don't know what ISO-8601 would do with a timezone that didn't have a time.  (Does FHIRPath allow that?)

Yes, the grammar allows it, but must admit that's probably not the final word on it - the spec is a bit unclear about it.

Ewout Kramer (Aug 01 2019 at 14:00):

Ewout Kramer (Aug 01 2019 at 14:03):

It's actually pretty explicit in the grammar:

DATETIME
        : '@'
            [0-9][0-9][0-9][0-9] // year
            (
                (
                    '-'[0-9][0-9] // month
                    (
                        (
                            '-'[0-9][0-9] // day
                            ('T' TIMEFORMAT?)?
                        )
                        | 'T'
                    )?
                )
                | 'T'
            )?
            ('Z' | ('+' | '-') [0-9][0-9]':'[0-9][0-9])? // timezone offset
        ;

The Z + timezone part could have been placed explicitly after the TIMEFORMAT pattern, but instead, it is at the bottom, allowing it after every form.

Paul Lynch (Aug 01 2019 at 14:06):

See https://chat.fhir.org/#narrow/stream/179266-fhirpath/topic/DateTime.20with.20timezone.20but.20without.20time.3F . (I had forgotten about that thread.) I think the consensus was (at least in that thread) that even though the grammar allowed it, it was not intended to be allowed in the spec.

Ewout Kramer (Aug 01 2019 at 14:18):

Well, I could accept that if it made the grammar too complex (like trying to specify a date regex that does not allow month 13), but here in this grammar it is so explicitly allowed....

Ewout Kramer (Aug 01 2019 at 14:20):

Yeah, I did not allow it back then, but I am busy refreshing my implementation, and just took the ANTLR grammar in the latest normative for granted, so I do support it now.

Ewout Kramer (Aug 01 2019 at 14:20):

Well, it's easy to disable again - let's see what the consensus is...

Bryn Rhodes (Aug 01 2019 at 20:41):

Yes, the fact that the grammar allows it is unintentional. The change to the spec was to remove the ability to represent Timezone offset on Date and Time values, on the grounds that specifying a Timezone offset only makes sense for a full DateTime value (which is what FHIR does).

Bryn Rhodes (Aug 01 2019 at 20:43):

So the consensus on that thread was that a tracker should be submitted to correct the FHIRPath grammar to disallow the T for Date and Time (it should only be allowed for DateTime). And we asked for a survey of implementations to try to determine if this was a misalignment between the spec and what implementations were actually doing (so we could apply it as a clarification on the spec).

Paul Lynch (Aug 01 2019 at 20:49):

Does GF#22818 meet that purpose?

Bryn Rhodes (Aug 01 2019 at 20:59):

@Paul Lynch, I think so, I put a comment on there that we should align the grammar with implementations and the intent of the change, to allow a Timezone only on DateTime literals, not on a Date or a Time.

Ewout Kramer (Aug 19 2019 at 15:02):

I have found another slight difference between decimals in FP and FHIR: the grammar for FhirPath allows leading zeroes, while FHIR does not. More indirectly, in FhirPath there is a unary sign operation, so you can say both -4 and +5, but the last is not allowed in the serialization for FHIR.

Bryn Rhodes (Aug 19 2019 at 18:12):

In FHIRPath though, the unary sign is part of the parser, not the decimal literal, right?

Bryn Rhodes (Aug 19 2019 at 18:12):

So there are two differences, FHIR allows a minus sign but not leading zeroes, FHIRPath allows leading zeros, but not a minus sign.

Bryn Rhodes (Aug 19 2019 at 18:13):

But that's really only a difference in the representation, the actual values allowed are the same, yes?

Ewout Kramer (Aug 19 2019 at 19:19):

Yes! It's just very confusing if you're a user of both fhir and fhirpath (which a lot of people are). And for parser writers like me that like to reuse code to get this stuff consistent, it's also not optimal, potentially having separate but almost identical code paths for fhirpath, fhir, mapping language and CQL for parsing basic stuff like integers.

Grahame Grieve (Aug 19 2019 at 19:28):

there's only 2 possibilites, right?

Chris Moesel (Aug 19 2019 at 19:52):

If you're only parsing then can't you just support the superset? E.g., supporting optional leading zeros isn't going to negatively impact parsing FHIR decimals (without leading zeros) or FHIRPath (with leading zeros). That said, if you're wanting to validate or serialize values then it's not as simple (since FP requires a leading zero for values >= 0.0 and < 1.0).

Grahame Grieve (Aug 19 2019 at 19:54):

if you're like me and Ewout, writing a validator

Grahame Grieve (Aug 19 2019 at 19:54):

otherwise you're correct

Grahame Grieve (Aug 19 2019 at 19:55):

but it's ok if writing a validator is a sucky job

Ewout Kramer (Aug 20 2019 at 16:38):

Yeah - I actually did that until now where the dialects differ. I accept the superset, and then have additional specific validation in the FHIR parsers. It's not that it cannot be done, but we're going to keep things simple, right?

Ewout Kramer (Aug 20 2019 at 16:39):

It's also confusing for fhirpath authors that are also creating FHIR instance examples, they need to be aware of these subtle differences.

Grahame Grieve (Aug 20 2019 at 21:05):

that small population will depend on a validator anyway

Last updated: Apr 12 2022 at 19:14 UTC