Stream: implementers
Topic: JSON decimals
Grahame Grieve (May 31 2019 at 00:34):
I don't know if anyone wants to add anything here: http://community.fhir.org/t/why-did-you-break-the-international-json-standard/1367/6
Jenni Syed (May 31 2019 at 19:31):
I know we had to modify our JSON lib to handle this. It wasn't as simple as one would hope, and it is a place I worry many developers will miss
Grahame Grieve (May 31 2019 at 19:41):
well, it certainly should be on the safety page, and isn't
Jason Walonoski (May 31 2019 at 19:42):
To be honest, I routinely ignore this rule in FHIR, because I'm not going to write my own JSON library anymore than I'm going to write my own XML or HTTP libraries. And I don't expect anyone else to either.
Grahame Grieve (May 31 2019 at 19:43):
i don't see why you need to write your own json library.
Grahame Grieve (May 31 2019 at 19:43):
Grahame Grieve (May 31 2019 at 19:46):
most of the discussion on this precedes our use of Zulip but I found these:
- https://chat.fhir.org/#narrow/stream/179166-implementers/topic/JSON.20Format.20for.20Primitives.20.26.20extensions
- https://chat.fhir.org/#narrow/stream/179166-implementers/topic/Quantity.2Evalue.20precision
Grahame Grieve (May 31 2019 at 19:48):
the discussion we had, from memory, we passionate and detailed - should we use string, or number. Those of us with laboratory backgrounds preferred string - some quite strongly - but the app developers strongly preferred number, and felt that it would be very unsafe to use string
Grahame Grieve (May 31 2019 at 19:49):
I recall that when we looked at this, there was far less general acceptance in the wider community that strings would be used when precision mattered. And I still don't see any strong statement about this in any of the JSON specifications
Jason Walonoski (May 31 2019 at 19:56):
i don't see why you need to write your own json library.
Well, I suppose you don't need to write your own JSON library -- but if you deserialize/inflate FHIR objects (in the programming language of your choice) from the raw JSON, and you want to use native floats/decimals/BigDecimal/Number/whatever, and then you want to round-trip this back into JSON, you need to do something to make sure the original value and the new value are identical.
Jenni Syed (May 31 2019 at 20:00):
In most libraries I've used (eg: Java and Ruby) BigDecimal by default serialized to and from String
Jenni Syed (May 31 2019 at 20:02):
And that's essentially the equivalent of what FHIR is specifying
Grahame Grieve (May 31 2019 at 20:06):
right. not all. but most do have some way to preserve precision.
Jason Walonoski (May 31 2019 at 20:06):
There was (not sure it it still exists) an example observation in the spec with a valueQuantity that looked like this:
"valueQuantity": { "value": -1.0e+245, "unit": "g" }
Ruby does not handle that well.
Grahame Grieve (May 31 2019 at 20:06):
what does it do?
Jenni Syed (May 31 2019 at 20:08):
@Grahame Grieve I mean into JSON String and from JSON String. We had to monkey patch ruby to try to parse from and to JSON Number
Jason Walonoski (May 31 2019 at 20:08):
On a roundtrip it failed to produce the same number -- it overflowed, I believe, but I don't recall the specifics.
Jenni Syed (May 31 2019 at 20:09):
I haven't had the pleasure of dealing with that Observation example b/c we currently only send Observation out, don't write
Jenni Syed (May 31 2019 at 20:11):
and I guess technically it's Rails we monkey patched
Jenni Syed (May 31 2019 at 20:11):
:)
John Silva (May 31 2019 at 20:59):
I think I commented on this before. Regardless of HOW we specify numeric quantities in FHIR (JSON, XML or Turtle RDF?) the concern I raised (before) is that there should be a way of specifying the precision expected in a particular use case. In the example of a particular binding to a specific lab measurement, for example, as a developer (and more importantly the clinicians guiding the developers) should be able to say what the expected precision of the lab measurement should be. This is because the measurement is typically communicated with a certain expected precision (and unit of measure -- these are obviously tied together) and the FHIR 'communication channel' shouldn't alter this precision simply because its transport mechanism (e.g. JSON) can't do the job! I think I even pointed to how they do this in OpenEHR where they recognize the importance of precision and unit of measure in their model. Yes, their DV_QUANTIFIED Class: https://specifications.openehr.org/releases/RM/latest/data_types.html#_dv_quantified_class
Vadim Peretokin (Jun 01 2019 at 04:40):
This is one of the major issues I ran into with making my own FHIR server as well. PostgreSQL's JSON support was stripping off the precision. Only solution would be to migrate things between string and numbers. So I agree that it's a problem.
Grahame Grieve (Jun 01 2019 at 04:47):
@John Silva that discussion lead to this: http://hl7.org/fhir/extension-quantity-precision.html
John Silva (Jun 01 2019 at 12:15):
@Grahame Grieve - OK, but an extension again -- not a core functionality? (maybe in R5?) Precision is SO important in healthcare (and folks in OpenEMR recognized this that it's part of their base modeling) that it seems like it shouldn't be an 'afterthought' in an extension. Also, as this is tightly coupled to units of measure, there needs to be a FHIR base data type that pulls these two togehter (like DV_Quantity) it seems to me (IMHO). Also, as this discussion points out about the inherent problems with the 'transports' (if you can call JSON data representation a 'transport') then it seems like this needs to be addressed at the core.
Scott Southworth (Jun 01 2019 at 15:26):
Moving comments into this forum:
I’ve been building a FHIR library that uses the provided XSDs as a code generation source.
The biggest issues I’ve encountered thus far:
1) I’ve had to create a special data structure to facilitate Decimal encoding – and fork the standard JSON decoder to handle decimal types.
2) (off-topic) There’s no version number in FHIR resources (as there was in HL7). Our integration system will be handling multiple versions of FHIR concurrently due to thousands of existing vendor connections.
Technically, I have it working on my end.
My concerns are mostly that deviations from widely used standards could
1) result in medical errors if someone writes an ad-hoc REST JSON in the browser or another stack and loses precision for a field in which it matters.
2) means that highly optimized, secure and trusted standard libraries (like JSON.parse) have to be avoided or reimplemented in many technology stacks attempting to work in this space.
Besides a simple string substitution, the float could be provided with an accompanying precision integer – but, at this point, that seems like a much larger change.
Every developer with a browser hitting the XML service already receives all numbers as strings. Anyone mapping the data with a library against the spec would have something that could auto-convert them from string to a numeric value with an associated precision without abandoning standard encoders/decoders for the JSON spec.
Grahame Grieve (Jun 01 2019 at 22:35):
versioning = sounds like you haven't read the page on versions.
Grahame Grieve (Jun 01 2019 at 22:36):
I'm not aware of any case where losing precision leads to actual wrong results, only to human concern
Grahame Grieve (Jun 01 2019 at 22:38):
@John Silva it's not an after-thought as an extension. We say that the precision is in the string representation, and as far as we can figure out, the few cases where this is not the case are extremely edge case.
there needs to be a FHIR base data type that pulls these two together
You mean, like Quantity?
Grahame Grieve (Jun 01 2019 at 22:41):
I believe that I opposed the original decision to use number not string in JSON, but I was overruled on the basis that I didn't really understand JSON ("too much XML indoctrination"). I can't find the original records though. (problematic, but before we won the battle about where to keep records)
Grahame Grieve (Jun 01 2019 at 22:46):
@Lloyd McKenzie @Josh Mandel @Ewout Kramer I should be able to find records of this in either gForge or the wiki, but I can't. Thoughts on where I should look?
John Silva (Jun 01 2019 at 23:19):
Yes, Quantity has units but it doesn't have precision 'built in'. This statement about the value is very vague:
"Numerical value (with implicit precision)"
What does "implicit precision" mean? Doesn't seem very spec-like (or precise, pun intended ;-) ).
Here's an example of why precision (and units) matter. If you tell me your EMR stores weight in Kg and you (as a clinician) are in a Neonatal unit, the precision better have enough decimal places to property represent the neonate's weight since it is critically important for tiny neonates.
Lloyd McKenzie (Jun 01 2019 at 23:39):
Other possibility would be the old Skype discussion - but I don't have access to that.
Implicit precision = the numeric expression conveys the precision. I.e. 3.0 has a precision of 2. 3.00 has a precision of 3. 3.00e2 has a precision of 3. The only time you need the extension is if you want to send "300" and want to indicate that it has a precision of 2 or 3 instead of 1. (and 3e2 and 3.0e2 isn't an acceptable representation).
Lloyd McKenzie (Jun 01 2019 at 23:40):
Was there a blog about it?
John Silva (Jun 01 2019 at 23:57):
OK, but the problem is that JSON would 'loose' the .0 (when sent as a JSON number). It seems like the precision should be more declarative rather than a 'random side effect' of what happens to be sent.
Josh Mandel (Jun 02 2019 at 01:00):
There's a bit of discussion here
Josh Mandel (Jun 02 2019 at 01:00):
2016-01-07 15:35:01Z Josh Mandel: @James: Indeed, this is a JSON implication that comes with our decision to use JSON numbers rather than strings.
Josh Mandel (Jun 02 2019 at 01:01):
2016-01-07 15:29:45Z Keith W. Boone: @James and others: JSON should be JSON. IEEE Format for numbers should be fine so long accuracy is not impacted. We use JSON and XML to support developers familiar with those formats, not to make JSON and XML consistent.
Josh Mandel (Jun 02 2019 at 01:01):
2016-01-07 15:29:27Z Josh Mandel: You mean having not having your serializer spit it out in exponential notation? That seems fine. But client code should be ready to deal with (parse) either format.
(etc. This was the discussion I remember.)
Grahame Grieve (Jun 02 2019 at 07:04):
I remember more discussion... but I cannot find it.
Grahame Grieve (Jun 02 2019 at 07:05):
@John Silva I think that it's clear but I wrote those words, so I don't mind making it clearer in a future version.
Grahame Grieve (Jun 02 2019 at 07:06):
this discussion is a little frustrating for me - we waited to 6 years to take something normative so that stuff like this didn't emerge. I'm hearing that this is a practical problem, a problem in theory, and that no one is really defending the status quo.
Grahame Grieve (Jun 02 2019 at 07:09):
I believe that our options are:
- add more documentation about the current situation
- add a precision indicator in the json format and .. encourage... people to use it (we did put a precision indicator in the protobuf format for this purpose, since that has no string representation at all)
If we have other options, they haven't occurred to me. I believe that changing to use string instead (or as well as) number is not an option for procedural reasons
Josh Mandel (Jun 02 2019 at 13:30):
OK, but the problem is that JSON would 'loose' the .0 (when sent as a JSON number). It seems like the precision should be more declarative rather than a 'random side effect' of what happens to be sent.
In theory I can say "this depends on your json serializer and parser; a proper implementation might preserve and interpret this distinction"— though that doesn't help in practice
Grahame Grieve (Jun 03 2019 at 07:41):
actually, strictly, having spent more time with the specs, though it's not very well stated, a proper implementation would not
John Silva (Jun 03 2019 at 10:00):
@Grahame Grieve - "a proper implementation will not" -- do you mean a proper JSON serializer would NOT preserve precision? I know our experience is that neither the Microsoft .NET JSON serializer nor the Newtonsoft JSON serializer preserve JSON numeric precision with the x.0 like values. I think Javascript serializers do not preserve it either. (Does anyone have experience with Java JSON serializers?) Regardless, the 'on the wire' or 'in the datastore' representation of numbers needs to have a way to specify intended precision, that just seems to be an important part of how numeric values are communicated in healthcare.
Grahame Grieve (Jun 03 2019 at 10:06):
it depends on how you read the json spec. The json spec says:
JavaScript Object Notation (JSON) is a lightweight, text-based, language-independent data interchange format. It was derived from the ECMAScript Programming Language Standard
Grahame Grieve (Jun 03 2019 at 10:06):
how derived is it?
Grahame Grieve (Jun 03 2019 at 10:06):
the original spec said:
Grahame Grieve (Jun 03 2019 at 10:07):
A number is very much like a C or Java number,
Grahame Grieve (Jun 03 2019 at 10:07):
the current spec says:
This specification allows implementations to set limits on the range and precision of numbers accepted
Grahame Grieve (Jun 03 2019 at 10:08):
if you assume - which people seem to do but isn't actually specified that number
refers to the number type defined in ECMAScript - then that is very fully specified to be an 8 byte floating number that ignores precision.
Grahame Grieve (Jun 03 2019 at 10:08):
bu no such statement is made concerning json. it depends on how much you think 'derived' means
Grahame Grieve (Jun 03 2019 at 10:10):
json schema doesn't further clarify the issue
John Silva (Jun 03 2019 at 10:10):
OK, 'set limits on' -- I suppose even that doesn't mean that it preserves precision, just that an implementation can say HOW many digits (max) of precision it will support. The practical limitation for now seems to be that actual implementations 'out there' do not preserve precision (maybe or because of this vagueness in the spec or 'just because' JavaScript is really a typeless language and that's what JSON is based on.
Grahame Grieve (Jun 03 2019 at 10:12):
right. many don't. It's hard for me to understand why this is only an issue now. Personally, I use my own JSON parsers for the simple reason that I want line numbers in the validator error messages
Grahame Grieve (Jun 03 2019 at 10:12):
and for some strange reason, there's no way to do that with the normal parsers. and it never occurred to me to lose precision (I guess I've been in healthcare too long)
John Silva (Jun 03 2019 at 12:02):
I just did a search for "json serialization preserve precision" and it's interesting to see how many problems people have with this, from some implementations 'adding an extra .0' and more. One useful piece of info was this page on JSON.org which lists (~> 50) JSON implementations in many different languages. http://www.json.org/ (who knows what variances exist in all these different implementations, especially related to precision)
Gino Canessa (Jun 03 2019 at 14:57):
Hi Grahame, re: why is this an issue now: I think part of the issue is that as FHIR becomes more popular, more people are using it in different ways.
Calling JSON.parse('{"val":3.10}') in the major browsers or Node converts the value to a float and loses the precision (3.1).
In the cases of someone developing in those environments, it is far more likely to use the built-in JSON support then to add a dependency for external code to parse JSON (which can be fair bit of work to find, since the environments include built-in support). In these scenarios, the system will work and the developer may never know an issue occurred.
I would say that at the least making a warning page would be useful, so that people new to FHIR do not unintentionally cause incidents.
Ewout Kramer (Jun 03 2019 at 15:19):
I had to tweak (rather than hack) the Newtonsoft Json library for .NET to make it retain precision, but it is doable. It is using the .NET Decimal class which retains the precision - so 3.0000 remains 3.0000. Unfortunately, .NET's decimal will NOT support anything bigger than approx 8x10^28, so as @Jason Walonoski points out I am no longer able to parse all examples in the spec, as there's now an example with 10^245 as a value. Could do that using double, but then precision is not retained. So, I decided to stick with decimal, and remove the example from my unit-tests.
Gino Canessa (Jun 03 2019 at 16:25):
@Ewout Kramer Knowing that, I would be curious:
-Does that feel like a reasonable amount of work for a new developer that wants to use FHIR?
-As you continued to update, are the changes maintainable for newer releases of Newtonsoft?
Michel Rutten (Jun 03 2019 at 16:30):
In general, I wouldn't expect much adoption/support for datatype formats that are not widely supported by popular programming languages and databases, such as enhanced precision or large exponents. Also, I wouldn't recommend anyone to implement their own numeric/datetime storage & operations; way too tricky.
Josh Mandel (Jun 03 2019 at 19:05):
I wouldn't expect much adoption/support for datatype formats that are not widely supported
Does this push you to conclude we should be using strings for numbers @Michel Rutten ?
Michel Rutten (Jun 03 2019 at 19:41):
No, actually the opposite - apologies for bad wording.
Josh Mandel (Jun 03 2019 at 21:00):
Well @nicola (RIO/SS) has been pushing for a fhir+json2
mime type for a while :) This would be one more feature to get right.
Ewout Kramer (Jun 03 2019 at 21:05):
"-Does that feel like a reasonable amount of work for a new developer that wants to use FHIR?"
Well, if you use the .NET FHIR API, this is handled for you - if you are writing your own - I think it's now just a simple setting: https://www.newtonsoft.com/json/help/html/P_Newtonsoft_Json_JsonSerializerSettings_FloatParseHandling.htm
But the default is double, so you'd still have to take care.
Josh Mandel (Jun 03 2019 at 21:09):
And anyone using standards database support for JSON, etc, might not even be able to control this behavior. What does FHIRBase do here btw @nicola (RIO/SS) ?
nicola (RIO/SS) (Jun 03 2019 at 21:49):
Looks like PostgreSQL JSONB preserve presicion - select ('{"a": 3.00000000}'::jsonb)->>'a'
=> 3.00000000
nicola (RIO/SS) (Jun 03 2019 at 21:51):
Internally in jsonb numbers are represented as numeric type - https://www.postgresql.org/docs/current/datatype-numeric.html
nicola (RIO/SS) (Jun 03 2019 at 21:54):
In clojure cheshire library we can force bigdecimals.
Gino Canessa (Jun 03 2019 at 23:42):
I've been going over this with @Josh Mandel and we thought we should loop back out with the larger group.
Background: right now the FHIR decimal has no limits on either precision or range (see GF#22655). The serialization behavior is defined as a union of xs:double and xs:decimal from XML (and listed to be applied to JSON). Otherwise, the value can be an arbitrarily long list of numbers. For reference, the xs:decmial is only guaranteed to have 18 digits of precision, and float is standard IEEE 64-bit float.
The Firely API appears to just use the C# decimal (range ±1.0 x 10^28 to ±7.9228 x 10^28, 28-29 significant digits). Attempting to parse an observation with a value in range for double but not decimal (e.g., 1E31) causes a parsing failure (@Ewout Kramer is this correct?).
In HAPI, decimals are converted to BigDecimal, so it does support arbitrary precision and range.
Additionally, the spec says that decimals may be represented as floats with precision, which would cause rounding errors in many cases (e.g., irrational numbers). In the cases of required precision, this should be unacceptable.
From what we can tell, the spec indicates that systems must be able to perform calculations with arbitrarily large/small and precise numbers, but could also cap serialization to 18 significant digits.
With that in mind, what are the use cases that we are trying to solve? What is the intention for required precision and range with this type that would be clinically useful?
Grahame Grieve (Jun 04 2019 at 00:51):
geolocation is actually the primary use case for very precise decimals. I've not seen observations that are reported to more than 4 significant digits even if the measurements are much more precise, since biological processes rarely justify that - e.g. even your weight and height are constantly changing in the 3rd significant place
Grahame Grieve (Jun 04 2019 at 00:51):
@Ewout Kramer can you find that example? I had a look earlier in this thread and couldn't find it
Grahame Grieve (Jun 04 2019 at 00:52):
I did add some ludicrous values to check my implementations - but I didn't mean for them to slip out into the real world
Grahame Grieve (Jun 04 2019 at 01:04):
I've asked Tim Bray for comment. I'll report back if he says something interesting / useful
Kenneth Myhra (Jun 04 2019 at 09:16):
observation-decimal is the one with 10^245 as a value, it's still in the example downloads
Grahame Grieve (Jun 04 2019 at 11:21):
Ok thanks. I’ll fix that
Jason Walonoski (Jun 04 2019 at 13:36):
Ok thanks. I'll fix that
How? Just removing it isn't really a fix -- in my opinion -- if it is actually a valid example. Removing it would just mask the issue.
Grahame Grieve (Jun 04 2019 at 13:36):
it's not really valid. It was just an edge case I added for testing implementations. I've removed it to 1e-24 instead of 1e-245
Josh Mandel (Jun 04 2019 at 16:32):
What do you mean by "not really valid"? Picking something like e-24 (which is in the decimal range for c# I think?) means we're not testing ability to handle doubles.
Lloyd McKenzie (Jun 04 2019 at 16:38):
I think he means "for which we have a valid use-case" - i.e. a number that could be expected - even in an edge case - to appear in a real healthcare resource.
Lloyd McKenzie (Jun 04 2019 at 16:39):
e-24 is on the molar mass scale, so is theoretically possible. 10^245 is well beyond the countable atoms in the known universe - so not a number we could expect to see a healthcare application sharing.
Josh Mandel (Jun 04 2019 at 17:08):
Okay --and I'd say if we want to limit our spec, we should limit the spec (in addition to the tests/examples).
Gino Canessa (Jun 04 2019 at 18:16):
If we can limit the precision (10 digits would allow CT-slices to be individually geo-tagged by lat/lon) and the range based on values expected in a clinical setting (e.g., the 10^24 from above), then we have a type definition we can hold everyone to.
In Java and C#/.Net, having unquoted precision works well (having addressed concerns about blowing up parsers with doubles vs decimals).
Looking at JavaScript, JSON support has been part of the language since 2009 so external parsing libraries have begun to rot. Dusting one off, I was able to modify it to parse numbers as strings (or decimals, bigdecimals, etc.) and keep precision. This modified parser results in the expected behavior, but I'm not sure how realistic it is to expect in a production setting (forked and modified ~10 year old unmaintained code).
Looking at Python, JSON has been built-in since 2.6 (also 2009). The only library with code I found in a quick search was simplejson - which is external version of the internal library used in Python. It is more complex to modify, but I assume it is doable. Otherwise, the behavior is the same - decimals are automatically rounded.
I was going to check Swift, but don't have access to a box with it right now.
I would guess that the question then becomes (after defining precision and range), what to do about the JSON format issue. Knowing that the built-in support in two languages I've already tested fail to maintain precision is worrying.
Michele Mottini (Jun 04 2019 at 18:35):
what to do about the JSON format issue.
Nothing. It is normative. Cannot and should not be changed.
Gino Canessa (Jun 04 2019 at 19:10):
Nothing. It is normative. Cannot and should not be changed.
I have to disagree about doing "nothing".
Action items such as adding information/warnings to the "Getting Started" or "FHIR Overview - Developers" are relatively painless and could prevent issues in patient care/safety. These items could fall under safety recommendations or best practices for popular languages.
Packages provided by the FHIR group (e.g., https://github.com/FHIR/fhir.js ) should probably be updated to ensure safe handling of data.
These are potential items I thought of in a few minutes [edit], I feel like it would be beneficial to give this some thought.
nicola (RIO/SS) (Jun 04 2019 at 20:49):
what to do about the JSON format issue.
Nothing. It is normative. Cannot and should not be changed.
but we can create another one! without breaking existing format
nicola (RIO/SS) (Jun 04 2019 at 20:50):
And let the future decide, which is more viable
John Silva (Jun 04 2019 at 21:14):
FYI, earlier I posted this link to http://www.json.org/ that has many (seems like > 50) languages, more than most have even heard of ;-) -- 8th, OCaml, Pike, etc. This webpage lists the many different JSON serializers that are 'out there'; wonder how they handle precision (or not)?
Gino Canessa (Jun 05 2019 at 00:55):
Hi John, I did see that post the other day.
I've been going back and forth on this internally for some time now. On the one hand, as long as we communicate the behavior to someone using an obscure language, they should be able to handle it (a danger of using an obscure or new language). On the other hand, any language adding JSON support is likely going to do so in the ECMA standard way - which we are not. Swift 4, for example (which I checked this evening) also loses precision on parse.
If we agree that the precision is clinically relevant and the default behavior causes loss of precision, it is quite likely that an incident will occur. I am particularly concerned about JavaScript in that it is very widely used, has a low technical bar to entry, and uses "standard" JSON internally.
I would like to determine what steps are needed to mitigate or eliminate that before an incident occurs.
Grahame Grieve (Jun 05 2019 at 05:28):
Comment from Tim Bray:
Well, I understand the problem, but you're not going to get much help from JSON. The spec language in https://tools.ietf.org/html/rfc8259#section-6 just talks about "numbers" and points out that trying to interchange anything not representable as an IEEE double is dangerous. There are many improvements that could be made to JSON, including timestamps and smarter numeric types, but they're not going to happen because JSON is in such universal use that everyone is terrified of any changes whatsoever no matter how small.
So, if you have a numeric quantity in a JSON text, for example your Potassium in serum, you're going to have to find another place to express that semantic for consumption by software. It's an interesting problem; suppose I wanted to do some fairly large-scale arithmetic on a bunch of Potassium numbers, e.g. statistics. I assume the proper way to do that would be with normal double-precision floating-point to avoid all the nasty error-accumulation scenarios that can happen if you don't, and then you'd present the result with one digit past the decimal point. So I guess I'm agreeing that this is a presentational requirement, not an arithmetical one.
Anyhow, I suppose that it's up to the FHIR specs to say "This quantity must always be presented for human consumption thus and so." I think that, to be safe, you'd also want to express the fact that for purposes of arithmetic, it's a floating-point quantity and should be handled with that in mind. It would be nice if JSON had a way to express that portably, but it doesn't.
Good luck!
Grahame Grieve (Jun 05 2019 at 05:29):
... which is pretty much what we do....
John Silva (Jun 05 2019 at 10:10):
TIm's comment, "you're going to have to find another place to express that semantic for consumption by software." is what I've been suggesting. If we need to convey the semantic of precision, the it has to be declarative in FHIR (i.e. in the datatypes) and not dependent on the (JSON or other) transport or encoding. I also like his point about, "nasty error-accumulation scenarios", that is one of the main issues with maintaining the proper precision in measurements and calculations based on those measurement --- error accumulation. Of course FHIR's job is not a 'calculation engine' but it needs to have a mechanism so that consumers of the data do not loose precision because of the FHIR store.
(I like this Wikipedia article about accuracy and precision that points out the problems of maintaining and computing with attention to precision: https://en.wikipedia.org/wiki/Accuracy_and_precision )
Grahame Grieve (Jun 05 2019 at 10:15):
we don't agree that it needs to be in the datatypes. The fraction of cases where precision is different to the string representation is very small, and all the cases we found were edge cases. We can add precision elements for formats that don't support string representation of decimals - which may or may not include JSON (but does definitely include protobuf).
Grahame Grieve (Jun 05 2019 at 10:16):
I don't know how doing anything about this in JSON will play out
John Silva (Jun 05 2019 at 10:22):
OK, not in datatypes (though I'm not comfortable with that, I suppose it's too late to do anything about it.)
A somewhat related question, when someone profiles an observation for a particular use case, e.g. a vital sign with a specific LOINC code, does that also imply a specific unit of measure and therefore a certain precision? If not, (LOINC code doesn't imply UofM) then different UofMs can cause problems with precision loss, e.g. Kg vs gram or mg vs ng, etc.
Josh Mandel (Jun 05 2019 at 12:51):
Well, one difference is that Tim says
to be safe, you'd also want to express the fact that for purposes of arithmetic, it's a floating-point quantity and should be handled with that in mind.
Whereas our specifications seems to say that you should treat it as an arbitrary precision decimal value. (Or do people disagree with this characterization?)
Gino Canessa (Jun 05 2019 at 15:31):
Given that we're almost a week into this thread, I would like to ask which way people are leaning on the various items in it. I am trying to list everything that looks actionable so each item can either be moved forward or left for dead.
This is not formal. I just feel there has been enough discussion that I would either like to start on proposals or move on. At the same time, this thread is too long for me to determine what people feel is worthwhile and thought this could give a quick measure.
Thanks!
Gino Canessa (Jun 05 2019 at 15:31):
/poll Should there be additional language for developers to be aware of the JSON decimal encoding behavior (e.g., "Getting Started", "FHIR Overview - Developers", Best Practices, etc.)
Yes
No
Gino Canessa (Jun 05 2019 at 15:31):
/poll Should there be clarifying language added to the decimal datatype for explicit precision and range requirements
Yes
No
Gino Canessa (Jun 05 2019 at 15:31):
/poll Should reference implementations be updated to retain precision by default on platforms where it is currently lost (e.g., fhir.js)
Yes
No
Gino Canessa (Jun 05 2019 at 15:31):
/poll Should there be a new extension to provide precision information generally for decimals
Yes
No
Gino Canessa (Jun 05 2019 at 15:31):
/poll Nuclear options
I see no action items for anything in this thread
I'd like to change the normative JSON encoding
I'd like to add a new JSON serialization model
Michele Mottini (Jun 05 2019 at 15:58):
Should reference implementations be updated
What happens to reference implementation cannot really be decided here - it is up to their maintainers. If someone feels that they need changes the way to go is open issues in their repos (or - even better - pull requests)
Josh Mandel (Jun 05 2019 at 16:50):
That's fair, but lots of those maintainers are here listening and answering questions, so the straw poll is still helpful; and they'll be interested in what "best practice" advice we provide in the spec.
Grahame Grieve (Jun 07 2019 at 04:33):
I added We should consider adding a new property in json for precision
Grahame Grieve (Jun 07 2019 at 04:34):
@Josh Mandel I disagree with your characterisation. The specification makes clear that we have a presentation issue, not an issue with the value itself
John Silva (Jun 07 2019 at 09:59):
@Grahame Grieve - by "adding a new property in json for precision" do you mean to the datatype (Quantity) definition so that it works for both JSON and XML (and any other representation like Turtle RDF)? If so, I'm all for it! (it could look like DV_QUANTITY that OpenEHR defines)
Also, I'm not sure about the presentation vs value issue -- FHIR is not about UIs so I'm not sure how this is a presentation issue. To me it feels like more of a transport issue (thinking of JSON in FHIR as a transport mechanism) or storage mechanism but since FHIR doesn't prescribe FHIR server persistence mechanisms I suppose it can't be that. (though many implementations that use noSQL approaches end up storing 'raw JSON' in which case any intended precision is lost or at least not specific -- i.e. by itself how do I know that 10 is supposed to be 10.0 if I'm doing calcs based on JSON stored numbers?)
Grahame Grieve (Jun 07 2019 at 10:07):
there's no reason to add it to XML and I would be very opposed to that. Nor to RDF. We already added it to ProtoBuf, since that uses raw decimals with no string format. So we could add it to JSON though we could not make it mandatory
Josh Mandel (Jun 07 2019 at 14:55):
Can you explain / show an example of what you mean @Grahame Grieve re: json property? And agreed with @John Silva this isn't a presentation issue, only -- it seems like an issue that could cause errors to accumulate (or at least propagate incorrectly) in automated calculations, if numbers aren't handled properly
Gino Canessa (Jun 07 2019 at 15:54):
I added
We should consider adding a new property in json for precision
If I understand, I think it would go a long way towards resolving this. Are you suggesting that:
{ "val" : .100, "another" : 1.0 }
could be serialized to something like:
{ "val" : .100, "val_precision" : 3, "another" : 1.0, "another_precision" : 2 }
The "_precision" field (or whatever was agreed to) would be a field optionally generated during serialization.
It would then be on the client to either parse the original value exactly or use the additional precision information to reconstruct it.
Is that correct?
Chris Moesel (Jun 07 2019 at 16:10):
I'm not sure that would solve the problem... Off-the-shelf JSON parsers wouldn't know anything about that convention, so they'd still process the field as a regular Number
. I think you'd need dual fields -- one w/ the normal number (for backwards compatibility) and one with the number as a string (to retain precision). This could be accomplished today via extensions, but perhaps Grahame was thinking of a more streamlined special-case approach.
Chris Moesel (Jun 07 2019 at 16:12):
Or perhaps I missed your point. Now that I look at it again, I suppose that the client could still use an off-the-shelf parser and then use the extra precision information to round the Number
appropriately using a non-float representation -- which would hopefully take care of any floating point issues.
Michele Mottini (Jun 07 2019 at 18:46):
'raw JSON' in which case any intended precision is lost or at least not specific
This is not true - if I create an observation with value 3.500 (that specifies the precision) and serialize that using the .NET library I get: {"resourceType":"Observation","valueQuantity":{"value":3.500}}
- and if I parse that I get back 3.500 as a fixed point decimal - so JSON can express the precision just fine - the issue is with the parsers (or the underlying languages not supporting fixed point decimals)
Michele Mottini (Jun 07 2019 at 18:47):
(and a _precision
field is pointless)
Grahame Grieve (Jun 07 2019 at 18:58):
well, the issue is that JSON is imprecise, so that behavior depends on the libraries choice - they are not wrong to choose that.
@Chris Moesel yes that use of precision is what I had in mind based on the example Gino gave. An alternative would be:
Grahame Grieve (Jun 07 2019 at 19:00):
{ "val" : 0.100, "val_formatted" : "0.100", "another" : 1.0, "another_formatted" : "1.0" }
or
{ "val" : 0.1, "val_formatted" : "0.100", "another" : 1, "another_formatted" : "1.0" }
Grahame Grieve (Jun 07 2019 at 19:00):
I'm not sure which I prefer. we could define it, but we couldn't make it mandatory
Paul Lynch (Jun 07 2019 at 19:46):
or
{ "val" : 0.1, "val_formatted" : "0.100", "another" : 1, "another_formatted" : "1.0" }
That one would work well for JavaScript, which does not keep track of trailing zeros.
nicola (RIO/SS) (Jun 07 2019 at 19:59):
If JSON parser supports precision, let's provide it as well {"val": 0.100, "val_formated": "0.100"}
; Puting all these attributes on the same level again create similar problems as primitive extensions and choice types :( Another representation hack can be "val": [0.1, "0.1000"], but probably to solve all of these problems it should be like this {"attr": {"value": ....., extensions, presicion}}
Lloyd McKenzie (Jun 07 2019 at 20:17):
We can't break the existing JSON syntax. Whatever we do has to remain valid against that. Introducing a new JSON syntax would effectively mean everyone would need to support both, so that's not really on the table.
Paul Lynch (Jun 07 2019 at 20:47):
From the perspective of processing or generating a FHIR resource, I don't see much difference between having to handle new versions of FHIR (handling differences between STU3 and R4) and handling a new version of the JSON format. I don't see why an "R6" couldn't introduce changes to the JSON format to handle this and other problems. It would just a part of the pain updating to R6.
Chris Moesel (Jun 07 2019 at 20:47):
Or to be more consistent w/ the current FHIR/JSON serialization approach:
{ "val" : 0.100, "_val" : { "formatted" : "0.100" }, "another" : 1.0, "_another" : { "formatted" : "1.0" } }
This would probably be less impact on implementors since they already need to check for the existence of "_val"
anyway to support ids and extensions.
Chris Moesel (Jun 07 2019 at 20:49):
I think we can't change the JSON syntax in R6 (or R-anything) because the JSON serialization is now normative. Breaking changes are off the table. We can only do additive non-breaking changes.
Lloyd McKenzie (Jun 07 2019 at 23:19):
@Paul Lynch The difference is that in R4 we moved certain things to normative. While there will indeed be pain in moving from R4 to R5 for all of the things that aren't normative, the whole point of normative is that at that point (for those things) the pain stops. (The eventual objective being that the pain stops for everything.)
Paul Lynch (Jun 10 2019 at 14:35):
@Lloyd McKenzie That is a helpful explanation. However, if issues begin to pile up with something called "normative", there must be some point where it is better to the fix the problems than to preserve stability. (Perhaps we haven't reach that point yet.)
Lloyd McKenzie (Jun 10 2019 at 15:53):
It would have to be badly broken indeed for the entire community to agree to the transition. It's not something we have plans to do and certainly not something we'd entertain lightly. Our general expectation is that once something is normative, any issues get worked around with changes that align with FHIR's interversion compatibility rules and/or extensions.
Mona O (May 05 2021 at 12:36):
For Money.value the description states with implicit precision does that mean precision is implied based on currency code? For monetary amounts is it required to include trailing zeros? Looking at various examples within FHIR there doesn't seem to be a consistency. Are these considered all valid based on currency code of "USD" : 15, 15.0, 15.5? Or should they be reflected as 15.00, 15.50?
Lloyd McKenzie (May 05 2021 at 15:53):
Hmm. I think money generally has an implicit precision of 2 decimal places. I.e. If you say $15, that most of the time, $15.01 would be wrong. (Though there are contexts where rounding happens and $15 actually represents $14.98 or $15.23.) @Grahame Grieve, do you think we need any special guidance around precision when it comes to monetary amounts?
Grahame Grieve (May 05 2021 at 20:17):
I don't think we want to say anything about it, honestly.
John Silva (May 05 2021 at 20:27):
Hopefully YOUR bank doesn't use only 2 digits of precision (and keep any fractional difference) :wink:
(many financial transactions of course use multiplication and division which can introduce imprecision and round-off -- yeh, probably always in the favor of the financial institution! )
Daryl Moehrke (May 05 2021 at 21:10):
From what I know, most err on the side of the customer as they can get in big legal trouble and fractions of a penny is much cheaper then the legal fees, but that's getting off topic
Grahame Grieve (May 05 2021 at 21:17):
right. it's a complicated area, but at least in interop, there's no transferring fractions of a cent. However: the less we can say about this, the less we can get in trouble for what we say
Last updated: Apr 12 2022 at 19:14 UTC