Stream: implementers
Topic: JSON Manifest Proposal
Grahame Grieve (Apr 20 2017 at 12:23):
All: I've made a proposal around the JSON format here: http://www.healthintersections.com.au/?p=2626
Grahame Grieve (Apr 20 2017 at 12:23):
comments wlelcome
John Moehrke (Apr 20 2017 at 12:44):
Nice explanation. I worry about the 'fragile' nature that you do describe. It presents rich opportunity for unintended collisions. That is two very different extensions choose the same short name. not easy to detect, unless you are exhaustive; and the target audience of this new approach are these that don't want to be exhaustive. I understand and sympathize with them. They are important. But we know the problems of 'Z' segments. This is better, but not much. Possible to require servers be strict, thus detect collisions; but servers can't detect your other 'fragile
Elliot Silver (Apr 20 2017 at 17:25):
How would this proposal handle new attributes added to resources in new FHIR versions? what happens if "trial" is added as a core element? One advantage of the current approach is that extensions and standard elements essentially live separate namespaces.
Elliot Silver (Apr 20 2017 at 17:29):
Similarly, if you use external manifests, what happens if both define the "trial" extension? There isn't the same namespace qualification that the current approach has.
John Moehrke (Apr 20 2017 at 17:30):
Elliot, my same concern. collisions are hard to detect.
Elliot Silver (Apr 20 2017 at 17:32):
Yeah, just reread your post. Exactly the issues you mention.
Elliot Silver (Apr 20 2017 at 17:34):
Also brings up issues of signing content. Is there a canonical form? What happens if we both use the same extension, but I name the attribute foo, and you name it bar? What happens if I do an inband manifest, and you do out of band?
Lloyd McKenzie (Apr 20 2017 at 19:44):
I think of this as being a "proprietary syntax" that only works in a limited context, but where there's a straight-forward transform to a "standard" syntax. When you're working with the proprietary syntax, you're going to want your object model to align with that syntax, which means the reference implementations aren't going to work well/at all. Furthermore, if you're going to do that, people are going to want slices to manifest too - here's my MRN, here's my SSN, etc. Would it not be cleaner to come up with a transform between a profile-driven syntax and the "standard" syntax? So long as the syntax-driving profile were identified in the instance and resolvable, it would be possible for the client to interact with a standard server. And it would mean that systems that wanted to deal with "pure" (non-fragile) FHIR could do so, while those who wanted to look at the data model their particular way could do so. (Heck, they could even have a profile that expressed elements in their language of choice and still be able to interoperate cleanly.)
Grahame Grieve (Apr 20 2017 at 20:02):
as I said in my post, it's fragile. Not at all good for interoperability. You need control over your context to make this safe. But the problem is, lots of implementers have that control over their environment....
Grahame Grieve (Apr 20 2017 at 20:03):
we could extend this to cover slicing but that's a big step in complexity and fragility
Elliot Silver (Apr 20 2017 at 20:06):
If it's not good for interoperability, and only applies for when implementers have control over their environment, why bother? Implementers who aren't interested in interoperability can implement their own FHIR-like thing on their own. They don't even need to bother with declaring extensions -- just add the extra elements without noting they are extensions. Who cares? It's a pre-arranged private exchange anyhow.
Lloyd McKenzie (Apr 20 2017 at 20:07):
But it would essentially then be implementer-specific. "I want this data - oh and by the way, use this serialization that matches my internal data model". It wouldn't be quite that tight, but with a profile-driven serialization, it could be quite close.
Lloyd McKenzie (Apr 20 2017 at 20:08):
@Elliot Silver In the end, interoperability means "implementers who are capable of exchange" - so making things super-easy for implementers enhances interoperability if there's a way to ensure that what's exchanged is consumable by others.
Lloyd McKenzie (Apr 20 2017 at 20:12):
I think if we had the ability to post data to a server that pointed to a rendering profile or request data from a server specifying a rendering profile, that would meet the needs of these simple/limited/(lazy?) systems. @Grahame Grieve, we'd have to decide what level of complexity in rendering was reasonbly for servers to undertake. I just think if we're going to essentially come up with what's essentially a profile-driven syntax, we may as well look at the whole thing. Slicing and "what extensions do I use and how do I know them" are pieces, but so is treating non-supported elements as extensions, treating elements that have been constrainted to x..1 as non-arrays, using local/friendly names, etc. Some of those things will be easy, some hard. But they may as well all be on the table for discussion.
Grahame Grieve (Apr 20 2017 at 20:23):
so there's a big line to cross here - the JSON manifest proposal I made is limited to things that can be resolved automatically with only the instance, and that are simple to compute. And could be supported by the reference implementations no problems. It's *not* a profile driven syntax. That's a real step up in terms of complexity
Grahame Grieve (Apr 20 2017 at 20:25):
As regarding why bother if it's not good for interoperability.... because few of us live in a binary place where FHIR is purely for interoperability, or purely for internal use. Almost always, there's a cross-over between one and the other. Which is why people are using FHIR internally. And the noise about parameters and extensions is ongoing.
Grahame Grieve (Apr 20 2017 at 20:26):
note that I originally came to this proposal from the Parameters resource, seeking to resolve the outcome that cds-hooks community refused to make cds-hooks a FHIR operation because of the JSON syntax
Jens Villadsen (Apr 21 2017 at 17:37):
Still - I find it a bit special to basically include 'part of the definition/schema' as part of the data being transferred. I recognize that it can pave the road for a more dynamic approach if 'spec first' is not what one is looking for. I myself cannot see how this provides value as I'm a 'spec first' kinda guy. While the proposal might make sense to others, I find it redundant and deeming it unnecessary, IMHO.
Abbie Watson (Apr 21 2017 at 19:10):
Looks good to me. I doubt there will be any concerns raised from the Meteor community. We're fine with the existing approach; so glad to see that this proposal is specifically listed as additional, since it's value proposition is a bit net-neutral for us right now. It might help with inbound resources that are using unknown extensions. Would this go onto the base Resource so it gets added to all Resources in one go?
Abbie Watson (Apr 21 2017 at 19:13):
@Lloyd McKenzie - Rather than thinking of them as simple/limited/lazy, it might help to think of them as direct-wire-to-database or wire-to-cache systems (and vice versa). They're simple in the same way that a Cisco router with fibre optics is simple; in that they're optimized for performance rather than trying to normalize into a relational database.
Lloyd McKenzie (Apr 21 2017 at 19:40):
I wasn't trying to be pejorative (except perhaps with that last one - there will be some of those too). I think it's a group we need to support. The question is whether we try to come up with a single solution that will work for everyone or come up with a "translatable" solution for those folks that can be converted to something that works for everyone else - i.e. separate the persistance/manipulation representation from the exchange representation but rather than having everyone figure out how to do that mapping themselves, provide a standardized mechanism that does most of the work.
Grahame Grieve (Apr 21 2017 at 20:37):
there's no more information in the instance than is already there. This is not including part of the definition/schema in the instance
Abbie Watson (Apr 21 2017 at 20:41):
No worries. No offense taken here. :)
Any particular reason why @manifest instead of @extensions? Also, if we're thinking about translatable solutions for persistance/manipulation representation, the two other items that would really help to address are meta.fhirVersion and how the DateTime datatype is handled.
Lloyd McKenzie (Apr 21 2017 at 20:42):
We're still targetting FHIR being "versionless" once normative, so are resisting adding fhirVersion. What are the issues w/ date/time? (Feel free to just link if it's already been discussed)
Grahame Grieve (Apr 21 2017 at 20:43):
@manifest because it's for the Parameters resource too.
Lloyd McKenzie (Apr 21 2017 at 20:49):
My fundamental challenge here is that this syntax is only useful to a client if all inbound content aligns with the particular names that client wants - otherwise they have to go through the work of parsing the manifest, mapping extensions from the regular extension format, mapping unrecognized "manifested" constraints back to the old format, and generally doing a bunch of extra work that's uglier than what they have to do now. And, generally speaking, I expect they won't. Which means they'll only accept inbound data that aligns with their particular manifest - giving us a ton of silos that can't talk to each other. In short, this promises to seriously break interoperable data exchange. And the use-case for it isn't interoperable data exchange - it's back-end processing and persistence. So rather than break the exchange format, why not provide a mechanism to convert between the exchange syntax and a persistenace/processing syntax that's easy to work with and share the libraries that do that work. That prevents silos, but makes it easy for those who want to use FHIR internally in a less painful way.
Grahame Grieve (Apr 21 2017 at 20:52):
The use case is not back-end processing and persistence. It's client/server in a tightly managed environment. Your alternative will be worse, in that it's much more work to bridge the silos. And you're ignoring the fact that already have silos a la cds-hooks. My real intent was to make a better representation for the Parameters resource, since the json form is seriously not good
Abbie Watson (Apr 21 2017 at 21:00):
Most JSON datastores use a Javascript Date() object to natively store date/time. But FHIR's DateTime is basically a string, so as to handle fuzzy/incomplete dates. Pragmatically speaking, boxing/unboxing DateTime is one of only two instances where we modify resources along the data layer. The other is converting between versions. Otherwise, the FHIR resources are fluid between client, server, and database (all using the V8 javascript VM). It would be simple enough to convert to String for the underlying DateTime, except we don't want to lose fast indexing/search along our time vector. So we box/unbox any Date/Time fields.
Lloyd McKenzie (Apr 21 2017 at 21:10):
@Grahame Grieve How is it more work to bridge the silos if everyone uses the "traditional" syntax for exchange - and it's easy to conver to and from the "internal" syntax?
Grahame Grieve (Apr 21 2017 at 21:12):
because the internal syntax will get used for exchange.
Lloyd McKenzie (Apr 21 2017 at 21:24):
Not conformantly if we say that's not allowed - and proprietary syntaxes are used for exchange all the time. At least this would be a semi-priorietary with an automated conversion
Michele Mottini (Apr 22 2017 at 15:32):
Do you rally want to introduce a breaking change in a maturity 5 part of the specs?
Josh Mandel (Apr 22 2017 at 18:37):
@Michele Mottini the proposal isn't breaking in the sense that it's an additional option rather than a change to the presently-defined format. That said, I'm not sure it's a net win for the community to have a second way of doing things (let alone a second way that's easy to trip on by writing non-robust parsers that ignore the manifest property and assume that all property names are fixed, "like in my own environment").
Josh Mandel (Apr 22 2017 at 18:38):
@nicola (RIO) I assume you'll weigh in here when you've had a chance to review :-)
nicola (RIO/SS) (Apr 22 2017 at 21:21):
@Josh Mandel @Grahame Grieve Cool! (i proposed similar solution couple of times :), in next version of aidbox we go even father - we extract Element from StructureDefinition into separate resource and allow add new elements to existing resources, using the same syntax like FHIR specs, but we treat user elements as extensions when converting back into FHIR. In our internal model there is no conceptual difference between FHIR Elements and UserDefined Extensions. I hope next FHIR improvement will be about polymorphic/union elements: {value: {Quantity: {....}}}! We also modified Reference type for internal references: {reference: '/Patient/pt-1'} => {id: 'pt-1', resourceType: 'Patient'} - more structured and convenient representation. If you give me some time in Madrid - i could present this.
nicola (RIO/SS) (Apr 22 2017 at 21:40):
Extensions are unavoidable, new FHIR elements could be considered (developed and introduced) like extensions (if we would unify extensions and FHIR elements). Key feature of interoperability for me - server provides meta-information about information! Wishful thinking: we need solid core - how to define resource and it's elements (it should be minimalistic, well-designed and stable), then using this core - FHIR defines standard resources and elements and vendors/users define their own's - there is clear way from extension to standard - just remove prefix :) like in css: -webkit-radius-border => radius-border
Michele Mottini (Apr 22 2017 at 23:41):
@Josh Mandel : if a server returns extensions using this new format a client expecting the old one would no longer see the extensions. A SMART-compliant FHIR server that switched to this new format will break all the existing SMART client - that will no longer see the authentication URLs extensions. Seems pretty breaking to me.
Josh Mandel (Apr 23 2017 at 02:06):
My understanding is that this behavior would only happen if a client asked for some new content type...
nicola (RIO/SS) (Apr 23 2017 at 05:23):
Best thought, as usually, in the morning: similar to hypermedia driven clients, FHIR clients should be metadata driven (with not hardcoded access to data elements)
Jens Villadsen (Apr 23 2017 at 11:29):
[sound of a coin dropping] - having read the proposal again I can see that I misunderstood it. As I understand it now (from my newly gained epiphany), I can see how it is more 'easy' to read from a Json-client perspective / easier accessible. What I also see is that the representation is changed without losing information. If this is true, a suggestion (in order not to break any existing implementations) could be to add an extra content-type - i.e. : application/fhir+json+manifest. One can then rely on regular http content negotiation to see if this new json representation is supported or not, thereby making the new representation purely optional.
Lloyd McKenzie (Apr 23 2017 at 15:31):
@nicola (RIO) "-webkit-" works as a prefix because you can count the number of relevant browsers in existence using your fingers. The number of FHIR servers will number in the 10s if not 100s of thousands - managing prefixes is going to be a pain - which pushes us to URLs which then starts to look like XML namespaces which I haven't generally seen a lot of enthusiasm for in the JSON space.
nicola (RIO/SS) (Apr 23 2017 at 16:08):
@Lloyd McKenzie yes I understand, but to be honest - it would be -cerner-border-radius or -epic-race ;) That's ok to work with manifest, uris and aliases - but moving broadly accepted extensions into core, should be - just remove prefix.
nicola (RIO/SS) (Apr 23 2017 at 16:11):
That's why I think extensions should be defined using same syntax as ordinary FHIR elements.
Lloyd McKenzie (Apr 23 2017 at 20:22):
Except it wouldn't be "-cerner". It would have to be something like "-com-cerner-border-radius" because we're not going to manage namespaces for hundreds of thousands of organizations and vendors.
Grahame Grieve (Apr 23 2017 at 21:15):
very early on we considered manually assigning namespaces by appliation for FHIR implementers to use as extensions. The community had a pretty strong message about that
Pascal Pfiffner (Apr 24 2017 at 17:43):
(dupe)
Pascal Pfiffner (Apr 25 2017 at 02:38):
Ha, so it wasn't a dupe and I deleted my comment. :P
2nd try: the whole JSON debate usually comes down to interoperability (current format) vs. JSON-purism (i.e. being able to use keypaths to get at a value). My thought was that I'd still expect extensions, like "trials" to live in extensions.trials. So, in Grahame's proposal, would it make sense to have "trials": "renal" live under extension? This would obviously break our current format.
{
"resourceType": "Patient",
"id": "ex1",
"extension": {
"trials": "renal"
},
"@extension": {
"trials": {
"url": "http://example.org/StructureDefinition/trials",
"type": "code"
}
},
"active": true
}
Personally I might prefer something that stays contained in extension, e.g. the following, but this one doesn't follow any established pattern that I'm aware of:
{
"resourceType": "Patient",
"id": "ex1",
"extension": {
"trials": "renal",
"@trials": {
"url": "http://example.org/StructureDefinition/trials",
"type": "code"
}
},
"active": true
}
nicola (RIO/SS) (Apr 25 2017 at 03:16):
@Pascal Pfiffner you could use specific prefix extension for your extension's keys or use extensions in extensions to separate elements, you want to visually distinguish. For me, it's very important, to not create ghetto for Extensions - they are first class citizen :)
Grahame Grieve (Apr 25 2017 at 13:15):
the discussion on this has been interesting but has largely born out my original description: both Lloyd and Nicola want to make extensions first class citizens - but they are worried about very different things, because of a very different perspective
natus (Apr 30 2017 at 07:30):
Hi @nicola (RIO) ,
I am very interested in your reflexion arround reference ( {reference: '/Patient/pt-1'} => {id: 'pt-1', resourceType: 'Patient'} ). I ask a related question here https://chat.fhir.org/#narrow/near/75849/stream/analytics.20on.20FHIR/topic/Apache.20Drill
For some reason, implementing FHIR as a relational model needs allowing joins between resources, and the way FHIR specifies references is not friendly for that.
Can you elaborate on that ?
Thanks
Grahame Grieve (May 01 2017 at 03:37):
have you looked at the way this is resolved in the RDF format?
nicola (RIO/SS) (May 04 2017 at 18:41):
@natus Yes, definitely! In aidbox we convert FHIR JSON into internal representation where reference becomes resourceType & id or absolute uri. We also fix polymorphic[x] into valueString => value: {string: ...} and inline extensions (similar to Grahame last proposal). Transformation is little bit tricky, because requires metadata - i could share algorithm or clojure code - if you would like. Better would be make alternative FHIR JSON format part of standard. @Grahame Grieve may be create technical work group to design it :)?
natus (May 04 2017 at 18:54):
Thanks for your answer.
reference becomes resourceType & id or absolute uri
This solution sounds good. However, I won't use it until FHIR aproves that as an official alternative to deal with. To me implementing FHIR is wether 100% FHIR or 0% FHIR; never between.
nicola (RIO/SS) (May 04 2017 at 19:50):
@natus we also convert back from internal representation into valid FHIR - so for external users almost 100% FHIR, internally 80% :)
Abbie Watson (May 04 2017 at 19:51):
@nicola (RIO) - What kind of datastore are you using? MongoDB? CouchDB? Something that stores in native JSON?
nicola (RIO/SS) (May 04 2017 at 20:27):
@Abigail Watson PostgreSQL with jsonb :)
natus (May 04 2017 at 20:40):
for postgres
I assume the uri you are using is an integer ? Then Encounter.Patient.id is a FK ?
Grahame Grieve (May 10 2017 at 08:55):
@Paul Knapp would like to bring this proposal back to life and discuss it further (after discussion in committee)
natus (May 26 2017 at 19:09):
@nicola (RIO) could you share the algorithm or clojure code ? I am also interested in thanks
nicola (RIO/SS) (May 28 2017 at 09:06):
@natus , I'm working on gist about this transformation - hope publish it next week
Dmitri Sotnikov (Jun 14 2017 at 18:34):
This proposal directly addresses a number of usability issues I have with the current FHIR specification.
Using an unordered collection of extension maps precludes the ability to lookup the extensions efficiently. Using a map keyed on the extension id would address this issue. As it stands, the data needs to be converted into a map before it can be used effectively.
The value* notation is problematic because it prevents you from knowing what keys are present in the map without inspecting it. A standard type key would provide a clean way to select the value key.
The reference proposal makes it easier to tell the type of the reference for the user. It's immediately obvious whether the resource being referenced is local or external, and no parsing is required to query the resource. Again, using a map structure for local references where they're keyed on their id makes the data immediately usable as they can be looked up in the map.
All of these changes would result in a better out of the box experience for the users.
Last updated: Apr 12 2022 at 19:14 UTC
