FHIR Chat · Backward compatibility · subscriptions

Stream: subscriptions

Topic: Backward compatibility

nicola (RIO/SS) (Dec 20 2019 at 16:30):

Hello everybody. Can we make a new topic-based subscription compatible with the old one? :) The simplest way to do it - just give it another name TopicSubscription or something like this (and keep old one, maybe with deprecation warn). We should grow non-breaking culture in the FHIR community!

Lloyd McKenzie (Dec 20 2019 at 16:33):

It would still be breaking - as the old Subscription wouldn't be part of R5. We expect things to break occasionally pre-STU. We don't want to be bound to compatibility with design decisions that turn out to have been wrong once we try to implement them.

nicola (RIO/SS) (Dec 20 2019 at 16:59):

At least for implementers, it would be easy to migrate users from old subscriptions to new one - if they can coexists

nicola (RIO/SS) (Dec 20 2019 at 17:01):

I would suggest publishing obsolete resources at least one or two next versions with deprecation notes. That's how backward/forward compat works :)

nicola (RIO/SS) (Dec 20 2019 at 17:03):

Here is an example of deprecation from guys who know how to be backward compatible - http://openjdk.java.net/jeps/277

nicola (RIO/SS) (Dec 20 2019 at 17:08):

Probably we can introduce "deprication" elements into SD and ED like deprecated, shutdownDate etc. @Grahame Grieve @Lloyd McKenzie what do you think?

nicola (RIO/SS) (Dec 20 2019 at 17:08):

And simple rule - never delete, rename etc resources and elements - just depreciate them!

nicola (RIO/SS) (Dec 20 2019 at 17:13):

Here is how Kubernetes handles this https://kubernetes.io/docs/reference/using-api/deprecation-policy/

Lloyd McKenzie (Dec 20 2019 at 17:15):

The point is that there's no expectation of forward/backward compatibility for STU content. That's why we have STU instead of jumping right to normative. We don't want implementers of R5 using the old subscription (or the old anything). We do follow the approach of deprecation once artifacts hit normative. And we do seek implementer feedback on resources that are FMM4+. But for low maturity resources, if we need to turn the resource inside out, we will. Implementers of low maturity resources need to recognize the risk of change and have plans in place for it going forward.

nicola (RIO/SS) (Dec 20 2019 at 17:20):

Can we ask "real FHIR users" - do they really understand and agree with this strategy? Most of them claiming "version hell" because of incompatibility :( Or they should wait couple of years for normative features - this will slow down adoption?

nicola (RIO/SS) (Dec 20 2019 at 17:21):

What we see from "other industries experience" - you can be agile and flexible if you handle changes as first class.

Gino Canessa (Dec 20 2019 at 17:21):

I'm generally quite in favor for non-breaking changes. That said, I think having two mechanisms in the same release that do the same thing but act differently feels like it would be very confusing for adopters.

In my mind, it's weighing the pain in forcing early adopters to move over at R5 (deprecation gives more flexibility for timing, but they still have to do it), vs pain for everyone else has in having both (e.g., if a client is based on R5, they will have some servers one way and some the other).

nicola (RIO/SS) (Dec 20 2019 at 17:23):

I'm generally quite in favor for non-breaking changes. That said, I think having two mechanisms in the same release that do the same thing but act differently feels like it would be very confusing for adopters.

In my mind, it's weighing the pain in forcing early adopters to move over at R5 (deprecation gives more flexibility for timing, but they still have to do it), vs pain for everyone else has in having both (e.g., if a client is based on R5, they will have some servers one way and some the other).

That's how compatibility works! If jvm would not do this or your hardware and OS (32 => 64)- you would throw it away :)

nicola (RIO/SS) (Dec 20 2019 at 17:25):

The rule of thumb - it's better to have a redundancy in features and some garbage than force users upgrade immediately in one transaction

nicola (RIO/SS) (Dec 20 2019 at 17:29):

By providing the mechanism of the coexistence for new and old features you give "such a critical room for maneuver" to seamlessly migrate/upgrade!

nicola (RIO/SS) (Dec 20 2019 at 17:30):

FHIR is gonna be a distributed system standard - you just could not "stop the world" and install newer version!

Gino Canessa (Dec 20 2019 at 17:30):

Yes, but if it was marked beta, you'd accept a breaking change - which is what < Normative is

nicola (RIO/SS) (Dec 20 2019 at 17:31):

Normative looks "too slow"!?

nicola (RIO/SS) (Dec 20 2019 at 17:32):

How much "normative" FHIR do we have now?

Gino Canessa (Dec 20 2019 at 17:33):

Yes it does feel slow.. and if building up to normative requires those additional processes, it will be even slower.

nicola (RIO/SS) (Dec 20 2019 at 17:33):

Why?

nicola (RIO/SS) (Dec 20 2019 at 17:35):

Normative is slow - because people are scared to make a mistake in the decision, they could not change later.

nicola (RIO/SS) (Dec 20 2019 at 17:35):

With a depreciation, backward and forward compat practices - this fear will go away.

Gino Canessa (Dec 20 2019 at 17:36):

Because that's the cost of process? Finding the line of where to do it is mental load if nothing else. During the redesign, we've renamed resources, fields, etc. But, there have been times when it was unchanged for months - what would be the decision there?

nicola (RIO/SS) (Dec 20 2019 at 17:37):

Let's imagine how it can be done with Subscription. You froze and deprecate current Subscription - no work to do this - just keep it.

Gino Canessa (Dec 20 2019 at 17:37):

Also, I'm not advocating that this shouldn't be considered in general - just that it would need to be defined before it could be adopted (and Subscription has already been walked down this path).

nicola (RIO/SS) (Dec 20 2019 at 17:37):

You design new TopicSubscription from scratch and publish in R5

nicola (RIO/SS) (Dec 20 2019 at 17:38):

It's not yet published. So it is not too late

nicola (RIO/SS) (Dec 20 2019 at 17:39):

Or we can add new elements to old Subscriptions to upgrade them to new one - mark obsolete elements

nicola (RIO/SS) (Dec 20 2019 at 17:40):

There only few naming clashes.

Gino Canessa (Dec 20 2019 at 17:41):

There is work, and a lot of it is procedural. What happens to deprecated resources re: technical corrections? What happens if a related resource/dependency/etc changes?
In the case of Subscription, a lot of behavior is changed, so clients also need to be aware of this (e.g., an endpoint/websocket/etc will receive different data)

nicola (RIO/SS) (Dec 20 2019 at 17:41):

And for us as implementers - it would be much nicer to support both of them - than forking implementation between R4 and R5

Gino Canessa (Dec 20 2019 at 17:41):

And for us as implementers - it would be much nicer to support both of them - than forking implementation between R4 and R5

But it forces generic R5 clients to support both - since some servers will continue to use the 'deprecated' version

nicola (RIO/SS) (Dec 20 2019 at 17:41):

Just create an element Subscription.version

nicola (RIO/SS) (Dec 20 2019 at 17:42):

Do you expect this behavior from your browser?

nicola (RIO/SS) (Dec 20 2019 at 17:42):

Handle old html documents? :)

nicola (RIO/SS) (Dec 20 2019 at 17:43):

If client implementation does not support this - users of these clients will do it anyway, because they have to connect to two systems with different versions :)

nicola (RIO/SS) (Dec 20 2019 at 17:43):

I support a couple of clients - users ask for multi-version support all the time

nicola (RIO/SS) (Dec 20 2019 at 17:43):

You do not solve the problem, you make it worse this way

Gino Canessa (Dec 20 2019 at 17:43):

Other way around - if I am implementing an R5 Subscription client, I only implement the new version. Then I go to an R5 server, and it has the deprecated one. I'm out of luck until the server updates, which could be R6 or R7 (using a 2 release example).

Gino Canessa (Dec 20 2019 at 17:44):

All existing clients talking to R4 servers (which they are R4 now) will continue to work

nicola (RIO/SS) (Dec 20 2019 at 17:44):

That's forward compatibility - a little bit harder :)

nicola (RIO/SS) (Dec 20 2019 at 17:45):

Use case: I have to subscribe to epic (R4) and cerner (R5) and aidbox (R6) - what should i do?

nicola (RIO/SS) (Dec 20 2019 at 17:46):

For example, if all of them backward compatible i can do it with R4 client

nicola (RIO/SS) (Dec 20 2019 at 17:46):

If none of them - I need 3 clients and understand, which server on which version

nicola (RIO/SS) (Dec 20 2019 at 17:47):

If my client is multi-version(R4,R5,R6) - I'm lucky as well

nicola (RIO/SS) (Dec 20 2019 at 17:47):

A good example is Browsers and HTTPs

Gino Canessa (Dec 20 2019 at 17:48):

I don't follow. If the old subscriptions mechanism is deprecated, the odds of an R6 server supporting them are minuscule (especially given that one of the primary drivers of the redesign was because servers weren't able to implement it).

Gino Canessa (Dec 20 2019 at 17:48):

Same would be true of a new R5 server - they wouldn't implement the deprecated version, so your client would still need both the R4 version and the newer one.

nicola (RIO/SS) (Dec 20 2019 at 17:48):

In an ideal world, clients are multi-version and servers are backward compatible

Gino Canessa (Dec 20 2019 at 17:49):

So new servers need to implement the deprecated version as well?

John Moehrke (Dec 20 2019 at 17:49):

this is a problem that is caused by people deploying STU ... Normative status is the answer. Trying to solve non-normative as if it was normative is folly

nicola (RIO/SS) (Dec 20 2019 at 17:50):

So new servers need to implement the deprecated version as well?

Users are more important than servers and clients :)

John Moehrke (Dec 20 2019 at 17:50):

that is to say.. your client, because it chooses to support multiple versions, must deal with multiple versions

nicola (RIO/SS) (Dec 20 2019 at 17:52):

Again take a look at Browser and HTTP versions

Gino Canessa (Dec 20 2019 at 17:56):

In an ideal world, clients are multi-version and servers are backward compatible

Yes, that would be ideal. But implementing the old version of subscriptions for production falls somewhere between non-trivial and not-possible (depending on architecture). Asking servers to implement it, even though it was agreed to be broken doesn't seem like a path forward.

So, even if the old version was left in, any client for R5 or higher would need the new version, because servers won't do the old one.

John Moehrke (Dec 20 2019 at 18:12):

Again take a look at Browser and HTTP versions

that is not the same, as those are formal versions. Normative versions. Thus, yes once a portion of FHIR goes normative (e.g. Patient, Observation) then no breaking changes can be made... but resources that are not yet normative MUST be expected to have breaking changes between versions.

nicola (RIO/SS) (Dec 20 2019 at 18:14):

In an ideal world, clients are multi-version and servers are backward compatible

Yes, that would be ideal. But implementing the old version of subscriptions for production falls somewhere between non-trivial and not-possible (depending on architecture). Asking servers to implement it, even though it was agreed to be broken doesn't seem like a path forward.

So, even if the old version was left in, any client for R5 or higher would need the new version, because servers won't do the old one.

:) We implement server

Gino Canessa (Dec 20 2019 at 18:14):

Yes, and was it trivial? :-)

nicola (RIO/SS) (Dec 20 2019 at 18:44):

Yes :) couple of days and month of bug fixing

nicola (RIO/SS) (Dec 20 2019 at 18:48):

But now i do not know - what to do? Make configuration with the version of subscriptions or fork the project. You do not give me options keep my code base together and gracefully migrate my users :(

Gino Canessa (Dec 20 2019 at 18:59):

We're trying - it's hard to balance the needs of a large community. I feel like Subscriptions are particularly painful because they didn't move (forward or backward) for a while.

Lloyd McKenzie (Dec 20 2019 at 19:53):

Having two different ways to do subscription just creates a longer period of non-interoperability. It's no longer enough to know "do you do subscriptions in R5", but now you have to worry about "do you do new subscriptions or old subscriptions"? When you have deprecated content and new content, you're imposing a burden on implementers. They either have to support both approaches or they have to deal the the problem that they can't talk to systems that - based on version and use-case, they should be able to. Deprecation really just means "slower migration" - which in turn means slower ability to actually lock down the standard. Because to lock down the standard, we need significant implementation of the new approach. Systems hanging onto a 'deprecated' approach doesn't help us achieve that.

HL7 has significant experience with maintaining backwards compatibility with stuff that got locked down as 'normative' without much implementation experience and turned out to be broken/sub-optimal. The result is greater implementer inconsistency, increased specification complexity and awkward implementation of use-cases. The notion of STU exists specifically to avoid that issue. With STU, we don't make breaking changes 'for fun'. But if making a breaking change makes the specification more implementable/useable for future users, we'll typically make it. If it's content that's level 4/5, existing implementers can indicate their disagreement/concerns and those will be taken into account.

From an implementer side of things, implementers want everything that works to stay frozen for ever and everything that doesn't work to be changed yesterday. The standards process of draft -> STU -> Normative is the mechanism HL7 has to try to balance those desires. Yes, it takes longer than we'd like. Get something designed, agreed to and implemented across the range of use-cases and jurisdictions our specifications are intended to cover (which for HL7 Int'l is very wide) takes significant time. Consensus and implementation experience are hard work.

We've tried to be very clear and up front about what STU means. In fact, for the first couple of release of FHIR, the section of the spec that talked about STU and the potential for breaking change was the only link in the spec that was bolded. If you think there are ways we can make the risk of change even more obvious, please suggest. But, in my opinion, locking into backward compatibility too soon costs more than the benefit it provides.

nicola (RIO/SS) (Dec 20 2019 at 20:07):

Lloyd do you know how http was evolved and how browsers and web servers work?

nicola (RIO/SS) (Dec 20 2019 at 20:19):

in terms of versions?

nicola (RIO/SS) (Dec 20 2019 at 20:29):

Just imagine you open google and browser says - i can not open this page - server is R4 but I'm R5 - please wait until google migrate to R5. Then you go to Facebook and got server is R6 - could not open ;(

nicola (RIO/SS) (Dec 20 2019 at 20:37):

What is better? Non-perfect working interop with multiple coexisting versions, or partially working interop with incompatible versions?

nicola (RIO/SS) (Dec 20 2019 at 20:42):

Or example from java world. You are using library "a", which works on jvm R4. Now you want to use library "b", which works only with jvm R5? If jvm is not backward compat - you are in trouble. But jvm is usually compatible 5-10 years back - so no problem.

nicola (RIO/SS) (Dec 20 2019 at 20:43):

You can use "a" and "b" together. And eventualy migrate to "a2" for R5.

nicola (RIO/SS) (Dec 20 2019 at 20:45):

But FHIR is "working standard". r2, r3, r4 are already in production

Gino Canessa (Dec 20 2019 at 20:45):

Nicola, I think Lloyd's point is that between HTTP 1.0 an 1.1 (for instance), there were many features which were tried. Some worked, some didn't. What made it into 1.1 had no promise of compatibility with any particular feature in the intermediate time - that's their release process.

The same is true in FHIR, compatibility is preserved once something becomes Normative. Before it is Normative, it is beta and subject to changes. Otherwise, there can be an arbitrary line, but the same will continue to be true (e.g., I don't think you're arguing items at FMM 0 cannot be broken).

Gino Canessa (Dec 20 2019 at 20:53):

I would argue that we (as a community) should be vigilant in making sure things move (e.g., can't sit at FMM 3 for multiple releases). I feel like specs should either move upwards (because they are good) or decay out (because there isn't adoption).

That said, it takes the people involved to make those decisions... and people involved are unlikely to say there isn't enough traction to standardize.

What would be your proposal on when something no longer allows breaking changes (generally, not specific to Subscriptions)?

Note: I will be away from this discussion for a bit, but will check in as able.

nicola (RIO/SS) (Dec 20 2019 at 20:54):

There are countries, which commited R2 and R4. Isn't this enough to care about compatibility?

nicola (RIO/SS) (Dec 20 2019 at 20:56):

My proposal is simple - keep non perfect resources unbroken (just deprecate them). Add new versions with new names.

nicola (RIO/SS) (Dec 20 2019 at 20:56):

It's not so hard to do

nicola (RIO/SS) (Dec 20 2019 at 20:57):

I'm a user of clojure language - no one breaking change in 10 years, even in betta.

nicola (RIO/SS) (Dec 20 2019 at 20:58):

https://youtu.be/oyLBGkS5ICk

Grahame Grieve (Dec 20 2019 at 23:00):

I actually agree with Nicola here on this one. Striking the right balance between breaking changes and other wyas of making changes is a delicate balance.

In this case, we are not 'fixing' something. We are saying 'we don't like the old way subscriptions worked, and we're going to make a new way'. Except that the way we've gone about it also means 'we believe that the old way is so wrong we're eliminating it'.

Yet people are actually using it now, and it works for them. So why eliminate at? We don't need to. We can just define a new kind of subscription, for a new approach, and say, you can keep using the old one.

That obviously raises the concern about supporting 2 different mechanisms going forward, and more work for servers - and it is more work for servers that don't yet exist or haven't yet implemented the old way. Servers that have already implemented the old way, it's little more work to keep the old way about and just build a new one. For servers that haven't yet implemented... that's valid question.

We don't need to deprecate something that's not normative. We can just take it away. but I don't think that's the wise course of action this time.

Vassil Peytchev (Dec 20 2019 at 23:09):

Around minute 34 in the video posted above - how is R5/Subscription different from new-ns/foo from the example given?

Lloyd McKenzie (Dec 20 2019 at 23:56):

If the decision is made that we need to support two styles of subscriptions because there are two parallel business cases, that's fine. That's not a question of backwards compatibility, that's a question of two resources for two different purposes. But if we decided to refactor how Questionnaire is organized and some folks liked the way the old one was organized, we can't keep both of them because that would break interoperability - same business case handled two different ways.

@Gino Canessa is correct. When W3C was working on HTTP 2.0, they went through several candidate releases. People wrote code against and implemented those. But they were under no illusion that the final official HTTP 2.0 would necessarily be compatible across all candidate releases. That's what the STU process is for FHIR.

Gino Canessa (Dec 21 2019 at 00:44):

I would disagree about “fixing” or not. Based on my understanding, many implementers found the old mechanism un-implementable. There were also several used cases not addressed (e.g., delete operations).

There are use cases it works for, but I could draft a Patient resource that works for just my use case too.

I’m open to trying to find a way to map them (e.g., a SubscriptionTopic that indicates old behavior), but the existing servers would still need to come forward with Payloads, etc. to be compliant for clients.

Grahame Grieve (Dec 21 2019 at 01:15):

Based on my understanding, many implementers found the old mechanism un-implementable.

Sure. that's why we made a new one. There's no discussion about that. the question is about the people who did find that it met their use cases.

People wrote code against and implemented those. But they were under no illusion that the final official HTTP 2.0 would necessarily be compatible across all candidate releases

that's true too, but they didn't release 1.0->4.0 before deciding. I'm not saying we can't, procedurally, because we can. But I do believe we shouldn't dismiss existing implementers so summarily.

Lloyd McKenzie (Dec 21 2019 at 03:02):

If the new one addresses all of the use-cases the old one supported, then the old one shouldn't be carried over. Doing so creates two ways of doing things in the same release and creates pain for everyone (new implementers and old). Yanking the old one only creates pain for old implementers.

If the new one doesn't address all of the old use-cases (and can't be adjusted to do so), then it should be adjusted (as much as feasible) to support only the distinct use-cases and should be retained.

nicola (RIO/SS) (Dec 21 2019 at 05:28):

I can describe history of subscriptions (as i see it outside). First version was hard to implement for fhir facades like epic and cerner, because it's too "fhirish", but there was no problems in implementation for generic fhir servers. Topic based subscriptions, moved all fhir dependency to topic (which supposed to be hardcoden in fhir facades) and made subscriptions just channel specification and key/value filter (which can be implemented on any existing queues). For FHIR server and its clients this almost does not add new value (there are some new features, which can be handled in compatible way).

Lloyd McKenzie (Dec 21 2019 at 05:37):

Does the old way offer capability the new way does not? If not, the old way needs to go away. The initial solution may have satisfied an initial subset of the stakeholders but didn't work for the complete set. To support interoperability, everyone needs to move to the solution that can actually work for everyone. If the old implementers choose to stick with the old approach and those that couldn't use the old approach use the new one, then we lose the interoperability the standard is supposed to achieve.

nicola (RIO/SS) (Dec 21 2019 at 05:37):

I understand Lloyd concerns about many ways to do same things, but with deployed to production different FHIR versions this problem does not go away, it just becomes responsibility of users.

nicola (RIO/SS) (Dec 21 2019 at 05:40):

In kubernetes designers make possible multiple versions coexisting in same runtime and it works

nicola (RIO/SS) (Dec 21 2019 at 05:41):

You can migrate to latest version to be "interoperable" in a safe way

nicola (RIO/SS) (Dec 21 2019 at 05:41):

In FHIR clash on resource names makes it tricky

nicola (RIO/SS) (Dec 21 2019 at 05:43):

Even after you do something "normative" - in future you will find the "better ways" doing this.

nicola (RIO/SS) (Dec 21 2019 at 05:43):

and you should be able introduce it in FHIR

nicola (RIO/SS) (Dec 21 2019 at 05:45):

I would say - there should be "one" in a specific point in time recommended "way" to do something.

nicola (RIO/SS) (Dec 21 2019 at 05:46):

And this can change. In few years we can discover another "one recommend way" and deprecate previous

Lloyd McKenzie (Dec 21 2019 at 05:47):

What matters is that a client system can talk to a single interface. When we go normative, we stabilize that interface (at the cost that adding new features/mechanisms may sometimes be clunky). Prior to normative, the interface isn't stable, which means we don't incur the cost of clunkiness for an initial attempt that didn't satisfy the community. Over time, it's possible the klunkiness will build up to the point that the community will say "give us a new clean resource and we'll deprecate the old one". But that's not a place we're in yet.

Lloyd McKenzie (Dec 21 2019 at 05:48):

(And given the cost of the change, the resource would need to have become klunky indeed - for the vast majority of implementers - for an agreement on replacement to occur.)

nicola (RIO/SS) (Dec 21 2019 at 05:51):

Ups, this is a problem of point of view - if we consider fhir as a "working" standard - if R4 published and used isn't this our responsibility to provide smooth migration for our users/early adopters to R5?

nicola (RIO/SS) (Dec 21 2019 at 05:52):

This people does not really care about status of standard - they use it

Grahame Grieve (Dec 21 2019 at 05:52):

actually, it's not a big upgrade from the old to the new - pull the query out into a topic....

nicola (RIO/SS) (Dec 21 2019 at 05:53):

But imagine a working system on top of generic fhir server

nicola (RIO/SS) (Dec 21 2019 at 05:53):

you have to in sync deploy new version of service and fhir server

nicola (RIO/SS) (Dec 21 2019 at 05:54):

and say you have external users of subscription api

Grahame Grieve (Dec 21 2019 at 05:54):

umm. yes. you have to anyway, at the moment

Grahame Grieve (Dec 21 2019 at 05:54):

unless the server offers both R4 and R5 services

nicola (RIO/SS) (Dec 21 2019 at 05:54):

That's mmy point

nicola (RIO/SS) (Dec 21 2019 at 05:55):

Can we design with this in mind?

Grahame Grieve (Dec 21 2019 at 05:55):

what's your point? A general thing about FHIR, or something specific to subscription?

nicola (RIO/SS) (Dec 21 2019 at 05:57):

Real servers should be able to serve multiple versions

Grahame Grieve (Dec 21 2019 at 05:58):

real servers can. I'll check in with an implemenation before the January meeting, but I don't right now see any reason why I couldn't run a dual version server

Grahame Grieve (Dec 21 2019 at 05:58):

(feb meeting this time)

nicola (RIO/SS) (Dec 21 2019 at 05:58):

No one did it in a proper way ;)

Grahame Grieve (Dec 21 2019 at 05:58):

?

nicola (RIO/SS) (Dec 21 2019 at 05:59):

Such breaking changes in FHIR make it a real challenge

nicola (RIO/SS) (Dec 21 2019 at 05:59):

to serve multiple versions

nicola (RIO/SS) (Dec 21 2019 at 06:01):

How to make R4 subscriptions work with R5 patient?

nicola (RIO/SS) (Dec 21 2019 at 06:02):

or R5 subscriptions with R4 patient?

nicola (RIO/SS) (Dec 21 2019 at 06:05):

We are programmers, we can solve this somehow

nicola (RIO/SS) (Dec 21 2019 at 06:06):

but isn't this multiversion reality should be part of fhir design?

nicola (RIO/SS) (Dec 21 2019 at 06:07):

Again i don't see essential difference between JVM and FHIR

nicola (RIO/SS) (Dec 21 2019 at 06:08):

Java guys grew the culture of compatibility.

nicola (RIO/SS) (Dec 21 2019 at 06:08):

If you ever encounter python 2=>3 hell

nicola (RIO/SS) (Dec 21 2019 at 06:09):

or ruby 1.8=>1.9, vb 6=>.net

nicola (RIO/SS) (Dec 21 2019 at 06:10):

you know "the wrong way" of making changes

nicola (RIO/SS) (Dec 21 2019 at 06:11):

I think, version migration problem should be responsibility of "breaking changes" makers.

nicola (RIO/SS) (Dec 21 2019 at 06:13):

if r5 for example will deprecate but keep r4 subscriptions - it would be straight forward to do fhir server with (r4, r5) support

nicola (RIO/SS) (Dec 21 2019 at 06:26):

Make old stuff living deprecated in same place with new looks like right balance/tradeoff

nicola (RIO/SS) (Dec 21 2019 at 06:30):

@Lloyd In case of Questionnaire redesign it is the same problem - imagine a FHR CDR hosted by some provider and dozens of external and internal apps reporting R4 Questionnaire. Now R5 released? How to migrate all of this systems?

nicola (RIO/SS) (Dec 21 2019 at 06:33):

Whose is this problem?

nicola (RIO/SS) (Dec 21 2019 at 06:35):

if we have Q1 (deprecated) and Q2 both in R5 - we can upgrade CDR

nicola (RIO/SS) (Dec 21 2019 at 06:35):

Apps will migrate one by one during the year

nicola (RIO/SS) (Dec 21 2019 at 06:36):

until R6 with Q3 will come :)

nicola (RIO/SS) (Dec 21 2019 at 06:41):

If you don't do this in a standard, implementers still have to do it, because of users.

nicola (RIO/SS) (Dec 21 2019 at 06:41):

Everybody love to design from scratch ....

nicola (RIO/SS) (Dec 21 2019 at 06:54):

(deleted)

nicola (RIO/SS) (Dec 21 2019 at 06:57):

Around minute 34 in the video posted above - how is R5/Subscription different from new-ns/foo from the example given?

the difference is small but important, two namespaces can coexist in one library or runtime. If fhir would make R4/Subscription and R5/Subscription part of R5 then it would be the same. But we drop R4/Sub from R5 as designers ;( and pushing it back (R4/Subscriptions into R5 Server) somehow as implementers

James Agnew (Dec 21 2019 at 12:43):

I have to say I find the fact that "old subscriptions" are going away kind of troubling too.

I know of many, many production solutions that have taken heavy advantage of the current subscription functionality.

There is a workaround for these people, which is to create a contained topic inside the subscription that keeps their old criteria and nothing else.. But what an ugly workaround.

Personally I don't see why it needs to be either/or. You could keep the "old way" alive in Subscriptions and still support the new way. Nobody is forcing servers to actually support both.

Gino Canessa (Dec 21 2019 at 14:19):

I've been mulling this overnight and wanted to consolidate/verify things, if everyone will bear with me..

1) There can be a path from R4 Subscriptions to topic-based, but even then the resources include breaking changes (e.g., what they send as a notification, etc.). Supporting both will require more work than supporting either one.

2) People that have implemented the existing behavior don't want to change their code (which works for them)

3) Outside of the basic "is this implementable for my architecture", several issues/limitations were identified (e.g., resource deletion vs falling out of criteria, content of notifications, etc.) for the wider community - some of these alone are also breaking changes.

4) Subscription sat at FMM 3 for a while, and people don't look at maturity levels, so they used it in production with the expectation of it remaining stable.

--

a) Even if topic-based subscriptions were implemented side-by-side to the existing model, they are inherently incompatible (notifications are different things) - an R5 client would then need to figure out version an R5 sever supports, and expect to implement both for broad compatibility.

b) Versioning resources is a possible design - but moves version hell from the release version to the resource version (e.g., R5 servers support v1/v2/v3 of any given resource/behavior and my client needs to sort it out by supporting the one I want and all the ones below it).

c) If this is the design paradigm we want, it needs to be defined - what is the rule we are looking for? Every resource included in a release needs to embrace backwards compatibility? 'Beta' releases alongside the existing ones so people can test things outside this behavior? Prepend the word 'Draft' to a resource name for testing? etc.

--

So, it will be painful: for people who have implemented the existing resources , for clients trying to connect to servers of a particular release, or for everyone defining specs moving forward (plus one of the other groups).

Josh Mandel (Dec 21 2019 at 15:18):

I'm catching up on the robust discussion here after being totally heads down in SMART documentation yesterday. Wow!

Josh Mandel (Dec 21 2019 at 15:24):

I don't have an opinion on "replace vs augment" yet, but I wanted to make a couple of comments:

@nicola said:

First version was hard to implement for fhir facades like epic and cerner, because it's too "fhirish", but there was no problems in implementation for generic fhir servers.

This wasn't the case -- we received specific feedback from "generic FHIR server" teams (from Google, Microsoft; also @Chris Grenz) that Topic-based subscriptions would give them a scalable path to implementing support, e.g., by taking advantage of existing cloud based queueing services. So we've been thinking of Topic-based Subscriptions as an improvement across architectures.

Josh Mandel (Dec 21 2019 at 15:27):

Second comment is about timing: I wish we were having this conversation earlier. We've done WGM seasons and connectathons and Dev Days meetings and webinars and so on. (Heck, @nicola (RIO/SS) I came to St Petersburg and presented this plan to you two months ago.) It's always harder to process this kind of feedback later in the revision process -- which isn't an argument one way or another, it's just a "here's what I'm feeling" comment.

nicola (RIO/SS) (Dec 21 2019 at 15:50):

@Josh Mandel that was my fault, i didn't put "right" attention in "right" time. I think this discussion not only about Subscriptions, but about how do we handle "breaking changes" in FHIR in general. Some time ago same happened with include:recursive.

nicola (RIO/SS) (Dec 21 2019 at 15:52):

I think, we need a mind shift to honor "backward" and even "forward" compatibility in FHIR community, because FHIR already in production!

nicola (RIO/SS) (Dec 21 2019 at 15:53):

And users are really concerned about this.

nicola (RIO/SS) (Dec 21 2019 at 15:55):

I will try to put my thoughts about this to more structured post.

Vassil Peytchev (Dec 21 2019 at 16:35):

Maybe I am missing something, but can we make sure we understand why R4/Subscription being different from R5/Subscription is worse than having R5/Subscription and R5/NewSubscription? Is it because R4/Subscription cannot reference R5/Observation (for example)? And yes, I understand that R4 and R5 are not really namespaces, but for the purposes of of this discussions, they seem to serve the same purpose.

nicola (RIO/SS) (Dec 21 2019 at 16:53):

@Vassil Peytchev with NewSubscription you can deprecate but keep R4/Subscription in R5. Like in your library you preserve deprecated code for some time to help your users start with a new version right now and gradually refactor codebase to new API.

Vassil Peytchev (Dec 21 2019 at 17:04):

That's what I don't understand. R4/Subscription is not going anywhere - it's still there, and it can be used. What is the exact problem caused by R4/Subscription and R5/Subscription coexisting?

nicola (RIO/SS) (Dec 21 2019 at 17:09):

Name clash of the resource name.

nicola (RIO/SS) (Dec 21 2019 at 17:09):

I would call it Homonym problem

nicola (RIO/SS) (Dec 21 2019 at 17:21):

Imagine you have FHIR server and spent one year and implemented dozen of services which use of R4/Subscriptions - now you want to migrate to R5. FHIR server vendor says you can install R5 and migrate all the data with small downtime, but R5/Subscriptions are different from R4 - so all of your services will be broken (what if you made R4/Subscriptions your public API and some external services use it and they out of your control?).

nicola (RIO/SS) (Dec 21 2019 at 17:27):

You will definitely do some workaround, keep old API, install new API on new URL, make synchronization between R4 and R5 servers, deprecate old API, AND "blame the standard"!

nicola (RIO/SS) (Dec 21 2019 at 17:31):

If you ever did such upgrades - you understand the "pain" of incompatibility. I claim by simple rules we can make the standard more friendly to upgrades/changes - just don't do breaking changes - only "add" and "deprecate".

Josh Mandel (Dec 21 2019 at 17:40):

Would you apply this reasoning to changes in resource definitions/models as well? in other words for non-normative resources would you propose that we avoid changing the structure in a breaking way, and instead define new versions of them that can overlap in time? Or do you see this as a different sort of case because it's about API definition rather the data modeling?

Grahame Grieve (Dec 21 2019 at 19:02):

@Gino Canessa :

Even if topic-based subscriptions were implemented side-by-side to the existing model, they are inherently incompatible (notifications are different things) - an R5 client would then need to figure out version an R5 sever supports, and expect to implement both for broad compatibility.

I don't think that's how clients have worked in the past - it's servers who've done both, and clients just don't work. But yes, if you want general purpose clients and servers, they would do both. if we keep both. The question is where we keep the pain.

If this is the design paradigm we want, it needs to be defined - what is the rule we are looking for? Every resource included in a release needs to embrace backwards compatibility?

That is probably what Nicola is looking for - but that's not what we're going to do as a general rule and we shouldn't be debating a general rule, only what we're going to do here.

Grahame Grieve (Dec 21 2019 at 19:02):

do you see this as a different sort of case because it's about API definition rather the data modeling?

I think it is different in degree of impact

Grahame Grieve (Dec 21 2019 at 19:05):

I'm still not sure that this is quite that big an issue. We've been clear that there is the possibility of breaking changes on the API. So here's an existing R4 subscription:

{
  "resourceType": "Subscription",
  "id": "example",
  "text": {
    "status": "generated",
    "div": "<div xmlns=\"http://www.w3.org/1999/xhtml\">[Put rendering here]</div>"
  },
  "status": "requested",
  "contact": [
    {
      "system": "phone",
      "value": "ext 4123"
    }
  ],
  "end": "2021-01-01T00:00:00Z",
  "reason": "Monitor new neonatal function",
  "criteria": "Observation?code=http://loinc.org|1975-2",
  "channel": {
    "type": "rest-hook",
    "endpoint": "https://biliwatch.com/customers/mount-auburn-miu/on-result",
    "payload": "application/fhir+json",
    "header": [
      "Authorization: Bearer secret-token-abc-123"
    ]
  }
}

Grahame Grieve (Dec 21 2019 at 19:05):

here's the R5 equivalent that has exactly the same functionality, I think:

Grahame Grieve (Dec 21 2019 at 19:10):

{
  "resourceType": "Subscription",
  "id": "example",
  "contained" : [{
    "resourceType" : "Topic",
    "status" : "active",
    "resourceTrigger" {
      "queryCriteria" : {
        "current" : "Observation?code=http://loinc.org|1975-2"
      }
    }
  }]
  "status": "requested",
  "topic" : { "reference" : "#topic"},
  "contact": [
    {
      "system": "phone",
      "value": "ext 4123"
    }
  ],
  "end": "2021-01-01T00:00:00Z",
  "reason": "Monitor new neonatal function",
  "channel": {
    "type": "rest-hook",
    "endpoint": "https://biliwatch.com/customers/mount-auburn-miu/on-result",
    "payload": {
       "contentType" : "application/fhir+json; fhir-version: 4.0",
       "content": "full=resource",
     }
    "header": [
      "Authorization: Bearer secret-token-abc-123"
    ]
  }
}

Grahame Grieve (Dec 21 2019 at 19:11):

given the other kinds of changes that will happen with R5, I don't think this will be particularly noticeable

Grahame Grieve (Dec 21 2019 at 19:11):

• if, that is, the server supports the fhir-version paraemter on the payload

Grahame Grieve (Dec 21 2019 at 19:49):

If I'm right that this is the same functionality, we should add this as an example and make a note on the page about this @Gino Canessa

Josh Mandel (Dec 21 2019 at 20:48):

A difference is on a client side, a client will need to be prepared to handle a bundle, and to receive all posts at the specified endpoint (rather than sub-paths), etc. In other words receiving notifications (not just creating a subscription) requires breaking changes for the client. I think this is part of what Nicola would like to avoid.

Grahame Grieve (Dec 21 2019 at 20:49):

"contentType" : "application/fhir+json; fhir-version: 4.0",

Asking for R4 subscription notification means you get what you used to get

Josh Mandel (Dec 21 2019 at 20:55):

That's an interesting way to interpret :) I would have expected R4 resource content (i.e.,the payload content type would be defined this field) with R5 delivery semantics.

Josh Mandel (Dec 21 2019 at 20:56):

If we were going to support multiple kinds of delivery semantics with a client-settable switch, I might propose using a distinct field for that switch

Grahame Grieve (Dec 21 2019 at 20:56):

but R5 delivery semantics are not possible in R4. Further, the document for fhir-version says that it specifies the exchange semantics

Josh Mandel (Dec 21 2019 at 20:57):

Can you clarify which document?

Grahame Grieve (Dec 21 2019 at 20:58):

from http://hl7.org/fhir/R4/http.html#version-parameter

When used in an HTTP request, the version parameter may be used on either the Content-Type header, or the Accept header, or both, and applies to the entire interaction (the behavior of the interactions as described on ths page, the search parameters and functionality, and the accompanying conformance resources)

Josh Mandel (Dec 21 2019 at 20:59):

Thanks! I'm interested to get @nicola (RIO/SS)'s take on this approach. It seems to split the difference and between "replace vs augment", but I'm not sure it addresses Nicola's concern.

Grahame Grieve (Dec 21 2019 at 21:03):

no but in the sense that it doesn't, that's the bigger question. We aren't going to start enforcing backwards compatibility until we call things normative. Our main focus for R5 is to move things to that state. People can be impatient, but it's a big challenge to get to that state. Compared to other things that are going to change, this is not a big deal.

Otherwise, a lot of people will just stick with R4 completely

nicola (RIO/SS) (Dec 21 2019 at 21:04):

Would you apply this reasoning to changes in resource definitions/models as well? in other words for non-normative resources would you propose that we avoid changing the structure in a breaking way, and instead define new versions of them that can overlap in time? Or do you see this as a different sort of case because it's about API definition rather the data modeling?

Yes - to all released resources and APIs. That's how the "compatibility" mindset can be cultivated.

nicola (RIO/SS) (Dec 21 2019 at 21:13):

no but in the sense that it doesn't, that's the bigger question. We aren't going to start enforcing backwards compatibility until we call things normative. Our main focus for R5 is to move things to that state. People can be impatient, but it's a big challenge to get to that state. Compared to other things that are going to change, this is not a big deal.

Otherwise, a lot of people will just stick with R4 completely

@Grahame Grieve do you know a success story with such approach in any industry? We will always find a better way, even for normative parts. You recommended book - "In the Land of Invented Languages" - following mainline from this book - only real users will make language work, but you will not get them until 'normative' version :)

nicola (RIO/SS) (Dec 21 2019 at 21:15):

You were talking about "working" standard - where normative status is less important than real usage?

nicola (RIO/SS) (Dec 21 2019 at 21:15):

Or I didn't get you right?

nicola (RIO/SS) (Dec 21 2019 at 21:16):

What is the problem to treat all FHIR releases as 'Normative'?

nicola (RIO/SS) (Dec 21 2019 at 21:17):

Or let say 'working'

Lloyd McKenzie (Dec 21 2019 at 22:24):

Implementers are a creative bunch. In the right space, pretty much anything can be made to work. Our objective in the STU phase is to move from our first theoretical stab at what might be workable to something we're pretty confident will be workable long-term. That doesn't mean that we won't occasionally learn things 5 years down the road that will make us wish we'd taken a different approach, but we can significantly reduce the likelihood. We don't want to be in a situation where we have to align forever with the very first (and possibly quite mistaken) initial first try of a spec because a group of implementers happen to have found a way to make it work for a portion of the problem space. We need to keep in mind what's going to be best long term. As implementers increase and the specification matures (level 4/5) we start considering impact of change on those who've already rolled out. And, once we have a degree of confidence in the solution across the breadth of the space, we can lock it down. Placing too high a premium on impact on initial developers (who were warned to prepare for breaking change) rather than on long-term breadth of coverage and ease of use is a trade-off we need to be wary of.

Grahame Grieve (Dec 23 2019 at 06:21):

@nicola (RIO/SS) sorry for the delay.. I'm having a life prior to Xmas

Grahame Grieve (Dec 23 2019 at 06:21):

I thnk that your concerns are very important, but I also think that you're looking at it from the wrong point of view.

Grahame Grieve (Dec 23 2019 at 06:23):

we could indeed maintain backwards compatibility in the API by duplicating things. Yes, we could have Patient1, Patient2, etc. At least for a couple of resources, we'd be up to #3.

I don't know, on the other hand, how that would work at all for some of the API things... we've just made breaking changes to everything. In practice, systems maintain backward compatibility by still maintaining older APIs for those things.... in fact, hold that thought.

Grahame Grieve (Dec 23 2019 at 06:25):

If we did clone resources when their design changes, so that we have Patient1 and Patient2, which are inconsistent with each other... would a client addressing Patient1 see the same set of patients as a client working with Patient2? Or would they be disjoint subsets?

Would an AllergyIntolerance resource reference Patient1? or Patient2? How would that work?

Grahame Grieve (Dec 23 2019 at 06:26):

Early on, we decided that the set of questions that brings up were places we just didn't want to go - supremely messy. If servers want to be backwards compatible, they can do it using the API as a group.

Grahame Grieve (Dec 23 2019 at 06:26):

And, in fact, that's exactly what is happening - we define a new spec, but most servers go on supporting the older versions as well as the new versions.

Grahame Grieve (Dec 23 2019 at 06:27):

your concern is a software concern, not an API spec concern, and all our tools support releases 2,3, and 4, and I still make bug fixes for R2 (made one yesterday). And the reference implementations all support R2 - R4 as well. Further, we provide bi-directional conversion code

Grahame Grieve (Dec 23 2019 at 06:29):

so I think that we do take backwards compatibility extremely seriously. Just like other infrastructure providers. But we do restrict the chaos by limiting the compatability by major version. Maybe an infrastructure software provider like kubernetes has an easier task than we do - I don't know. But I know that FHIR is not an application development framework, it's an interoperability standard, and so it's a different beast

Grahame Grieve (Dec 23 2019 at 06:31):

but I do think that the place at which to ask about compatibility is software. And so if you think that supporting both the different ways of doing subscriptions is a problem for software (servers), then I'm all ears. But I think that the conversion example above shows that it won't be too hard (though I will implement it and check!)

nicola (RIO/SS) (Dec 23 2019 at 08:13):

Thank you Grahame. I will try to incorporate this vision into our "software". We and many people look at FHIR broader than just interop standard, but as "software platform specification" , that's why expectations are so high. I haven't seen yet "true" multiversion FHIR server implemented - but challenge is accepted! ;)

Gino Canessa (Dec 27 2019 at 16:46):

If I'm right that this is the same functionality, we should add this as an example and make a note on the page about this Gino Canessa

Hopefully in my branch today, otherwise next week. I will need people more familiar with R4 to review it then, but in broad strokes:

R4-REST:
Ping was requested by leaving off the payload field, which I believe maps to contentType="application/fhir+json; fhir-version: 4.0" with content="empty". This will be different for the receiver, as the POST is a bundle instead of empty body (will note).

Full Resource was requested by setting the payload field to a valid content-type, which I believe now maps to contentType="application/fhir+json; fhir-version: 4.0" with content="full-resource". This will be different for the receiver as well, since the POST will be a bundle instead of a resource (will note).

ID-only is not available in R4, so there is no mapping for it.

R4-WebSockets:
We are still fleshing out WebSockets in R5 - is there any concern about backwards compatibility/mapping for the protocol here (e.g., the requests I've had were to bring functionality up to par with REST, which makes it generally incompatible)?

Paul Church (Dec 27 2019 at 18:55):

The issue of how and whether to continue supporting R4 Subscriptions is very relevant and unresolved for the Google Cloud implementation. We have something like a multi-version FHIR server - you can't intermingle resources from different versions in one store, but the server is the same. In a DSTU2 store with DSTU2 resources, you can do STU3/R4 API things like PATCH, Patient/$everything with _since, search with _has, etc. Any functionality will typically be available across versions if it doesn't directly conflict with other versions (e.g. DSTU2's conformance statement and _sort syntax).

Our subscriptions support has been sitting in alpha for quite a while, in part because we're not sure if it would be more useful to our users to have an R4 implementation with arbitrary limitations on active subscriptions per store, or force everyone into an R5 style API that we are more comfortable scaling up, or find a way for them to co-exist. So I am definitely sympathetic to Nicola's concerns, because our end goal is to offer a consistent level of API functionality regardless of what resources are being used. But that level has to satisfy certain use cases, and R4 subscriptions don't.

(As a side note, occasionally I get asked to support true multi-version stores with mixed resources. This is technically feasible but we have chosen not to do it - I don't think people asking for it have really considered how difficult it would be to use and how weird it would get.)

Grahame Grieve (Dec 27 2019 at 23:18):

what weirdness have you identified?

Paul Church (Dec 27 2019 at 23:59):

Having searchset bundles containing a mix of versions presents functional challenges even if the client is capable of parsing such a bundle - we'd have to put the version somewhere in entry. If I'm searching for MedicationAdministration on a patient, can I _include both MedicationOrder and MedicationRequest?

References between versions - you can have an R4 Patient referenced by an STU3 Observation and referencing a DSTU2 Practitioner, that's probably easy to handle because those relationships were stable. Can I have a Reference(Any) field or a typed reference extension on an R4 resource that refers to a resource type that does not exist in R4? If I have a 0..* reference field, it could also contain a mix of versions.

Paul Church (Dec 28 2019 at 00:04):

Can I have a contained resource of a different version than its container? What does my capability statement say I support? How do I $export a mix of resources?

Paul Church (Dec 28 2019 at 00:05):

There are so many details that haven't been explored. Really what people are saying is "I am getting data from sources in a variety of FHIR versions, how should I deal with it?"

Last updated: Apr 12 2022 at 19:14 UTC