FHIR Chat · r4b subscription data lineage · subscriptions

Stream: subscriptions

Topic: r4b subscription data lineage

view this post on Zulip Mike Lohmeier (Dec 01 2021 at 16:25):

In the r4b subscriptions spec, how would a subscriber perform data lineage on each resource when using the id only option? For example, I have an analytics subscriber and I need to be able to store all versions of a record accurately so reporting and modeling can accurately do longitudal analysis.

Is the id only example's eventNumber the meta.versionId for a resource?
https://github.com/HL7/fhir-subscription-backport-ig/blob/master/input/fsh/BackportNotification.fsh#L204

Otherwise, would a subscriber need to line up the subscription timestamps and version independently from the FHIR version?

view this post on Zulip Notification Bot (Dec 01 2021 at 17:15):

This topic was moved here from #implementers > r4b subscription data lineage by Josh Mandel

view this post on Zulip Josh Mandel (Dec 01 2021 at 17:22):

With id-only, I don't think there's currently a way to know the version of the resource that triggered a notification -- just the version-independent URL (in Bundle.entry.fullUrl, example here). If you have an id-only notification and you need a history of versions, you should be able to start from the fullUrl and do GET $fullUrl/_history

view this post on Zulip Mike Lohmeier (Dec 01 2021 at 17:53):

cool, thanks. Yeah, that makes sense since most subscribers will just want the latest version anyway and push the logic for needing a specific version to clients that actually need it like a data analytics subscriber.

view this post on Zulip Ged Webb (Dec 13 2021 at 10:53):

Is SSE being considered as a FHIRCAST transport. I prefer SSE over websockets because its much lighter and is better for battery on mobiles, etc

https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events

view this post on Zulip Josh Mandel (Dec 13 2021 at 14:31):

Do you mean this for #FHIRcast ?

view this post on Zulip Gino Canessa (Dec 13 2021 at 17:38):

Hi @Ged Webb , I have had it on my list to test SSE with FHIR Subscriptions for quite a while, but there hasn't been any interest so I haven't pushed on it.

If you are asking about FHIRcast, I think it's a little harder as the participants/events are often bi-directional (e.g., most apps want to be able to update the user's context). That said (and as Josh mentioned), the #FHIRcast people can give a more authoritative answer.

If you are looking at Subscriptions though, there's no reason it can't be designed as a custom channel today. If there's interest/support, the channel could be considered 'standard' in R5 or later.

view this post on Zulip Ged Webb (Dec 13 2021 at 22:44):

Hey @Gino Canessa

I know. I just wanted to get feedback here and FHIRCAST as the architecture overlaps both areas.

Heres what i am doing currently:

Clients pull over HTTP.
Severs push over SSE.

My setup is basically a GraphQL Architecture:

On the Client they make a HTTP request ( REST or Grpahql). When they make the request they indicate if they want a real time view ( aka subscription).
On the Server, the request is served and response sent, and the FHIR Models are watched via nosql Materialised Views and the changefeed goes to whomever has already make a previous HTTP Query . I just keep a virtual router list of who subscribed to what.
Under the hood is a Message broker that runs to make it scale out and allow failures and upgrades of the Servers without any clients having any downtime. Hence why its highly async, and hence why pure unidirectional async network transport for Subscriptions allows things to be decoupled. Makes geo physical load balancing much easier too.

In terms of UX, the GUI / CLI clients get a real time feed, and so we can do real time collaboration and never have old data on clients.

For Other FHIR Servers how do they want to be told ? Webhooks maybe .

Also what about Compliance ? how do i check that our system works with other FHIR systems ?

I am using feature flags quite a bit so that we can innovate but also be compliant.
Presume the headers tell me the things talking to me also ?

Sorry about the avalanche of questions. Its my first post on this forum, and so have a few things that i know i dont know.

view this post on Zulip Gino Canessa (Dec 14 2021 at 17:27):

No worries @Ged Webb , this is a great place to ask questions.

I'll start with saying that's an interesting architecture. I think it can be built in such a way that it is compliant with 'standard' clients and still provide all the functionality you describe.

For compliance, I'll start by noting I'm no authority on the topic =). I believe the main testing tool of interest for REST testing is Inferno. As long as you are following guidance on the RESTful API page, it should be straightforward.

Moving into GraphQL quickly moves out of my experience. There is the relevant GraphQL page, but I do not know if there are any testing tools available. I would suggest the #graphql stream for information there.

On the subscription side, I have more thoughts. Obviously, these are from looking at your description for a few minutes so I'd advise taking it with a grain of salt.

With disclaimers out of the way, I'll start with a little history - the initial versions of FHIR Subscriptions were closer to what you are describing. There was no topic resource, clients simply registered for subscriptions based on queries and the server notified on changes to result sets. Overall, there ended up being enough issues with that design that a major overhaul was necessary (it worked great for some use cases, but did not work at all for others - there's a lot of content about all of that history available, so I won't rehash it here).

With the changes in R5 (and the Backport IG to support R4B), FHIR Subscriptions became topic-based. With that in mind, there's a bit of a tree in deciding how you want to go about it, e.g.:

  • topic rooting
    • topics are content-based (e.g., 'Observations for this patient', etc.).
    • topics are meta-based (e.g., 'Queries for this client', etc.).
  • number of topics
    • single topic (e.g., all queries for this client)
    • multiple topics (e.g., each query for a client)
  • who creates topics
    • client
    • server

And so on..

You should hopefully be able to follow any of the paths and arrive at a working system (that's been a goal we've all worked hard on at least, I cannot guarantee we haven't missed anything).

In terms of UX, so far the push has been Websockets, with the idea that an active client is subscribed to whatever array of topics the client is interested in (e.g., medication dispenses, patient discharges, etc.) with filters established on the subscriptions (e.g., 'medication dispenses' in this ward, 'patient discharges' from this location, etc.). I'll also note that servers can send 'recent' events after a subscription is started in case overlap is desired and that if a server supports the $events operation, a client can also ask for recent events without running a new query. I actually think SSE would be great here, but it hasn't seen the level of adoption that websockets has overall. Defining a custom channel is pretty straightforward, so this is an easy addition.

For other servers, so far the concept has been rest-hook into that server. We have a proposal for a $publish event to help make that consistent as well, but it is a relatively new proposal.

view this post on Zulip Ged Webb (Dec 15 2021 at 08:58):

Thanks :pray:

That all makes perfect sense .

So then it terms of interoperability who decides the topics names and subjects within them ? I can’t get my head around this because this. It’s not like REST where that API is the API.

I can support web sockets, SSE and HTTP/3 Web Transfer and have all 3 working. It’s just a transport proxy on top of the message bus and so easy stuff.

So in terms of another server or client knowing thev topics and subjects, normally there is a Registry to allow Late Binding / IOC ( inversion of control ) style decoupling with discovery.
Maybe that’s the bit I am missing ?

view this post on Zulip Josh Mandel (Dec 15 2021 at 15:58):

There's a built-in API for discovering topics (e.g., GET /SubscriptionTopic?...) and even (potentially, if a server is flexible enough) allowing for clients to define their own (e.g., POST /SubscriptionTopic). Each topic defines trigger events, etc (http://build.fhir.org/branches/R4B/subscriptiontopic).

(The channel types supported by a server aren't currently disoverable through metadata, to my knowledge.)

view this post on Zulip Ged Webb (Jan 03 2022 at 10:03):

thanks @Gino Canessa Ok so its changed with R4B. Anyone got a link to the R5 Subscriptions Spec ?

view this post on Zulip Ged Webb (Jan 03 2022 at 10:04):

Does Zulip allow you to reply to a thread like you can in Slack ?. I cant seem to find that in zulip.

view this post on Zulip Ged Webb (Jan 03 2022 at 10:05):

thanks @Josh Mandel for the Link ! You read my mind

view this post on Zulip Ged Webb (Jan 03 2022 at 10:12):

Just quickly read the http://build.fhir.org/branches/R4B/subscriptiontopic spec. SO the subscriptions can be saved. My initial thoughts are that its like a stored procedure in a DB with a Materialized View and a change feed off the Materialised view. The reason i use the analogy of a DB is because the way the spec if written is such that the subscriptions are stateful in that they can be saved in the System.

view this post on Zulip Ged Webb (Jan 03 2022 at 10:28):

Is there a github repo for the Subscriptions Spec ? Looking for deeper info on this. https://github.com/microsoft-healthcare-madison/team seems to be a decent repo that then links to the other repos in this space ?

view this post on Zulip Gino Canessa (Jan 03 2022 at 16:34):

Happy new year @Ged Webb ! I'll try and get to each of your questions below.

Hopefully I didn't miss anything, cheers!

Last updated: Apr 12 2022 at 19:14 UTC


An HL7 Website -