Stream: subscriptions
Topic: Event Definition
Grahame Grieve (Mar 27 2019 at 02:42):
Several connectathons ago, we looked at tieing subscriptions to event definitions, so that systems could offer a small set of possible subscriptions to consumers, to limit the system exposure to indefinite amounts of processing. What happened with that - anyone still interested?
@Jenni Syed @Isaac Vetter @Joyce Dunlop
Jenni Syed (Mar 27 2019 at 14:00):
I don't think I participated in that directly outside of some discussions around making Subscription more event driven or leaving the definitions more "rest-like"
Jenni Syed (Mar 27 2019 at 14:00):
If there's interest, I could add something to the upcoming connectathon track
Eric Haas (Mar 27 2019 at 14:59):
looking at subscription for Argonauat and will definately look at this as a topic
Pascal Pfiffner (Mar 27 2019 at 18:03):
I hadn't heard of that approach but sounds interesting to me, would be interested to talk about it in May.
James Agnew (Mar 27 2019 at 18:33):
I still think this is a tremendous idea.. Less because of the coded event definitions (which are neat but less useful to me), but I loved the model for using a FHIRPath expression. I'd be in for participating in a connectathon track around it.
Grahame Grieve (Mar 27 2019 at 19:05):
the EHR interest was around capping the workload associated with processing subscriptions. e.g. event + patient. My current subscription model does not scale to 100k+ subscriptions
Josh Mandel (May 02 2019 at 12:21):
Following up around this conversation: we've been thinking about using EventDefinnition as the basis of some of our subscription work in #argonaut this year. "Simple" use cases include subscribing to admission events for a given patient, or for all patients with a given Patient.givenPractitioner. It wasn't clear how to do this with simple EventDefinitions, so I've written up some thoughts here about how to filter a very broad event stream, with a simple/small set of pre-defined filters (each backed by a FHIRPath expression, for systems that know/speak FHIRPath).
Paul Church (May 02 2019 at 13:30):
I think the idea of parameterization is really key to scaling - if there are use cases that genuinely require subscribing to many thousands of arbitrary and unrelated FHIR search queries, I'd be interested to hear of them but they're going to be prohibitively expensive to support. A few parameterized conditions with thousands of instantiations (triggers) is much better.
In your proposal, what gets communicated to the endpoint? It might be very helpful to have a general-purpose endpoint that receives the trigger values in the notification payload.
Paul Church (May 02 2019 at 13:56):
Also, if your conditions can look at both the previous and updated state of the resource, can they trigger on deletions? Whatever syntax handles creates (%previous doesn't exist) ought to also handle the updated state not existing.
Josh Mandel (May 02 2019 at 14:25):
In your proposal, what gets communicated to the endpoint? It might be very helpful to have a general-purpose endpoint that receives the trigger values in the notification payload.
For the Argonaut use cases: just a ping (i.e., a POST with empty payload) meets the basic requirements.
Josh Mandel (May 02 2019 at 14:26):
I think the idea of parameterization is really key to scaling
Agreed; that's why I'm hoping we can get down to a few key filters, parameterized in this way (but I can't tell if you're agreeing with this parameterized approach I'm proposing, or if you're expressing a critique ;-))
Josh Mandel (May 02 2019 at 14:27):
can they trigger on deletions? Whatever syntax handles creates (%previous doesn't exist) ought to also handle the updated state not existing.
Yes, that's right on both counts (either %previous or this may be null)
Paul Church (May 02 2019 at 17:32):
I'm also not sure if I'm agreeing or critiquing, that depends on whether I can come up with a highly scalable implementation of EventFilter evaluation. ;-)
It's a good design in terms of expressive power. I just want to be confident that it's a direction that will really address scalability. Are the expressions on EventFilter.condition limited to "<field> = <parameter>"?
Patrick Cosmo (May 04 2019 at 04:47):
In your example, your subscription is tied to a single patient through the eventStreamParameter:
"eventStreamParameter": [
{"name": "patientId", "value": "Patient/123"}]
Assuming we are monitoring very large numbers of patients, we either end up with a very large number of subscriptions (if there are different endpoints for different patients), or (if they’re all going to the same endpoint) perhaps just one subscription with a very large number of patientId values in the eventStreamParameter list (I’m assuming the patientId parameter can be repeated):
"eventStreamParameter": [
{"name": "patientId", "value": "Patient/1”},
{"name": "patientId", "value": "Patient/2”},
…
{"name": "patientId", "value": "Patient/n”},
]
In these examples, it’s the patient ID that’s the factor that results in a large number of expression evaluations that we want to be sure we can optimize for. Or more specifically, every resource change will need to be evaluated once for EACH eventStreamParameter value, to check if that resource matches that value.
If we’re monitoring 10,000 patients, then we’re going to need to evaluate the EventFilter expression (which is where the eventStreamParameter value is actually used) 10,000 times for every resource change (to see if that change matches any one of the 10,000 patients we’re interested in). There might be some interesting optimizations we can do if we can keep that filter expression simple and make some assumptions about it, but that gets very tricky. Right now, I think that expression could even be changed to “not equals” ("expression": "%current.subject != %patientId”), in which case the list of patientIds becomes the patients we’re NOT interested in (probably no real use case for that, but theoretically possible).
However, if we had a mechanism that could be read as “notify me when <Field> has one of these <values> AND <complicated expression>”, then we could optimize with a pre-filter on the 1st half of that statement and only have to evaluate the <complicated expression> if the pre-filter is passed.
In the patient example, this would read as “notify me when <Patient ID> has one of these <list of patient ID values> AND <complicated expression>”. The pre-filter is whether “this” patient is one of the patients in our list.
This would allow you to build an index that maps field (e.g. Patient ID) to Subscriptions. When a resource change comes in, you could look up this index to find the set of Subscriptions that might be relevant to this change. Which would be only the set of Subscriptions relevant to the one patient, a small set. Then evaluating <complicated filtering expression> is only being done for the patients it needs to be done for, and only once per patient.
One way to do this might be to add a “targetedResources” (better name needed?) list to the subscription, which becomes our pre-filter:
{
"resourceType": "Subscription",
"eventStream": [
{"reference": "EventDefinition/admission"},
“targetedResources”: [
{“field”: “%current.subject“, “values”: [
“Patient/1”,
“Patient/2”,
“Patient/n”,
]
}]
}
The interpretation of this is that for every entry in targetedResources, the specified field must have one of the specified values, or the subscription doesn’t apply. In this case, the patient must be one of those in the list, else you can ignore this subscription.
Jenni Syed (May 04 2019 at 13:46):
For those at the connectathon, we have a breakout scheduled today at 4pm Eastern in Salon 1 to discuss this
Josh Mandel (May 04 2019 at 14:06):
Are the expressions on EventFilter.condition limited to "<field> = <parameter>"?
In general no, though of course a server might impose limitations -- and some servers might just have a fixed list of expressions they support (i.e., just named events, without really using the fhirpath)
Josh Mandel (May 04 2019 at 14:12):
For efficient server implementations: many distinct client subscriptions need to land in and be evaluated by consolidated servers processes -- i.e., a new client creating a new (small/narrow) subscription might update a behind-the-scenes list of patients for which the "by-patient" filter is applied (or any other filter, for that matter).
Jenni Syed (May 04 2019 at 21:48):
Notes from the previous conversation are now on the track page: https://confluence.hl7.org/display/FHIR/2019-05+Subscriptions+Track#id-2019-05SubscriptionsTrack-NotesfromSaturdayBreakout
Jenni Syed (May 04 2019 at 21:48):
The next breakout for this topic is Sunday at 3pm in Salon 6
Jenni Syed (May 04 2019 at 21:49):
@Josh Mandel @Isaac Vetter Feel free to add more context to the notes where it's missing
Michele Mottini (May 07 2019 at 15:42):
I have an alternative proposal: https://github.com/CareEvolution/Public/blob/master/Subscriptions.md
Josh Mandel (May 08 2019 at 04:53):
Will you be at Q1 discussion tomorrow (Wed) @Michele Mottini ?
Josh Mandel (May 08 2019 at 04:56):
Re: your proposal, thanks for writing this up! The following:
that fire when the data set changes = the List change = resources are added or removed from it or any of their associated properties change.
Seems to indicate that there's no way to restrict firing of subscription events to specific events (e.g., specific status changes on the set of resources that belong to the list). Is that right?
Josh Mandel (May 08 2019 at 04:58):
Also, for a list like "Admissions in last 24 hours", where is the constraint of "24 hours" expressed? Just in the description?
Josh Mandel (May 08 2019 at 04:59):
(And otherwise, how do Lists avoid growing arbitrarily large over time?)
Christiaan Knaap (May 08 2019 at 10:26):
In preparation for FHIR-I Q1: The proposal discussed in the Sunday meeting that is linked to from the notes, seems not to be publicly accessible. @Josh Mandel @Jenni Syed ?
Josh Mandel (May 08 2019 at 11:19):
Thsnks -- I'm fixing this
Josh Mandel (May 08 2019 at 11:21):
Hmm, it looks like it should already be accessible here
Michele Mottini (May 08 2019 at 12:40):
@Josh Mandel : no, I am no longer in Montreal, I was there only for the conectathon
Seems to indicate that there's no way to restrict firing of subscription events to specific events
Yes, you can - changes in the data sets (List with properties) are the events, so you can restrict to whatever you are interested in - assuming the server knows how to identify that thing.
Also, for a list like "Admissions in last 24 hours", where is the constraint of "24 hours" expressed? Just in the description?
How the data sets are defined is not specified - so there is no definition there of what count as an admission, or an admission in the last 24 hours. It is assumed that the server knows (more likely it has some internal logic to define those things) and that knowledge is exposed in the data sets. (I am thinking about a way to define data sets using query criteria and paths)
(And otherwise, how do Lists avoid growing arbitrarily large over time?)
Depends on the list - if there is some kind of time limit like 'Admission in 24 hours' they won't grow too much, but of course if you have just 'Admissions' eventually it can be as large as your entire patient population (if it is a pretty sick population...)
Last updated: Apr 12 2022 at 19:14 UTC
