Stream: implementers
Topic: subscriptions
nicola (RIO/SS) (Nov 28 2016 at 13:47):
What is the conceptual difference between messaging & rest hook? Where is http long-polling, most used & robust technology for event driven notifications in web? For web-sockets and long-polling - creation of subscription could be part of communication (i.e. first phase), so we do not need to coordinate two operations on servers.
nicola (RIO/SS) (Nov 28 2016 at 13:52):
For REST hooks additional security configuration & workflow should be specified (like signatures or message verification).
John Moehrke (Nov 28 2016 at 13:59):
The main difference in my understanding is that in the case of Messaging, the configuration of the trigger events is out-of-band... Where in rest hook they are inline. -- Security should be considered equally for both, risk based.
Grahame Grieve (Nov 28 2016 at 18:43):
do we need to specify anything for long-polling? I think not...?
as for additinoal things about REST hooks.... what else do you think we need to specify?
nicola (RIO/SS) (Nov 28 2016 at 21:38):
For example, for long-polling it could be like this - POST with subscription resource will be kept until new event with update message response or timeout. Ephemeral Subscription resource will be created during long-polling request.
nicola (RIO/SS) (Nov 28 2016 at 21:42):
Or post with long-polling subscription will return long-polling url with offset.
nicola (RIO/SS) (Nov 28 2016 at 21:44):
For REST hook - secret token could be added to subscription.
Grahame Grieve (Nov 28 2016 at 21:46):
but why not just make the query?
Grahame Grieve (Nov 28 2016 at 21:46):
as for secret token, how would that work? be part of the URL? a header? - already both supported
Brian Postlethwaite (Nov 28 2016 at 21:46):
Looks like I missed getting the refactoring of the endpoint resource into it too. (don't recall if I created a task)
nicola (RIO/SS) (Nov 29 2016 at 05:48):
@Grahame Grieve, do you mean just do read, search or history with long-polling parameter?
nicola (RIO/SS) (Nov 29 2016 at 05:51):
For REST hook, I think, secret should not be exposed for read/search i.e. masked or hidden.
Grahame Grieve (Nov 29 2016 at 06:19):
yes that's what I meant.
Grahame Grieve (Nov 29 2016 at 06:19):
does it work>
Grahame Grieve (Nov 29 2016 at 06:20):
for rest hook, how not to expose it? don't understand?
nicola (RIO/SS) (Nov 29 2016 at 07:12):
Then we need new parameter to say it's "long-polling" request and some capability in conformance, and markers for search parameters (could be used in "long-polling")
Grahame Grieve (Nov 29 2016 at 09:19):
how would you typically use long polling?
nicola (RIO/SS) (Nov 29 2016 at 10:34):
We've implemented secure messaging using long-polling: for every subscriber there is current offset (stream) in event/message log, client (subscriber) do long-polling request with current offset, if server offset is bigger, then it respond instantly with new data or hangs this request until incoming events (messages), respond includes next offset. Client is responsible to reconnect and manage offset. (this pattern comes from popular kafka message queue pattern - https://kafka.apache.org/documentation). Looks like the same pattern could be used for robust subscription stream.
Grahame Grieve (Nov 29 2016 at 10:38):
well, how would you imagine that? A history operation with a wait=true parameter?
nicola (RIO/SS) (Nov 29 2016 at 10:38):
Yes
Grahame Grieve (Nov 29 2016 at 10:41):
so that's kind of a useful thing to add, but not equivalent because it doesn't have the search.
Grahame Grieve (Nov 29 2016 at 10:42):
harder for me to imagine it on a search. a parameter future=true. - ignore what you could find now, and return a new resource next time it matches...?
nicola (RIO/SS) (Nov 29 2016 at 10:48):
We need to identify from which point in time we want to subscribe - it should be something like offset, etag or link in search bundle like paging - updates_url. Internally we gonna use transaction id as version id for resources and as natural offset for subscriptions (singular operations will be transaction with one operation).
nicola (RIO/SS) (Nov 29 2016 at 10:50):
Another concern that not all search queries could be efficiently implemented as reactive, so may be some restrictions on search parameters for subscriptions are required.
nicola (RIO/SS) (Nov 29 2016 at 10:52):
Also long-polling often fall back to just request if server has something to response
John Moehrke (Nov 29 2016 at 13:55):
Note that with long-polling; the socket needs to be created with keep-alive enabled. This allows for detection of a broken connection, thus allowing cleanup of resources.
Grahame Grieve (Nov 29 2016 at 17:56):
Probably going to need a SubscriptionCapability resource so clients can introspect the subscription capability of the server. that'd be an STU4 thing. In the meantime, I'm still not clear on how search would / should work. Maybe
GET [base]/Observation?code=[LN]&_future=true
Then, every time there's a match, the server sends a chunked budle, with trunk for each entry. Client has to read each chunk individually. Well, can, at least
nicola (RIO/SS) (Nov 29 2016 at 18:03):
Hope, we will prototype and evaluate this approach soon. In most cases notifying client - that something has changed - even more valuable, than deliver this changes, because client could just retrieve changed data with ordinary REST request.
Grahame Grieve (Nov 29 2016 at 18:26):
ok, well let us know...
Josh Mandel (Nov 30 2016 at 13:03):
As for not exposing secrets: @nicola (RIO) are your concerns not addressed by the use of TLS? I usually see these schemes for things like GitHub web Hooks where they need to be able to deliver over regular http.
nicola (RIO/SS) (Nov 30 2016 at 13:05):
I just followed analogy with password :) It's good practice never expose password.
Josh Mandel (Nov 30 2016 at 21:21):
Typically during a sign-in flow (like, for a web app), clients do send a password to a server (protected via TLS). Then the password is hashed and checked on the server side against a stored hashed/salted password.
John Moehrke (Nov 30 2016 at 21:26):
Josh, that would be true for some forms based authentication. I thought we were recommending OpenID Connect authentication flows. For which there might be a forms based interaction between the client and the OpenID Connect authority; but that communication does not involve the Client-Server communication. It is orthogonal, yet results in OAuth credential used in Client-Server communications. Yes that form is covered by https. But the most important part is that the authentication step is independent. Thus that authentication step can leverage any authentication methods that the authentication authority wants to use (smart-card, id-token, biometrics, etc). And none of that is exposed to the Client or Server.
John Moehrke (Nov 30 2016 at 21:29):
an OAuth token can be used in a subscription. It is important to recognize expirations of the token. Often this is managed through the long-polling resulting in only a notification that something has changed, requiring the client to try again. That try again might need a refreshed token... All the kinds of Security-Risks one needs to think about and manage.
John Moehrke (Nov 30 2016 at 21:32):
Hence the Apple model of simply holding the TCP in FIN-WAIT; with the server closing when it has something to say.
Josh Mandel (Nov 30 2016 at 21:43):
@John Moehrke Sorry, we're talking past each other. @nicola (RIO) was using an analogy and I was replying in kind.
Josh Mandel (Nov 30 2016 at 21:43):
Certainly SMART on FHIR uses OpenID Connect for authentication, and a user should not be entering their EHR or portal password directly into an app.
John Moehrke (Nov 30 2016 at 21:44):
understood. I was mostly trying to get details into the thread, not so much worried about responding to you. It is hard to get people to use 'good practices', with the hope that someday we have a 'best practice' :-)
Josh Mandel (Nov 30 2016 at 21:45):
I'm just saying that when you sign into a web site, like your bank or whatever... when you fill in a password box, that password is sent directly as a form param to the bank (not hashed client-side, or whatever).
John Moehrke (Nov 30 2016 at 21:46):
Yes, with Forms based (authentication), like any form, the result is sent to the server. In authentication it is hoped that the server uses hashes (salted), rather than non-secure comparison mechanisms.
John Moehrke (Nov 30 2016 at 21:47):
the 'good practice' I want to encourage is the use of 'off-the-shelf' OpenID Connect solutions where ALL of this is not re-invented by healthcare developers.
nicola (RIO/SS) (Dec 01 2016 at 08:21):
....
nicola (RIO/SS) (Dec 01 2016 at 10:48):
I mean, if we have subscription as resource - so we could list & search subscriptions, it looks more secure do not show sensitive information like secret (access_token or whatever) as result of search/read operations, because this is vulnerability for recipient - i.e. when attacker will get access to list of subscriptions, she will get access to REST hook or message endpoint of subscriber.
Jens Villadsen (Sep 04 2017 at 07:50):
Regarding the text on subsciptions on http://build.fhir.org/subscription.html - section ""2.47.2" - As an alternative to subscriptions, the RESTful API describes a polling-based subscription method using bundles and the history operation. This method of polling allows for a much tighter relationship between the client and the server that doesn't involve missing updates and/or deletes.
Jens Villadsen (Sep 04 2017 at 07:50):
What does polling a REST service has to do with subscriptions?
Jens Villadsen (Sep 04 2017 at 07:52):
I can't see how polling a server is any guarantee that updates are missed. Actually, it more or less guarantees the opposite.
Jens Villadsen (Sep 04 2017 at 07:54):
With polling you are guaranteed to become stale right after you as a client has polled. So the sentence makes little sense
Lloyd McKenzie (Sep 04 2017 at 15:32):
The polling based approach allows you to see what's changed since you last looked. If things have changed more than once - and you care about interim changes - you'd need to do a subsequent "history" query to get the remainder or your polling query would itself need to be a history query.
Grahame Grieve (Sep 05 2017 at 01:29):
polling with any filter creates problems where a resource changes to not meet the filter
Grahame Grieve (Sep 05 2017 at 01:29):
it silently falls out
Jens Villadsen (Sep 05 2017 at 09:37):
Regarding the phrase - 'The consequence of this is that there is no notification when a resource is deleted, or when a resource is updated so that it no longer meets the criteria.'
Jens Villadsen (Sep 05 2017 at 11:06):
Imagine the following flow:
When the client issues a subscription, the server acknowledges the subscription and returns a bundle including the subscription as well as the evaluated result of the criteria (initial result set). now, whenever a change occurs that would result in a different result set, the change is sent - this could be either an update, create or delete. To my understanding there is no such mechanism defined currently within this spec. Does this sound out of scope to you?
Jens Villadsen (Sep 05 2017 at 11:09):
It probably requires versioned bundles ...
Grahame Grieve (Sep 05 2017 at 11:22):
I don't think we've talked about returning the initial set. the rest is ok. Are you on the subscription stream here?
Jens Villadsen (Sep 05 2017 at 11:42):
I now subscribe to the subscription channel ;)
Allan Bro Hansen (Oct 16 2017 at 09:10):
Hi
My system handles appointments and I am working with the business need: Notify external systems when a patient with an appointment arrives.
We only have appointments with one patient - not many patients.
My system has a high volume of appointments, an appointment is updated around every second.
Initially my approach was to go with the subscription&rest-hook pattern with a pooling done by the subscrieber.
The frequency of patients arrivals is assumed to be low enough for this pattern to make sense, it could also make sense to narrow the criteria into specific organizations/locations.
However I can see some issues with this:
1) One would naively think that you should use the criteria: "&status=arrived".
But the subscrieber is probably looking for something like "&previousStatus!=arrived&status=arrived" (where previousStatus is a homemade representation for "the status of the previous version of this appointment")
2) using the pooling pattern will possible hide updates for you
Eg. if the appointment has already changed to status=fulfilled when the pooling-search "&status=arrived&_lastUpdated=gt<lastQueryTime>" is done by the subscrieber
Issue 1) could be handled by introducing a criteria like "&patientDidArrive=true" to cover "&previousStatus!=arrived&status=arrived" (I haven't seen any standard for querying with criteria on previous version of a resource)
In fact we could (and probaly also will) introduce an patientArrived (Date) extension which could solve problem 2. But then we go against the description at FHIR and assume that that the subscrieber will do a different search than the subscription criteria, because the pooling will be:
"&patientArrived:missing=false&_lastUpdated=gt<lastQueryTime>" (I assume that that the _since parameter is a typo at FHIR)
This will remove the possibility of loosing updates but will widening the search results to changes at appointments for already arrived patients.
Alternative the subscrieber should use a different criteria narrowing down "the appoitments of my interest" and just subscriebe to &patientArrived:missing).
Or maybe I should look for a different mechanism altogether.
Any thoughts on this?
Grahame Grieve (Oct 16 2017 at 09:14):
you should look at the last connectathon stream on subscriptions:
Grahame Grieve (Oct 16 2017 at 09:14):
http://wiki.hl7.org/index.php?title=201709_FHIR_Subscriptions
Grahame Grieve (Oct 16 2017 at 09:14):
and see here:
Grahame Grieve (Oct 16 2017 at 09:15):
https://chat.fhir.org/#narrow/stream/subscriptions
Allan Bro Hansen (Oct 16 2017 at 09:26):
Thanks- I'll have a look and maybe repost the question in the core "subscriptions"-stream
Allan Bro Hansen (Oct 16 2017 at 10:46):
My conclusion after reading at the main-subscriptions stream (if others are reading along):
1) The connectathon introduces a extension "topics" with a reference to an eventDefinition.
For my scenario this would be something like:
{
"criteria": "Appointment", -- could include filtering of appointments - if subscrieber only wants to hear of specific appointments
"extension": {[
"url": "http://hl7.org/fhir/subscription/topics",
"valueReference": {
"reference": "https://example.com/EventDefinition/123",
"display": "Patient with appointment arrived",
}
]
2) the connectathon mainly addresses pushing the updated resource(s), no pooling mechanism.
Grahame Grieve (Oct 16 2017 at 10:54):
we didn't focus on polling, no. The subscription stream addressed event based subscriptions rather than than search based subscriptions - so you can do before and after comparison
Christiaan Knaap (Oct 16 2017 at 11:13):
I just mentioned polling as an alternative to using Subscriptions, since not every FHIR server will implement Subscriptions.
Allan Bro Hansen (Oct 16 2017 at 11:17):
It is not polling as an alternative to Subscriptions, it is rather Subscription with polling (as opposed to Subscription with pushing (adding a payload))
Grahame Grieve (Oct 16 2017 at 11:22):
subscription with polling?
Allan Bro Hansen (Oct 16 2017 at 11:24):
yes, a rest-hook subscription with a notification with no payload, and a subsequent polling of data by the subscriber.
Grahame Grieve (Oct 16 2017 at 11:25):
oh - so that's just how it works
Christiaan Knaap (Oct 16 2017 at 11:26):
To avoid confusion I would say 'subsequent GETting the data by the subscriber', since 'polling' is often understood as 'periodically querying the server to see whether there is any data of interest'.
Allan Bro Hansen (Oct 16 2017 at 11:32):
I agree but still I think polling is the correct term: With the POST of the no-payload-Subscription, the subscriber just knows that there is a big chance of getting data of interest.
Anyways: sorry for the confusion :-)
Abbie Watson (Oct 16 2017 at 20:15):
What does polling a REST service has to do with subscriptions?
With polling you are guaranteed to become stale right after you as a client has polled. So the sentence makes little sense
Correct. There are pull based subscriptions (GET polling), and there are push based subscriptions (database hooks/triggers). Much of the FHIR spec is designed around REST based architectures, so most people think of GET polling by default. But messaging and other architectures are supported; and the Meteor community has been using push-based FHIR subscriptions over websockets for nearly 2 years now; with at least a half-dozen production implementations.
Jens Villadsen (Oct 25 2017 at 22:35):
The current spec of subscription needs some TLC and I agree with @Allan Bro Hansen . IMHO, the scheme defined with an empty body sent from the server to which a client subscribes more sounds like a notification to me - as in: you as a client is notified about that stuff has changed. It then forces the client to poll for the information - AND - in the mean time the state COULD have changed. This design can however be justified by means of security
Jens Villadsen (Oct 25 2017 at 22:37):
@Christiaan Knaap I agree - polling is polling, IMHO. Polling by design is not determined by time - it is determined by initiative.
Jens Villadsen (Oct 25 2017 at 22:41):
And also - once a subscription is created, the server ought to send all the data matching the criteria. Otherwise, events can be lost in the timespan as stated currently where the client 1) is encouraged to do the search 2) submit the subscription
Jens Villadsen (Oct 25 2017 at 22:41):
between step 1 and 2 all kinds of stuff can happen
Jens Villadsen (Oct 25 2017 at 22:49):
Also, why is the following mandatory behaviour: "The consequence of this is that there is no notification when a resource is deleted, or when a resource is updated so that it no longer meets the criteria.". I would say it undermines the idea of an event based channel of data if only a selected part of the updates are sent. I could understand if such limitation was scoped by a given vendor by eg. being able to state this in the subscription resource, but I do not understand why such limitation is the baseline
Jens Villadsen (Oct 25 2017 at 23:00):
A scenario would be the following: I as a client have an idea of the current state. I would now like to get notified if that idea of the current state changes - be that additions, updates or deletes. Omitting a notification of either one of those means that no matter how many events I'll ever receive, my idea of the state will never be complete and it will be (I guess) impossible to say how the state is not correct - without doing additional work
Eric Haas (Oct 25 2017 at 23:41):
That is why there is a discussion of event based vs search based criteria. Is currently defined as a search based criteria. I think that would make the delete be discoverable
Jens Villadsen (Oct 26 2017 at 06:47):
@Eric Haas - the case where a resource changes so that it no longer matches a given criteria is just as important
Christiaan Knaap (Nov 07 2017 at 06:53):
I know why it is very hard to implement reporting deletes, but I agree that it should not be prohibited to do so.
Brian Postlethwaite (Nov 11 2017 at 00:22):
In the case where the item no longer matches the criteria, what would you expect to be reported?
If you receive the resource do you assume that the server knew that it _was_ in the scope previously, or now is, or is the server to fake a delete?
Jens Villadsen (Nov 13 2017 at 13:04):
Lets talk about it in amsterdam ;)
Kevin Marks (Nov 15 2017 at 15:17):
link to Nikko's subscriptions issue: https://gist.github.com/jkiddo/e773d033c245c2c4fdef6d35cbe1a75b
Jens Villadsen (Jun 11 2018 at 20:29):
Subscription.reason ... why's that mandatory ... wouldn't the reason of the subscription be kind of obvious given its criteria? It seems a bit weird ...
Jens Villadsen (Jun 11 2018 at 20:32):
and would eg. GET [base]/?_type=Observation,Patient…[and so on]&_lastUpdated=gt2018-10-01 as Subscription.criteria effectively be a way of specifying that all resources must be sent [except when deleting]
Grahame Grieve (Jun 11 2018 at 23:15):
reason... a human can provide additional context... 'this subscription is for the clinician ward round app' - says things that are useful that the technical query does not
Grahame Grieve (Jun 11 2018 at 23:15):
and yes
Jens Villadsen (Jun 12 2018 at 21:48):
@Grahame Grieve
reason... a human can provide additional context... 'this subscription is for the clinician ward round app' - says things that are useful that the technical query does not
actually not ... with the current modelling a human/bot/whatever SHALL provide additional context - that is what I don't get. Why is that modelled as 1..1? Why this hard requirement? Why would one mandate a field where the content is mainly a narrative/unstructured data?
Grahame Grieve (Jun 12 2018 at 22:55):
pretty much the reason for that is that any resource design, we make as much as we can 1..1 to find out whether that's supportable. It's very hard to ever go the other way. In this case, is is it a problem to have some reason?
Lloyd McKenzie (Jun 12 2018 at 22:58):
@Grahame Grieve, Umm, that's not the normal design rule. The normal design rule is make things minOccurs=0 unless:
a) you believe they're essential to be able to use the resource instance for pretty much every conceivable use-case
b) you believe almost all implementation should be able to populate the element all the time
Grahame Grieve (Jun 12 2018 at 23:02):
well, ok, within the framework of (b), if you think they should, it's better to try for 1.1 at the start than to try to elevate to that later
Lloyd McKenzie (Jun 12 2018 at 23:03):
Sure. But it's hard to make a case for (a) here...
Sanjay Karmakar (Jun 26 2018 at 06:06):
I am being able to create the subscription using Vonk server, rest-hook, through proper channel endpoint but the notifications are not being seen.
A K Azad (May 15 2020 at 06:59):
How do you match the search criteria/pattern against a resource in memory? I can see two options:
FhirPathEngine: but I need to evaluate each criterion, and multiplied by all Fhir Versions, it is very cumbersomeIGenericClient + InMemoryResourceMatcher: but it needs a Fhir Server with SearchParameter capability. I do not have such a server.
Has anyone efficiently tried matching a resource against a search pattern using the HAPI FHIR library?
OS (Oct 19 2020 at 15:47):
I'm building a system for taking FHIR data and mastering (MDM) certain elements, and pushing certain elements to a data lake. The data we get (mostly practitioner/practitionerRole resources) can come in massive batches at times (1000s of resource entries). If that happens, and if we have a SubscriptionTopic and Subscription set up with a rest hook, how will that data be pushed. I know during the querying process, bundles are limited to 200 resource 'pages'. Will the subscription post multiple bundles in that case of 200 resources? Or Would it post a larger bundle? Or would it fail? Any guidance here would be super helpful. Thank you so much.
Lloyd McKenzie (Oct 19 2020 at 15:48):
You might want to raise this on #subscriptions
Martin Hoffmann (Apr 12 2021 at 16:28):
Hello everyone, I would like to ask if there are already solutions for FHIR R4 that can be used as an alternative to subscriptions to notify applications as soon as certain resources are deleted.
If it's possible, the method should not be polling-based.
If I am properly informed, the SubscriptionTopic resource from Version 4.5.0: R5 Draft should allow the notification by deletion.
Still, I'm looking for a solution for R4.
ryan moehrke (Apr 12 2021 at 16:52):
if the r5 subscription changes work for you there is an IG to backport them into R4: https://build.fhir.org/ig/HL7/fhir-subscription-backport-ig/
Martin Hoffmann (Apr 12 2021 at 17:28):
My current framework does not yet have the option to use the functionality of R5.
I am rather looking for a short-term possibility to implement a deletion notification in R4.
René Spronk (Apr 13 2021 at 06:48):
The backport is your best option, the backport = R4. You will need to configure your server somehow to detect deletes. If you can't do that, then you'll have to use a client side solution based on history and/or polling.
Last updated: Apr 12 2022 at 19:14 UTC
