Stream: subscriptions
Topic: Large Message Bundle
OS (Oct 19 2020 at 15:56):
X-Post - https://chat.fhir.org/#narrow/stream/179166-implementers/topic/subscriptions/near/213805700
Hi all -
I'm building a system for taking FHIR data and mastering (MDM) certain elements, and pushing certain elements to a data lake. The data we get (mostly practitioner/practitionerRole resources) can come in massive batches at times (1000s of resource entries). If that happens, and if we have a SubscriptionTopic and Subscription set up with a rest hook, how will that data be pushed? I know during the querying process, the pagination limits a page to 200 resources. Will the subscription post multiple bundles in that case of 200 resources? Or Would it post a larger bundle? Or would it fail? Any guidance here would be super helpful. I'm sure the 200-item limit is a configuration that can be changed, but my question stands regardless of if the posted bundle is > the limit of # of resources on a page. Thanks in advance
Gino Canessa (Oct 19 2020 at 16:07):
Interesting question - so far we've been leaving this up to the server's discretion (e.g., when to bundle, and how many). In this case, we probably do need some way to establish a 'max' for the bundling process, so that the server doesn't try to send more entries than the receiver accepts. Do you want to open a Jira for this?
OS (Oct 19 2020 at 18:13):
Sure thing! Just so I understand though, is the current out-of-the-box behavior that the resources will not be paginated and the (HAPI) FHIR server will try to send the entire set? Or will it still send a paginated/truncated bundle over?
OS (Oct 19 2020 at 18:21):
OS said:
Sure thing! Just so I understand though, is the current out-of-the-box behavior that the resources will not be paginated and the (HAPI) FHIR server will try to send the entire set? Or will it still send a paginated/truncated bundle over?
Added the following issue in JIRA: https://jira.hl7.org/browse/FHIR-29315 and cc'd you on it as well. If you have any thoughts on what the current out-of-box behavior would be, that'd be very much appreciated! Thanks!
Gino Canessa (Oct 19 2020 at 18:35):
Notification bundles don't have any sort of pagination support. There's simply a bundle that contains 'x' number of notifications within it. So far, we've left it entirely up to the server implementation to decide when to bundle multiple notifications together and how many to do.
I think there should be some way for a client to specify their upper bounds on the notifications, but I don't know if that should be an element on the Subscription or an extension, and if it should be count or size.
David Pyke (Oct 19 2020 at 18:36):
It seems to me that it should follow the default paging size for $everything. That way it's consistent and defaults to 20
Gino Canessa (Oct 19 2020 at 18:41):
That's an option - though it sets a single expectation instead of leaving it per-client (e.g., a lightweight client connected via websockets has the same hard cap as a large batch-processing server).
Vassil Peytchev (Oct 19 2020 at 18:46):
Maybe I am missing some context, but why is it expected that with subscription there will be notifications for 1000s of resources?
Josh Mandel (Oct 19 2020 at 18:55):
This could happen, for example, if you have broad subscriptions looking for all changes to resources of a particular type. Imagine a subscription to all changes in Practitioners, coupled with a server that has just performed a transaction-based loading of a pile of practitioners -- a client's subscription could accumulate a bunch of events all at once.
OS (Oct 19 2020 at 19:34):
Josh Mandel said:
This could happen, for example, if you have broad subscriptions looking for all changes to resources of a particular type. Imagine a subscription to all changes in Practitioners, coupled with a server that has just performed a transaction-based loading of a pile of practitioners -- a client's subscription could accumulate a bunch of events all at once.
This is precisely our use case.
I like your option @David Pyke - from a developer's point of view, that gives me something that I can start from in my conversations with clients. I totally get your pov @Gino Canessa - however, I think the safety of having a default to start with would result in more predictable behavior, rather than encountering edge cases when a server unexpectedly spikes for example.
In any case, @Gino Canessa is correct this is an implementation question, I probably need to pose this to a HAPI workgroup. Thanks for the responses!
Gino Canessa (Oct 19 2020 at 20:03):
Sounds good - please do keep in mind that the decision to batch is (and will stay) up to the server (e.g., whether to batch notifications or send single ones).
OS (Oct 19 2020 at 20:17):
Gino Canessa said:
Sounds good - please do keep in mind that the decision to batch is (and will stay) up to the server (e.g., whether to batch notifications or send single ones).
Absolutely! Humble suggestion - perhaps the documentation should make that explicit. It may just be me, but when I saw that the results are returned as a bundle, I automatically assumed it was batch (but that may be my own mistake)
Gino Canessa (Oct 19 2020 at 20:34):
Thanks for the feedback - I'll look at that section when I'm updating docs this week.
Last updated: Apr 12 2022 at 19:14 UTC
