Stream: bulk data
Topic: Asynch as a fall-back
Lloyd McKenzie (Jul 13 2020 at 22:36):
I did a quick search and didn't find anything. Feel free to redirect if this has already been discussed and landed.
I'm wondering if there is a documented convention (and if not, if it's worth creating one) for a situation where a client is attempting a synchronous search or other operation, but the server decides "nope, that's going to take too long, it needs to be asynch" and returns an HTTP code or some other indicator that indicates to the client that they should retry their request with prefer=respond-asynch set.
Jenni Syed (Jul 13 2020 at 22:59):
The rfc referenced in http://hl7.org/fhir/r4/async.html allows for that
Jenni Syed (Jul 13 2020 at 22:59):
Well, I thought it did... hrm
Jenni Syed (Jul 13 2020 at 23:01):
There is a way to say the operation takes too long today in operation outcomes and http status
Jenni Syed (Jul 13 2020 at 23:01):
if we want to specify this, we could start from there?
Lloyd McKenzie (Jul 13 2020 at 23:10):
That was my thought. We could either use a 408 HTTP (Request timeout) or a 400 + a "too-long" OperationOutcome code (or perhaps a brand-new OperationOutcome code?) to signal. Do we think that would be a useful enhancement to the spec?
Josh Mandel (Jul 14 2020 at 14:05):
To be clear Lloyd, you're not thinking about this as a specific enhancement for the $export operations but rather as a general enhancement for FHIR's async framework? (This channel has been mostly focused on $export/$import -- where we assume async.)
Thinking about the general case: sure having some way to hints to the client that a specific response cannot be provided synchronously seems like a good idea to me.
Lloyd McKenzie (Jul 14 2020 at 14:07):
Yes, this would be for asynch in general, not specific to $export (where you already know you want asynch). I raised it here because I figured that's where most of the asynch-aware folks would hang out.
Lloyd McKenzie (Jul 14 2020 at 14:07):
(And I think this group is principally responsible for driving changes to the core async page)
Josh Mandel (Jul 14 2020 at 14:08):
To the extent that changes are being driven, yes. But there's not actually a good process for keeping these things... in sync.
Lloyd McKenzie (Jul 14 2020 at 14:10):
Does anyone have a preference for HTTP 408 vs. HTTP 400 + "too-long" vs. new OO code vs. something else?
Josh Mandel (Jul 14 2020 at 14:11):
408 doesn't quite seem right to me because it implies the client May retry its request... and we would be saying that a client would be silly to do so without upgrading to asynchronous.
Josh Mandel (Jul 14 2020 at 14:13):
The existing "too long" code is about a client supplying too much input, on my reading.
Lloyd McKenzie (Jul 14 2020 at 14:14):
So you would vote for a new OperationOutcome code, or is there something else we should leverage?
Josh Mandel (Jul 14 2020 at 14:14):
I sure wish we had used coding and not code in operationOutcome.code
Josh Mandel (Jul 14 2020 at 14:15):
I think a new operation outcome code makes sense. We could also explore some kind of header.
Jenni Syed (Jul 14 2020 at 16:17):
Actually, we typically use "too-costly" in the operation outcome for things that would take too long
Jenni Syed (Jul 14 2020 at 16:20):
it looks like that's tied to 422 today, since it's a bit of a biz rule
Jenni Syed (Jul 14 2020 at 16:20):
(again, just in our implementation - doesn't mean it's correct :) )
Vic Hindocha (Jul 14 2020 at 16:52):
Hello all
Lloyd McKenzie (Jul 14 2020 at 16:59):
Hello :)
Carl Leitner (Aug 05 2020 at 16:55):
as opposed to the server deciding at the time of the request that a query is going to take too long and providing an async fallback, is there any good option pre-coordination, in particular in the case of frequently offline clients? Something along the lines of a client (or set of clients) saying "I am going to need an export with these parameters on a routine basis, but I dont know that I will have good enough connectivity to wait for an async operation or for a long running query to execute on the server, can you please go ahead and cache the export with these parameters and make sure it stays fresh?"
Jenni Syed (Aug 05 2020 at 16:59):
The pre-coordination is what exists today - the app can say it prefers the respond-async response.
Jenni Syed (Aug 05 2020 at 16:59):
It's the server saying "I know you said you wanted sync, but you just asked me to do something that will take several minutes or more.. I'm going to respond async" that seems to be missing
Carl Leitner (Aug 05 2020 at 17:58):
@Jenni Syed maybe to elaborate my understanding: Let's say I have a set of parameters X for an export. Right now, in the sync scenario, Client A would submit X, the server would then generate the export and then respond back. If Client B wants that same export with X, then it would not know about Client A's request and would need to initiate it's own export operation. It would be up to the server to decide if they want to implement something like, oh I have seen this request for export X before, let me reuse the results (with perhaps a refresh for new data). There is no current way to declare to the server "export under X parameters is going to be of high interest to several clients, please try to keep an export ready for immediate response to the clients."
Carl Leitner (Aug 05 2020 at 17:58):
have I misunderstood something here?
Jenni Syed (Aug 05 2020 at 19:19):
That's accurate - though I'm not sure a client is going to know when many apps will have access to the same data (security wise). I'm not sure it can be anyone other than the server that determines that. I assume you're referring specifically to the bulk spec? I think this was originally a question that was a bit broader/more general than bulk export (but reusing the parts of the spec that added that async response functionality)
Carl Leitner (Aug 05 2020 at 20:02):
this is for bulk spec. maybe more context would help..... we support a number of mobile apps that work in largely offline environments (sub-Saharan Africa). many of these deployments include where there is no network connectivity and the devices need to be taken on a weekly/monthly basis to sync data. this seek window can sometimes be small. there may be several health workers working out of the same clinic and so would, for example, need the same patient cohort identified by a Group to pull in any updated information on the patients/clients. I would like to avoid a situation where they miss a sync window because the server is still processing.
Michele Mottini (Aug 05 2020 at 20:08):
The server can keep the exported data always up to date (independently from bulk export requests), and when a bulk export request for that set comes in just return the links to the files? Does not seem to need any change in the bulk export specs
Carl Leitner (Aug 05 2020 at 20:11):
@michele how does we indicate to the server which exports to keep up to date? this could be quite voluminous especially if we are subsetting from a large server (e.g. national)
Michele Mottini (Aug 05 2020 at 20:22):
A config file? A form in its user interface?
Carl Leitner (Aug 05 2020 at 20:24):
right, but that would be server implementation specific.
Carl Leitner (Aug 05 2020 at 20:24):
i would prefer something that wasn't
Carl Leitner (Aug 05 2020 at 20:53):
something to specify a "cache hint" for the server. something like an $export-hint operation that takes the same parameters as $export
Carl Leitner (Aug 06 2020 at 12:54):
or even a cache-hint parameter under $export
Josh Mandel (Aug 06 2020 at 16:22):
If Client B wants that same export with X, then it would not know about Client A's request and would need to initiate it's own export operation.
Just to note: this all gets more complex if there are any relative requirements like _since, because in general each client would want to supply its own "last synced timestamp" value here. Client A and B likely have different needs.
Josh Mandel (Aug 06 2020 at 16:24):
But Carl, maybe your use case is simpler -- like, if at export time a client can say "Do this export now, but I'm going offline for a bit, so please keep the result available for 1 week, or 1 month, or until I call delete"... is that enough? (Or are you saying you need to have the server maintain some kind of cron-like schedule-driven proactive process?)
Carl Leitner (Aug 06 2020 at 16:43):
yes, agreed A and B would need in general individual timestamps, even if almost the same. _since is a potential special case here, though I don't think the other parameters would be different for A and B. perhaps some guidance on on server implementations to index cached results on timestamp. the _type, _typeFilter and _outputFormat should be consistent for A and B.
i don't think the simpler use case would work, at least not as stated. maybe if it was "Do this export now, but I'm going offline for a bit, so please keep the result available and refreshed until I call delete" could work. I was imagining a more proactive process.
Carl Leitner (Aug 06 2020 at 16:44):
as a side note, this is to help inform and facilitate some sync workflows that will be a part of an android fhir sdk and the openhie architecture.
Last updated: Apr 12 2022 at 19:14 UTC
