Stream: implementers
Topic: Transaction Processing and Ordering
Gino Canessa (Nov 09 2021 at 16:10):
Hi implementers! At this week's FHIR-I call we were working on a ticket about how transactions are processed (spec: here, ticket here) and needed some community feedback.
In order to clarify some language and requirements for transactions, the group wanted to get more feedback about how people are using them today - specifically in the context of request ordering/resolution. This is for both servers (how things are implemented) and clients that use transactions (how clients expect things to be implemented).
Some notes about the spec:
- This area is normative, but has a lot of 'room' in the language - we're hoping to clarify/narrow so that servers and clients have more aligned expectations and implementations.
- The spec is explicit that the order of requests in a transaction is irrelevant. One of the reasons for creating transactions was to handle cases where two resources refer to each other and need to be created together. This implies that the server is required to sort out all necessary ordering for their own purposes (e.g., if the implementation needs a Patient before and Encounter and they are both present, the order in the transaction doesn't matter), but may not be clear. The key language is:
The outcome of processing the transaction SHALL NOT depend on the order of the resources in the transaction.. - The spec defines an order for processing based on HTTP verbs:
DELETE,POST,PUTand/orPATCH,GETand/orHEAD, conditional reference resolution.
Some questions:
- Did you know about that ordering, and if so, do you follow/expect it?
-
Are searches supported in transactions? If so, do POST-based searches happen early (after delete) and do GET-based searches happen late (after all delete/post/put/patch/etc.)?
2.a. Bonus for clients: if you are using this, are you depending on it? E.g., issuing a search before and after creates/updates to see what happens? -
Are operations supported in transactions? If so, what are your rules around when they are applied - e.g., POST is before PUT, but if an operation requires a resource that is created 'later' in the process ordering, does it pass/fail?
3.a. Bonus for servers: if you support operations, when do you do them? E.g., do you perform operations and then POST-based creates, creates and then operations, or some other ordering?
Tagging people that were involved with the discussion and/or that I thought would have an opinion (sorry if I got any wrong or missed anyone! =) @Grahame Grieve @Josh Mandel @Alexander Zautke @Bas van den Heuvel @Paul Church @Vassil Peytchev @Brendan Kowitz @Caitlin Voegele @Lee Surprenant @James Agnew
Thanks!
René Spronk (Nov 10 2021 at 07:15):
On question #1: some translations of v2 to FHIR definitely depend on the order of processing of the HTTP verbs.
Grahame Grieve (Nov 10 2021 at 09:07):
how so?
Grahame Grieve (Nov 10 2021 at 09:09):
Note that I think that the bit about which verbs to process in which order is advice to servers, not mandatory - that is 'if you process in this order', then you'll meet the rule 'order doesn't matter' (except for reconciling references in advance). but it's possible that defining the order somehow defines something extra over 'The outcome of processing the transaction SHALL NOT depend on the order of the resources in the transaction' - if it does, what?
Josh Mandel (Nov 10 2021 at 15:08):
Of course our current documentation, which provides a specific order of verbs, defines something extra beyond the basic prohibition of "SHALL NOT depend on the order of the resources in the transaction" -- for example, @Grahame Grieve imagine if we did not include any more specific guidance. Server A might decide to process DELETEs before GETs, and Server B might decide to process GETs before DELETEs. Neither server would violate the basic prohibition, because neither server would produce different results in the face of differently ordered transaction request bundles. But interoperability across servers would be poor.
Gino Canessa (Nov 10 2021 at 15:37):
Yes, it can also change the results on a simple transaction with one delete and one create. One order removes a previous record and add the new one. A different order either fails (record exists) or adds the record then deletes it.
If the server is re-ordering requests in the bundle, then having a defined order is necessary. What that order is will change where people draw boundaries around the number of transactions.
edit: The alternative is to say they are processed in-order, but that servers need to handle forward references. I have no doubt that is a significant breaking change and off the table - just noting to have it out in the open.
Paul Church (Nov 10 2021 at 16:07):
The Google implementation does not support searches or operations in transactions, so that part isn't relevant for us. We do follow the order of processing, except that conditional references are evaluated first - before the transaction starts. Because of asynchronous indexing, we don't have a practical way to do a search that takes into account changes happening inside the current transaction.
Lee Surprenant (Nov 10 2021 at 16:26):
I've been meaning to double-check on some of the finer details, but let me at least get this out there before I forget.
- Did you know about that ordering, and if so, do you follow/expect it?
yes, we’re aware of it and try to follow it
- Are searches supported in transactions?
yes, but not sure how i feel about it
- Are operations supported in transactions?
yes. they weren't initially but it was a common user request so we did it. not sure how i feel about this one either
net: I think we follow the method-based ordering from the spec, but i agree that leads to 'interesting' behavior that you touch on well with your sub-questions...will stay tuned
Cooper Thompson (Nov 11 2021 at 14:29):
René Spronk said:
On question #1: some translations of v2 to FHIR definitely depend on the order of processing of the HTTP verbs.
My understanding from discussion in v2 to FHIR was that you can't (in practice) do v2-to-FHIR using transactions. Or maybe I'm just not understanding how you could execute the query/match/create/update/delete pattern for a segment in a transaction.
René Spronk (Nov 11 2021 at 15:56):
As usual it depends one wishes to do. Messaging uses snapshots, and of one has single source of truth for e.g. GT1 or AL1 segments, one can simply rip/replace any existing corresponding FHIR resources with whatever is contained in a v2 message.
If one has multiple source systems, it gets a bit tricky, but using Provenance helps with the replacement of snapshots: rip/replace just those FHIR resources that were derived from v2 messages that were derived from v2 messages from the same sender as a later update v2 message.
The way that is suggested in the aformentioned v2 to FHIR stream tries to merge all sorts of resources on the FHIR side (merge the content of existing resources with updates via v2, potentially from multiple sources). No way you could do that with a transaction (unless you had an operation to hide such logic, and invoke it from the transaction).
Cooper Thompson (Nov 11 2021 at 17:47):
For the rip/replace, I'm not sure how you'd do the "rip" part. Conditional delete doesn't work since you don't know what you are deleting yet. You need to do a search first, get the list of resources to delete, then submit them for deletion. So you'd need, at a minimum, a search for every resource type + the transaction. I don't think you can take the results of a search and delete them in a single transaction bundle right?
Cooper Thompson (Nov 11 2021 at 17:47):
Other than an operation.
Alexander Zautke (Nov 18 2021 at 19:14):
Coming back to the original proposal of the ticket: Do we think that changing the list to contain FHIR interactions would be too much?
Alexander Zautke (Nov 18 2021 at 19:16):
To answer the questions for Firely Server:
- Yes, it's implemented according to the order mentioned in the spec.
- Searches with POST and GET are supported. They are sorted according to the spec and no-one complained about until now.
- Same for operations, there is no special handling to figure out when the operation should be performed.
Last updated: Apr 12 2022 at 19:14 UTC
