Stream: argonaut
Topic: US Core GF#20066 re the Fetch DocRef operation
Eric Haas (Feb 21 2019 at 19:07):
See US Core GF#20066
This tracker brings up a couple of issues making the case that that this $docref could be a search
- Who is using it?
- Based on the the definition of operation it is clearly an operation. The case is being made that this could be a search too since many times there aren't a bunch of FHIR resources being persisted ... so the first two bullets in the operation definition don't make a lot of sense since always creating "new" resources on the fly in the background in response to a search request
- what should it be? ( I am inclined to agree with John)
- do we ned to edit the operation definition?
@John Moehrke @Lloyd McKenzie @Andrew Torres
Drew Torres (Feb 21 2019 at 19:11):
https://fhir.cerner.com/millennium/dstu2/infrastructure/document-reference/#operation-docref
Drew Torres (Feb 21 2019 at 19:11):
We are using it to meet MU3.
Drew Torres (Feb 21 2019 at 19:11):
The operation was used because we are asking the API to take action to generate a document.
Michele Mottini (Feb 21 2019 at 19:13):
Yes, same for us
Lloyd McKenzie (Feb 21 2019 at 19:16):
We don't explicitly say anywhere that servers are prohibited from creating data on the fly in response to a query, however there's an implicit assumption that if I query the data today and I query the data tomorrow, all of yesterday's data will still be there unless it's been explicitly deleted. With an operation, there's no expectation that information will be retained.
Michele Mottini (Feb 21 2019 at 19:24):
$docref supports start and end parameters - that do not make much sense for a search
John Moehrke (Feb 21 2019 at 19:56):
My comment was specifically to express the difference between $docref and a query on DocumentReference. If the difference is indeed that there is an expectation that $docref will result in "a" DocumentReference that does not currently exist, and will not return all the other DocumentReference that do exist.. THEN I would be very happy. What I am worried about is that $docref does something similar but not exactly the same as a query on DocumentReference... I would still be concerned with this kind of OnDemand mechanism that seems to be different than doing a query on DocumentReference. This because, as Lloyd points out, it is very possible to create a DocumentReference during a query on DocumentReference. This is indeed what is described in IHE-MHD for support of OnDemand. So I would be happy with a clarification of the behavior of $docref, and and note that DocumentReference query can achieve the same result...
Michele Mottini (Feb 21 2019 at 20:01):
DocumentReference query cannot achieve the same result $docref?start=x&end=y creates and returns a document with all the data for that period, that is not something a search can do
John Moehrke (Feb 21 2019 at 20:07):
why not?
Eric Haas (Feb 21 2019 at 20:09):
@Michele Mottini I think that would be the same as DocumentReference?patient=[patient]&period=[period] assuming the server would respond with a an "on the fly' response to it as lloyd explains above
John Moehrke (Feb 21 2019 at 20:09):
seems to be a simple use of the DocumentReference -period query parameter.
Michele Mottini (Feb 21 2019 at 20:09):
Search parameter that result in modifying the data inside one of the returned resources?
John Moehrke (Feb 21 2019 at 20:09):
right. Yes there needs to be some way that a server can declare that it will respond that way... but it is fully possible
John Moehrke (Feb 21 2019 at 20:10):
but that is simply a profile declaration in the CapabilityStatement...
Eric Haas (Feb 21 2019 at 20:10):
however we do default to 'the latest the most recent or current document is in scope.' when no time period is given
Michele Mottini (Feb 21 2019 at 20:11):
I'd say that search parameters should modify the set of resources that is returned. not what is inside the resources
Eric Haas (Feb 21 2019 at 20:11):
@Michele I don't understand 'result in modifying the data inside one of the returned resources'
Michele Mottini (Feb 21 2019 at 20:12):
Maybe the specs do not explicitly prohibit - but it seems to me that would be a very strange behavior indeed
John Moehrke (Feb 21 2019 at 20:13):
It is not strange... when I use various RESTful api against classic internet providers, results are returned to me that clearly didn't exist prior to me asking. There is no way for a map API to have all possible responses.
John Moehrke (Feb 21 2019 at 20:14):
FHIR is an API, not a persistence model. Else we couldn't be using FHIR as an API to an EHR database... :-)
Michele Mottini (Feb 21 2019 at 20:15):
@Eric Haas calling $docref?start=2019-01-01 and $docref?start=2018-01-01&end=2018-12-31 returns one document in both cases, but _different_ documents: the first has all the data for this year, the second all the data for last year
Eric Haas (Feb 21 2019 at 20:15):
and why is that an issue?
John Moehrke (Feb 21 2019 at 20:15):
again.. I am okay with $docref, as long as it is clear that is where one goes for ondemand documents, and as long as DocumentReference query is not forbidden from doing the same thing given the same query parameters.
Michele Mottini (Feb 21 2019 at 20:16):
because search parameters do searches? and searches returns different sets of things depending on the value of the search parameters - do not change the things
John Moehrke (Feb 21 2019 at 20:17):
more specifically I think $docref needs to clearly NOT return non-ondemand document entries
Eric Haas (Feb 21 2019 at 20:18):
The Client does not know whether the resources are persisted or created on the fly so I not sure that matters for them. If the resources are persisted then it would matter for the server.
Eric Haas (Feb 21 2019 at 20:20):
so back to @John Moehrke why can't one overload the operation with regular search? what is the issue there?
Josh Mandel (Feb 21 2019 at 20:39):
a naming opportunity: calling it something like $generate-ccda would be a heck of a lot clearer.
Jenni Syed (Feb 21 2019 at 20:39):
Agree with Josh :)
Jenni Syed (Feb 21 2019 at 20:40):
Also, the doc for us is not persisted, and wouldn't come back in a general search
Jenni Syed (Feb 21 2019 at 20:41):
if that is persisted, and changes the state of the server/resource, that would be unexpected behavior for a GET
Jenni Syed (Feb 21 2019 at 20:41):
which is precisely why we went operation (all the above)
Lloyd McKenzie (Feb 21 2019 at 20:49):
If the operation can be defined in a non-US-centric manner, that would be better. CCDA is US-specific.
John Moehrke (Feb 21 2019 at 21:04):
Lloyd, the type of document is a parameter. so the client explicitly asks for C-CDA. it is not baked into the API
John Moehrke (Feb 21 2019 at 21:13):
so back to @John Moehrke why can't one overload the operation with regular search? what is the issue there?
is this a question to me? Or is this a echoing my statement and asking others why not?
Because I am happy to keep the operation with a well defined behavior, but am also happy for it to go away and use query for that functionality.
Lloyd McKenzie (Feb 21 2019 at 21:18):
@John Moehrke I was pushing back against the proposal for $generate-ccda as a name.
John Moehrke (Feb 21 2019 at 21:24):
@John Moehrke I was pushing back against the proposal for $generate-ccda as a name.
ah, right... $generate-doc
John Moehrke (Feb 21 2019 at 21:27):
actually the current $docref does not have a parameter for the encoding of the document... it does expect C-CDA output...
which is a good behavior problem... if a server can give a document for the given LOINC code in 5 different formats (mime-type + profile-conformance), does it return 5 different DocumentReference? Or does the server guess at the best response?
Eric Haas (Feb 21 2019 at 21:31):
@John Moehrke it was a question for you. As it stands it does overload with search which at the time seemed like a good idea as a convenience for the client. They don't know how or care how the document is fetched. So would like to know the specific issues when is truly a search ( assume you would get the best match )
Eric Haas (Feb 21 2019 at 21:31):
@Josh Mandel the operation is not just for ccda
John Moehrke (Feb 21 2019 at 21:32):
right. That is why IHE XDS just folded OnDemand into normal XDS query. Most clients don't know they should ask two questions when they simply want to know what documents are available.
Eric Haas (Feb 21 2019 at 21:33):
so we went with a operation. which is better?
Eric Haas (Feb 21 2019 at 21:35):
the current $docref does not have a parameter for the encoding of the document..
The server chooses what it want to serve
Eric Haas (Feb 21 2019 at 21:36):
...implementation detail... but should mention that in the spec
Josh Mandel (Feb 21 2019 at 21:36):
Re: naming, what's the common thread? $generate-document?
Josh Mandel (Feb 21 2019 at 21:36):
Is it just about summary documents?
Eric Haas (Feb 21 2019 at 21:40):
@Josh Mandel like John and I said its more a client get doc operation adn trying to mush both onthefly and existing docrefs into single transaction. so I think any name should reflect the client side more like $return-docref
Josh Mandel (Feb 21 2019 at 21:42):
I'm still stumped then. What's the use case for an operation to return an existing doc?
Eric Haas (Feb 21 2019 at 21:43):
like John mentioned above
Most clients don't know they should ask two questions when they simply want to know what documents are available.
Eric Haas (Feb 21 2019 at 21:45):
Ideally one ask for a docref no matter whether is created anew or persisted somewheres
John Moehrke (Feb 21 2019 at 21:45):
I'm still stumped then. What's the use case for an operation to return an existing doc?
if we make it clear that the operation is to only return potential ondemand documents, and will NOT return existing documents; then the client can know that it doesn't have to have logic to look for redundant entries. One way or the other... I just want deterministic behavior.
Eric Haas (Feb 21 2019 at 21:47):
And IHE went with a search that does it all. And the intent was for $docref to do it all as well. ( which is the focus of his tracker - although I could counter his tracker with a tracker on IHE and say the same ... :-))
Josh Mandel (Feb 21 2019 at 21:49):
$generate-or-retrieve-document
Eric Haas (Feb 21 2019 at 21:49):
Bottom I am OK either way do it all with search or do is all with an operation. Just need the community to decide what we want to do and then we can address the name
Josh Mandel (Feb 21 2019 at 21:50):
It's the reference that throws me off, I think -- it's not expressing the intent of the caller.
Eric Haas (Feb 21 2019 at 21:50):
I see your point there
Lloyd McKenzie (Feb 21 2019 at 21:51):
$get-documentsallows for the possibility of generation or retrieval and is shorter...
Eric Haas (Feb 21 2019 at 21:52):
$get-doc ??
Michele Mottini (Feb 21 2019 at 21:53):
Maybe is not a good name, but it had been there a long time and it is deployed in production already - why break things?
Josh Mandel (Feb 21 2019 at 21:54):
Mostly I'm just trying to understand it; sometimes naming things helps :) It might not be worth renaming in real life.
Michele Mottini (Feb 21 2019 at 22:01):
The way we implemented it is that we (1) generate the documents for the specified type(s) and period if we know how (2) fetch the existing document for the specified type and period (3) return the combination of (1) and (2)
John Moehrke (Feb 21 2019 at 22:40):
what I am confused at is... a DocumentReference resource is a metadata resource, it does not return the document. The client needs to invoke the URL in DocumentReference.content.attachment.url to get the bytes.
What I expect is that DocumentReference.content.attachment.url will be equal to the $document operation.
So, we have cascading operations... why?
Drew Torres (Feb 21 2019 at 22:59):
We preferred it that way to not have to mix the generated document vs documents that live in the system. When you query DocumentReference?Patient=1234 we only return documents that exist in the system (not the generated ones). Your point is valid. In our implementation you can skip the DocoumentReference by doing: Binary/$autogen-ccd-if?patient=1316035, but support the operation on Documentreference lets the consumer get the metadata about the document without actually generating the document.
Michele Mottini (Feb 22 2019 at 01:12):
(We actually stick the document inside attachment.data so there is no second GET needed)
Brett Marquard (Feb 22 2019 at 18:57):
Just finished reading this 65+ chain and I am little fuzzy on the conclusion:
A) Remove operation
B) Rename operation $generate-or-retrieve-document
C) Be more clear about difference in expectations on search vs $docref operation
Drew Torres (Feb 22 2019 at 18:58):
I vote for C. Stuff is in production. Haven't had many complaints. Why rock the boat?
Eric Haas (Feb 23 2019 at 00:26):
C is not completely correct. John correctly understood the fact that $docref was defined to return both on the fly and pre-exisiting docrefs. IHE does this same thing using search. As stated above the client doesn't give a hoot about the mechanics behind returning a docref and should not be required to make two separate calls (i.e., a search and an operation ) each time. So its seems the consensus is to keep $docref and therefore I think the choice should be:
d) go all in on $docref for all searching (sorry John) since having both search and operation both confusing and onerous for the client. this means making all search on docref optional and removing from docref.
Eric Haas (Feb 23 2019 at 00:31):
If that not acceptable then a) remove operation and supercharge the search as John suggests
John Moehrke (Feb 24 2019 at 17:49):
correct. I prefer ( a )... query is all that is needed. It will work against a basic FHIR server, where as forcing the operation means that everyone must add the operation... and there is no functional benefit to an operation.
Michele Mottini (Feb 25 2019 at 13:25):
It is not a search - 'Search operations traverse through an _existing_ set of resources _filtering_ by parameters supplied to the search operation.'
Michele Mottini (Feb 25 2019 at 13:26):
Besides that: it has been implemented, it works, implementers are OK with it - why change things?
John Moehrke (Feb 25 2019 at 15:07):
I am not forcing change, I am asking for clarity. I think you have expressed that in your case the operation will only return one ondemand document. That would be a fine clarification. In that case I can know that I will not get duplicates between the operation and a search. As long as I can understand this, I am good with the result. but I also don't think we should have an operation when one is not needed, as it forces clients to do two things.
Brett Marquard (Mar 12 2019 at 15:35):
FYI - after long discussion Eric/I plan to propose deprecation of $docref on SD this Thursday
Eric Haas (Mar 12 2019 at 15:36):
In favor of IHE approach using search. Deprecate means keep in for one more cycle before removing.
Michele Mottini (Mar 12 2019 at 16:11):
You cannot use search to return document created on the fly: 'Search operations traverse through an _existing_ set of resources _filtering_ by parameters supplied to the search operation'
John Moehrke (Mar 12 2019 at 16:23):
We to distinguish something... the Query on DocumentReference can return DocumentReference items that indicate the ability to request that a document be created. If a client does a GET on the DocumentReference.content.attachment.url, that is when the document is created. Not at query time. This is the model that has been in XDS/XCA/MHD for a long time.
I recognize the distinction that some see the $docref as returning the DocumentReference and also the Document that was created during that operation.
John Moehrke (Mar 12 2019 at 16:25):
so, if we can clarify that distinction, then I will be happy.. Can we make clear that $docref is used to create an ondemand document, where the result is both the DocumentReference and the document ? Specifically it would NOT return static entries, just the one requested to be created.
Michele Mottini (Mar 12 2019 at 16:29):
Implementers seems to be happy with how things are - once again, why do you want to change things? Especially things that are already in production?
John Moehrke (Mar 12 2019 at 16:30):
I submitted ballot comment against the IG aligning with FHIR R4... not DSTU2... many things changed between DSTU2 and R4.
Michele Mottini (Mar 12 2019 at 16:31):
That's not a reason to change even more things
John Moehrke (Mar 12 2019 at 16:35):
The comment points out that some servers are acting differently than other servers. I am just looking for everyone to implement the same behavior. Some return all results, some just one newly created document. Some return the document , some don't.. I am looking for deterministic behavior that a client app can rely on. I don't care what that behavior is. It just needs to be consistent, else we are not talking 'standards' we are talking custom code in client apps to deal with custom behavior by various servers.
Eric Haas (Mar 12 2019 at 16:37):
We are concerned about having 2 ways to do the same thing between IHE MHD and US Core. We feel that these are two broadly implemented specs and see an opportunity to align. After other feedback, we are considering limiting the scope of docref to CCD as a compromise and adding in the IHE like search.
Thinking aloud ....Or should we support both as an interim solution.
RE John;'s concerns the operation is clearly defined to return DocRef. So that is not an issue with the specification..
Michele Mottini (Mar 12 2019 at 16:40):
Add more explanation / examples to the specs - do not change them
(and having said that - there will be always incompatible / wrong implementations - that's not something that can be solved just writing specs unfortunately)
John Moehrke (Mar 12 2019 at 16:43):
does $docref return only onDemand documents? or not? It says SHOULD today, which is confusing. A client can't tell if it will get duplicate results or not. Can we make this a SHALL-NOT?
John Moehrke (Mar 12 2019 at 16:43):
does $docref return the DocumentReference and the Binary of the document? Seems there is strongest support for YES, both.
John Moehrke (Mar 12 2019 at 16:44):
does $docref return one entry, or a set of documents that meet the criteria? I think the strongest support is that the server determines THE best match, returning one entry that best matches, not all entries that are close.
Michele Mottini (Mar 12 2019 at 16:48):
No, not only on demand: 'The server SHOULD return at least all references for documents that it has made available in a document indexing system. This is the same as a simple query '
John Moehrke (Mar 12 2019 at 16:49):
so then you want it to say SHALL
John Moehrke (Mar 12 2019 at 16:49):
That is clarity I can live with. I don't think it is right, but it is more clear.
Michele Mottini (Mar 12 2019 at 16:50):
'This is the same as a simple query' is crystal clear to me
John Moehrke (Mar 12 2019 at 16:50):
SHOULD is not the same as SHALL
John Moehrke (Mar 12 2019 at 16:51):
SHOULD means to a client... you might get them, you might not. You are looking at the spec from a server perspective. I am looking at the spec from a client.
John Moehrke (Mar 12 2019 at 16:52):
make it a SHALL... but I suspect that is going to make other server vendors upset... as they return only the onDemand entry.
Michele Mottini (Mar 12 2019 at 16:52):
Again, they look clear to me - from both sides - and implementers seems to be ok
John Moehrke (Mar 12 2019 at 16:53):
Im an implementer... I am not ok with it... hence my ballot comment... the whole reason for standards governance
Michele Mottini (Mar 12 2019 at 16:54):
OK - clarify it if you want - but do not _change_ it please
John Moehrke (Mar 12 2019 at 16:55):
I presented three decisions.. what is your company position? We can then poll all the EHR vendors and see if there is a consensus.
Michele Mottini (Mar 12 2019 at 17:30):
The three options above? We'll do C) Be more clear about difference in expectations on search vs $docref operation
Brett Marquard (Mar 12 2019 at 17:33):
reading through chain I am not crystal clear on 'option C)' Please re paste here
Drew Torres (Mar 12 2019 at 17:42):
Just finished reading this 65+ chain and I am little fuzzy on the conclusion:
A) Remove operation
B) Rename operation $generate-or-retrieve-document
C) Be more clear about difference in expectations on search vs $docref operation
Drew Torres (Mar 12 2019 at 17:42):
I like C too.
John Moehrke (Mar 12 2019 at 18:49):
I don't know about what C means... I pointed out three things that I think are not clear in the current specification. I am okay with keeping $docref if it has a well defined behavior. Note that I think renaming is helpful to clarity, but clarity is more important first.
1. does $docref return only onDemand documents? or not? It says SHOULD today, which is confusing. A client can't tell if it will get duplicate results or not. Can we make this a SHALL-NOT?
2. does $docref return the DocumentReference and the Binary of the document? Seems there is strongest support for YES, both.
3. does $docref return one entry, or a set of documents that meet the criteria? I think the strongest support is that the server determines THE best match, returning one entry that best matches, not all entries that are close.
Eric Haas (Mar 12 2019 at 19:19):
mmmm when I read it I don't interpret any of that...seems pretty clear to me
1. does $docref return only onDemand documents? or not? It says SHOULD today, which is confusing. A client can't tell if it will get duplicate results or not. Can we make this a SHALL-NOT?
it SHOULD return DocumentReferences for both pre-indexed ( I agree SHALL would be better) and on demand : "a Bundle... containing resources conforming to the US Core DocumentReference Profile for the patient" I don't understand the duplicate comment.
2. does $docref return the DocumentReference and the Binary of the document? Seems there is strongest support for YES, both.
NO each DocRef has a URL pointing to a document. This is a 2 step process get the docref and then the document. "The document itself can be subsequently retrieved using the link provided from the DocumentQuery search results. The link could,for example, be a FHIR endpoint to a Binary Resource or some other document repository."
3. does $docref return one entry, or a set of documents that meet the criteria? I think the strongest support is that the server determines THE best match, returning one entry that best matches, not all entries that are close.
"The operation takes the input parameters:
patient id
start date
end date
document type
and returns a Bundle off type “searchset” containing resources conforming to the US Core DocumentReference Profile for the patient. "
" If neither a start date nor an end date is provided, the most recent or current document is in scope."
" If no type is provided, the CCD document if available SHALL be in scope and all other document types MAY be in scope."
Specific behavior by a implementation should be documented in its CapStatement.
John Moehrke (Mar 13 2019 at 12:34):
2 - @Eric Haas you are asserting the exact opposite of what @Michele Mottini indicated earlier.
John Moehrke (Mar 13 2019 at 12:34):
(We actually stick the document inside attachment.data so there is no second GET needed)
see 2 inline
John Moehrke (Mar 13 2019 at 12:35):
thus, the specification is not clear. on 2
John Moehrke (Mar 13 2019 at 12:39):
DocumentReference query cannot achieve the same result
$docref?start=x&end=ycreates and returns a document with all the data for that period, that is not something a search can do
note this is the strongest argument for keeping $docref, and having it do different things than a query. Totally agree with @Michele Mottini . This is why I am NOT trying to eliminate, but rather to make clear THIS is the behavior.
Eric Haas (Mar 19 2019 at 19:54):
here my proposed edits: https://gforge.hl7.org/gf/project/fhir/tracker/?action=TrackerItemEdit&tracker_item_id=20066&start=0
John Moehrke (Mar 21 2019 at 13:19):
I think I understand your edit proposal, and it seems to address my concerns. I understand that $docref shall return all static results that match the paramaters of $docref while also returning dynamicly created entries for onDemand. Thanks
Last updated: Apr 12 2022 at 19:14 UTC
