Stream: implementers
Topic: Paging through a resource content
Michele Mottini (May 06 2019 at 16:43):
There are various resources that can contain very long lists of elements (eg List, Group, ValueSet) - it can be unpractical to GET one of those in one go. Is there any way to 'page' through their content that I missed? If not - should we define maybe operation(s) to page through those lists?
Michel Rutten (May 06 2019 at 17:58):
I guess an API could implement something like this in the transport layer. Not sure if it is wise to pull this into the FHIR spec. Note that resource fragments are not valid json/xml.
Yunwei Wang (May 06 2019 at 17:59):
ValueSet expansion has paging parameters in the $expand operation.
Grahame Grieve (May 06 2019 at 18:05):
nothing general so far
Jim Steel (May 06 2019 at 18:07):
so is the implication that we need something like _elements=someProperty[0..10] ?
Michele Mottini (May 06 2019 at 18:12):
Maybe ... but I was thinking more along the lines of the way the $expand operation works, specifying a range (and relying on the fact that there is only one element that needs the paging)
Grahame Grieve (May 06 2019 at 18:21):
sounds like graphQL
Michele Mottini (May 06 2019 at 18:22):
@Michel Rutten yes, you can do things at the transport layer, but the problem I am thinking about is the client having to limit the elements it gets for display or processing reasons, and so it would need valid json/xml resources, that transport layer stuff does not get you
Stefan Lang (May 06 2019 at 18:26):
As Michel, I'm also not sure whether this should be covered in FHIR core.
If I was in the need, I would probably put up a custom operation on Group/List.
What is your use case? Especially: is it necessary to have a Group or List, or might a search give the same results (e.g. using tags on the resources) - since a searchset Bundle has all that stuff out of the box
Michele Mottini (May 06 2019 at 18:29):
Our specific use case is List, and we did solve it with a custom operation that returns a subsetted List with only a range of items
Michele Mottini (May 06 2019 at 18:29):
But seems a fairly general case, hence my question here
Michele Mottini (May 06 2019 at 18:32):
(Group are in the proposed bulk data specs, so systems that implement that are going to have large Groups to deal with)
nicola (RIO/SS) (May 07 2019 at 04:04):
Huge resources maybe a bad idea. That's why for terminology in aidbox we introduced Concept resource and sptit StructureDefinition into Element (Attribute) resource. Because after paging you will want sorting and filter, concurent update problems, hisory problems etc - you will duplicate half of api :(
nicola (RIO/SS) (May 07 2019 at 04:07):
We also need ndjson representation of bundle for efficient stream processing
Michele Mottini (May 07 2019 at 15:40):
Huge resources maybe a bad idea
Maybe they are a bad idea, but it is what we have....
Yunwei Wang (May 07 2019 at 15:45):
@nicola (RIO/SS) I don't understand how Concept resource could reduce the number of items in value set expansion.
Michael Donnelly (May 07 2019 at 15:45):
@Cooper Thompson and I were talking about that following a discussion earlier today in PA.
Michael Donnelly (May 07 2019 at 15:46):
FHIR already has an elegant solution for this for the Bundles returned by searches. How directly can we leverage that?
Michael Donnelly (May 07 2019 at 15:47):
List has the List.entry BackboneElement. What if we had a new resource, ListEntry, that had the same elements as List.entry?
Michael Lawley (May 07 2019 at 15:47):
Following @Grahame Grieve 's comment, I think you first have to answer why graphql is not an appropriate solution.
Michael Donnelly (May 07 2019 at 15:47):
Then the List could have a new element, List.entryBundle to point to a Bundle of ListEntry resources.
Michael Donnelly (May 07 2019 at 15:50):
Following Grahame Grieve 's comment, I think you first have to answer why graphql is not an appropriate solution.
I don't think it's unreasonable to consider multiple possible solutions and evaluate their relative merits. (Except when one has already been adopted; in that case I agree that whoever wants to present an alternative first needs to explain why we should discard what we're already doing.)
Yunwei Wang (May 07 2019 at 15:54):
When Bundle is used as search result, the client can indicating paging using _count parameter. Similarly for valueSet expansion, client can provide offset and count for paging. When client request List/Group resource, what is the parameter for client to ask server for paging?
Michael Donnelly (May 07 2019 at 15:57):
For the ListEntry solution, a client would GET [base]/List/[id], find the url for the List.entryBundle, and GET the linked Bundle. If the server didn't want to return the contents of the Bundle in a single page, it would provide a URL for the next page.
For graphQL, how does this work? It looks like the client would POST to [base]/List/[id]/$graphql
Michael Donnelly (May 07 2019 at 15:57):
What does the client POST there?
Michael Donnelly (May 07 2019 at 16:00):
When Bundle is used as search result, the client can indicating paging using _count parameter. Similarly for valueSet expansion, client can provide offset and count for paging. When client request List/Group resource, what is the parameter for client to ask server for paging?
It could use the same syntax as a search. In fact, it could be a search. If ListEntry has a List element that's a Reference to the parent list, the List.entryBundle element could be a uri to the search. [base]/ListEntry?list=[the List's id]
Michele Mottini (May 07 2019 at 16:00):
If we have a separate ListEntry resource then we can do normal paged searches directly on that I guess?
Michael Donnelly (May 07 2019 at 16:01):
If we have a separate ListEntry resource then we can do normal paged searches directly on that I guess?
Yes!
Michele Mottini (May 07 2019 at 16:01):
(I typed it at the same time...)
Grahame Grieve (May 07 2019 at 16:01):
I was using graphQL to get the list, I'd do Patches to update the list
Michael Donnelly (May 07 2019 at 16:01):
A client could hit [base]/ListEntry?list=[the List's id]&_count=100
Grahame Grieve (May 07 2019 at 16:01):
graphQL is really neither here nor there for mutations
Michele Mottini (May 07 2019 at 16:02):
I assume Grahme was not proposing GraphQL as a solution - implementing a complete different API just to solve this does not seem very practical
Grahame Grieve (May 07 2019 at 16:02):
I have to say, I really don't like the idea of ListEntry- feels like cracking a nut with a bulldozer
Michael Donnelly (May 07 2019 at 16:02):
That's fair.
Michael Donnelly (May 07 2019 at 16:03):
How complex would graphQL be? What would the body be?
Grahame Grieve (May 07 2019 at 16:03):
I was reacting to the invention of a whole different API anyway that was like graphQL
Michael Donnelly (May 07 2019 at 16:03):
I'm not following then; is there a solution you recommend for paging through a List?
Grahame Grieve (May 07 2019 at 16:06):
we've never really resolved the problem of a list getting too big. But i would say that the obvious solution is to add $subList $add $remove operations, and anything more sophisticated --> graphQL
Jim Steel (May 07 2019 at 16:06):
I suppose you could look at List fragments in the style of CodeSystem fragments
Jim Steel (May 07 2019 at 16:06):
(not that it gets much use in CodeSystem implementations AFAIK)
Jim Steel (May 07 2019 at 16:07):
of course, that would only work if Lists were defined with and referenced by something like a canonical
Michael Donnelly (May 07 2019 at 16:09):
I don't disagree with Grahame's bulldozer point. That said, every FHIR server has a bulldozer already, and every FHIR client knows how to drive one.
Yunwei Wang (May 07 2019 at 16:09):
I am a little bit lost in this discussion. So @Michael Donnelly What is the proposed ListEntry resource and what is the difference from List?
Grahame Grieve (May 07 2019 at 16:10):
yes we all know how to drive the big bulldozer (brrrrrrrm) but I don't think we always want to have to start it up just to get down the end of the driveway
Michael Donnelly (May 07 2019 at 16:11):
My idea (just noodling, not a proposal yet) is to have a ListEntry resource that has the same elements as the List.entry (List-dot-entry) BackboneElement. Then a client can search for the ListEntry resources that are on a List.
Michael Donnelly (May 07 2019 at 16:11):
And then the client, like Michele said, can page through those results because it would be a normal search.
Michele Mottini (May 07 2019 at 16:11):
As I wrote above we solved it in our case adding a ../List/$getPage?pageSize=N&pageNumber=M operation (and we tag the returned partial list as SUBSETTED) - that seems to be Grahame's $subList
Grahame Grieve (May 07 2019 at 16:12):
yes your $getPage is pretty much the same as $subList - doesn't matter what it's called
Michael Donnelly (May 07 2019 at 16:12):
Michele, how easy did clients find implementing that?
Michael Donnelly (May 07 2019 at 16:13):
And Grahame or Michele, what happens if the server wants to enforce paging even if the client doesn't ask for it?
Michele Mottini (May 07 2019 at 16:13):
Don't know! Do not have any doing that yet (but it does not seem difficult)
Michael Donnelly (May 07 2019 at 16:13):
I agree, but I've been a bad judge of that sometimes in the past.
Michele Mottini (May 07 2019 at 16:14):
If the list is too big we return a 422 'too expensive'
Michael Donnelly (May 07 2019 at 16:14):
And then the client can come back and ask again with a pageSize (or a smaller one)?
Grahame Grieve (May 07 2019 at 16:14):
and maybe a header link see..aslo... if you really want to be thorough
Michele Mottini (May 07 2019 at 16:14):
Yes
Michael Donnelly (May 07 2019 at 16:15):
Do you have an OperationOutcome that lets the client know how big a request is allowed?
Michele Mottini (May 07 2019 at 16:15):
No, did not go that far
Michael Donnelly (May 07 2019 at 16:15):
But it wouldn't be hard.
Michele Mottini (May 07 2019 at 16:15):
Don't understand 'see also' ?
Michele Mottini (May 07 2019 at 16:15):
No, not hard
Michele Mottini (May 07 2019 at 16:16):
(and our lists are read-only, so no need for $add, $remove....)
Michael Donnelly (May 07 2019 at 16:18):
I think that's a perfectly reasonable solution.
I still think breaking out ListEntry would be easier for implementers (if harder for the standard), since developers of clients and servers would already have code to handle search paging, so there wouldn't be incremental effort for this. BUT I'm not going to die on that hill.
Grahame Grieve (May 07 2019 at 16:18):
see also - you could add a related link in an http header to advise clients that if they want to access a list you have said is too big, they can go there. (that's HATEOS). Of course, they will only know how to use 'there' if they have the secret knowledge from this conversation somehow, so it's not that useful
Grahame Grieve (May 07 2019 at 16:19):
breaking into listEntry would be better for searching for List, but really bad for almost every other use of list - e.g. in documents, or just maintaining the list
Jim Steel (May 07 2019 at 16:20):
Could an alternative be to have an operation that could generate/return a sublist, perhaps even in the form of a bundle if you want to use that paging mechanism? (not advocating for it, just spitballing)
Michael Donnelly (May 07 2019 at 16:21):
Hmmm.
Jim Steel (May 07 2019 at 16:21):
e.g. List/[id]/$enumerate?_count=10
Grahame Grieve (May 07 2019 at 16:22):
that's what Michele has done and I proposed
Michael Donnelly (May 07 2019 at 16:22):
That's pretty much what @Michele Mottini did, I think .
Jim Steel (May 07 2019 at 16:22):
oh, sorry. great minds, I guess
Michele Mottini (May 07 2019 at 16:23):
OK - so agreed that the operation is the 'right' solution...should that be in the specs? Or not worth is?
Stefan Lang (May 07 2019 at 16:24):
Should it be an operation or might it be just a link element (as in Bundle) plus search-like parameters?
Grahame Grieve (May 07 2019 at 16:25):
yes please create a task to add $getPage, $add $remove to the spec.
Yunwei Wang (May 07 2019 at 16:30):
Is $add operation for adding item to the list?
Michele Mottini (May 07 2019 at 16:30):
Yes - so you do not have to POST or PUT the entire thing
Michele Mottini (May 07 2019 at 17:32):
Michael Donnelly (May 07 2019 at 17:54):
I think count is a great idea, but I'm concerned about offset.
What if instead we had an equivalent to the next link in a Bundle? As a server, if we have to support offset, we will need to iterate through the previous entries to count up to the offset. If we return an opaque session parameter (like we do for a search), the client can pick up where it left off without an additional burden on the server.
Grahame Grieve (May 07 2019 at 18:07):
see extensive discussion of this on the terminology stream with regard to expand. What I've learnt is that people feel very passionate about this but I can't figure out why
Michele Mottini (May 07 2019 at 18:08):
Copied that from $expand - for a List and Group the server should have the complete list of entries / members stored somewhere, so jumping to an offset does not seem that hard? (On searches is more complex because those are generated on the fly)
Michael Lawley (May 07 2019 at 18:11):
What should happen if the List resource changes between iterations?
Grahame Grieve (May 07 2019 at 18:14):
offset problems should happen. Whatever the approach is
Yunwei Wang (May 07 2019 at 18:16):
@Michael Lawley Good point! since ValueSet expansion does not change that often.
Michele Mottini (May 07 2019 at 18:18):
What should happen if the List resource changes between iterations?
mhh..nothing? If offset is beyond the end return no entries / members (that would be the same behavior even if the resource did not change)
Yunwei Wang (May 07 2019 at 18:21):
@Michele Mottini For example, client got first 100 item, then a new item was inserted at index 50, then the client requested next 100, the client could get the same item (index 100 before insert and index 101 after insert) twice.
Michele Mottini (May 07 2019 at 18:22):
Yes - I'd be ok with that
Nick Radov (May 07 2019 at 18:24):
What should happen if the List resource changes between iterations?
It should work the same way as a typical relational database cursor. Although there is some variation between database products, the usual approach is to take a snapshot during the initial request. Then as long as the cursor is open the client can page back and forth through that snapshot and won't see any changes.
Yunwei Wang (May 07 2019 at 18:24):
It is possible for Server to inform client that the list is changed. Other than that I don't know what else the server could do.
Yunwei Wang (May 07 2019 at 18:26):
@Nick Radov In theory this is good. I guess this put burden on server side complexity.
Michele Mottini (May 07 2019 at 18:28):
I guess this put burden on server side complexity.
Not only on the server - the client should know / communicate which snapshot it is paging through, you'll need the same approach as a normal search - with next / previous links in the response etc. I would not go there
Michael Lawley (May 07 2019 at 18:34):
If you use the ETAG of the underlying Resource, then that should be usable to detect whether a change has happened; simple for client & server
Michael Donnelly (May 07 2019 at 19:00):
Not simple for the server. If there's a list of 3500 entries and a client asks for 100 of them, the server would need to look at all 3500 to generate an etag.
nicola (RIO/SS) (May 07 2019 at 20:27):
There are problems not only with the size of huge resources but with concurrent updates and history as well. ListEntry, GroupEntry, Element, Concept is an approach to efficiently reuse existing FHIR API and implementations
nicola (RIO/SS) (May 07 2019 at 20:29):
Otherwise, we have to add more and more irregularity and duplication into implementations.
Jim Steel (May 07 2019 at 20:31):
I'm not sure why adding resource types would be any less disruptive than adding operations
Yunwei Wang (May 07 2019 at 20:32):
Still don't understand how Concept could help paging ValueSet expansion
Jim Steel (May 07 2019 at 20:32):
... which already works
nicola (RIO/SS) (May 07 2019 at 20:32):
Because they duplicate logic of Search/ POST/PUT/PATCH
Yunwei Wang (May 07 2019 at 20:33):
Can you clarify?
nicola (RIO/SS) (May 07 2019 at 20:33):
Still don't understand how Concept could help paging ValueSet expansion
Concept resource turn expansion into FHIR Search
Jim Steel (May 07 2019 at 20:34):
expansion does a very different thing to FHIR Search
nicola (RIO/SS) (May 07 2019 at 20:35):
What is the difference?
Yunwei Wang (May 07 2019 at 20:37):
@nicola (RIO/SS) I guess your solution is that ValueSet.expansion contain [0..*] Concept and each Concept has a reference back to ValueSet so you search Concept?valueSet.id=12345. Is that correct?
nicola (RIO/SS) (May 07 2019 at 20:37):
In 80% yes :)
nicola (RIO/SS) (May 07 2019 at 20:37):
may be more
nicola (RIO/SS) (May 07 2019 at 20:38):
In real apps probably 95%
nicola (RIO/SS) (May 07 2019 at 20:41):
I'm pretty sure efficient terminology server do the same materialization internally?!
Jim Steel (May 07 2019 at 20:41):
Nope
nicola (RIO/SS) (May 07 2019 at 20:42):
Please, fix me if i'm wrong
Jim Steel (May 07 2019 at 20:43):
Our product doesn't do that. And we certainly consider it to be an efficient terminology server.
nicola (RIO/SS) (May 07 2019 at 20:43):
So you eval each expansion on every GET?
nicola (RIO/SS) (May 07 2019 at 20:44):
I think ratio read to write for terminology server in production is about 1000 / 1
Jim Steel (May 07 2019 at 20:44):
Well, that's complicated, but we don't store mappings of codes to the ValueSets of which they are a member
nicola (RIO/SS) (May 07 2019 at 20:45):
Give me an example, where this approach will not work
Jim Steel (May 07 2019 at 20:45):
You can always supply a new ValueSet with your expand request. Either as an implicit ValueSet, or as a POST parameter to $expand
Jim Steel (May 07 2019 at 20:45):
And this is a powerful thing to do, and something we do often
nicola (RIO/SS) (May 07 2019 at 20:45):
That's good for authoring - not for expluatation
Jim Steel (May 07 2019 at 20:45):
Ours is not primarily an authoring product
Jim Steel (May 07 2019 at 20:47):
I think we're well off topic. ValueSet is different from List and Group, in that it already has a Normative mechanism for paging content
nicola (RIO/SS) (May 07 2019 at 20:48):
Ok. But for me, it's a similar technical problem/anti-pattern.
nicola (RIO/SS) (May 07 2019 at 20:49):
We've already got a request to support List with millions of references :(
nicola (RIO/SS) (May 07 2019 at 20:50):
I'm a little bit scared - guys want to search by reference on these resources, probably next step is _revinlcude=* :).
nicola (RIO/SS) (May 07 2019 at 20:52):
Yes, we can hack it and apply a different strategy for storing and manipulating Lists - but that's will bring duplication
nicola (RIO/SS) (May 07 2019 at 20:57):
And yeah - internally we will do ListEntry for that :)
Michael Lawley (May 08 2019 at 11:05):
@Michael Donnelly I don't follow. If I have a List resource with 100000 members and you ask for 100..200, the ETAG for the List containing the subset is the ETAG for the original List, i.e., based on the technical version. No need to inspect anything.
Michael Donnelly (May 09 2019 at 00:38):
That depends on how the List is represented in the server. If it's a native FHIR server, and there's a List record with all those lines in it, you're great. But if the list is a virtual thing in a façade server, it's a horse of an entirely different color.
Michael Donnelly (May 09 2019 at 00:41):
For us, functionally speaking, returning entries if a List would be basically the same as returning entries in a search.
Michael Donnelly (May 09 2019 at 00:42):
In both cases, we (usually) use iterator functions. So each time we do a search, we call the "get the next result" function until we've hit the _count limit.
Michael Donnelly (May 09 2019 at 00:43):
When we hit that limit for a search, we set a bookmark, generate a session token associated with it, and return the session token with the results.
Michael Donnelly (May 09 2019 at 00:43):
When a client hits the next url, it includes the session token, and we can find the bookmark and keep searching.
Michael Donnelly (May 09 2019 at 00:43):
If we had to do an offset instead of a bookmark, we'd start at the beginning and count the entries over which we iterated.
Michael Donnelly (May 09 2019 at 00:44):
And we hit the offset, we would start getting results.
Michael Donnelly (May 09 2019 at 00:44):
So we'd have to loop over a bunch of entries - more each page - even though the client didn't need them.
Michael Donnelly (May 09 2019 at 00:45):
An ETAG for what's basically a search would be even worse. We'd need to iterate over the entire list, store it somewhere as an entire list, and take a checksum of it.
Michael Donnelly (May 09 2019 at 00:46):
We wouldn't be able to count on keeping that in memory, so we'd have to write the list to disk, checksum it, and then discard that storage.
Michael Donnelly (May 09 2019 at 00:46):
Not excellent from a performance standpoint.
Michael Lawley (May 09 2019 at 00:48):
Right...so your List is really a view of a search result / a dynamically generated thing; you don't have an internal list of references that are pro-actively maintained?
Michael Donnelly (May 09 2019 at 00:50):
Correct.
Michael Donnelly (May 09 2019 at 00:50):
So I think we have a three by two array of options (if we're using operations instead of new resource and a search):
1. The client specifies an offset.
2. The client specifies a bookmark (basically, the last entry it got last time).
3. The server specifies a session token / a next link.
Michael Donnelly (May 09 2019 at 00:51):
The other axis is with or without ETAGs.
Michael Donnelly (May 09 2019 at 00:51):
Option 2 or option 3 would be equally good for us.
Michael Donnelly (May 09 2019 at 00:51):
Would option 2 be as good for you as option 1?
Michael Lawley (May 09 2019 at 00:52):
But if a client wants items 200..220 then they still need to iterate through with the next links and thus all the round-trip requests, rather than doing that server-side
Michael Donnelly (May 09 2019 at 00:52):
Instead of "give me 100 entries starting with entry 201" the client would say "give me 100 entries starting after f2e65bf5-ea34-4430-88ff-faa8a9fb816f" where f2e65bf5-ea34-4430-88ff-faa8a9fb816f is the 200th entry.
Michael Donnelly (May 09 2019 at 00:53):
Would the client need 200..220 if it hadn't already gotten 1..199?
Michael Lawley (May 09 2019 at 00:54):
how does the client know about f2e65bf5-ea34-4430-88ff-faa8a9fb816f? Is it reasonable to assume that a client will only be interested in entries from 201 if she has previously looked at/retrieved up to item 200?
Michael Donnelly (May 09 2019 at 01:31):
The client would only know that if she had retrieved the list through entry 200.
Michael Donnelly (May 09 2019 at 01:32):
I think a client would only care about 201 having seen 1..200, but I'm definitely interested in hearing about other use cases.
Grahame Grieve (May 09 2019 at 10:21):
this is the same discussion as for ValueSet.expand - state based or offset based.
@Michael Donnelly I expect that the statement "your List is really a view of a search result / a dynamically generated thing" will not necessarily be true for all lists you deal with (in fact, it's likely that most legacy systems will have a mix of literal and inferred lists like this)
Michael Lawley (May 09 2019 at 10:26):
Similar, yes, although I expect that there's a lot more literal Lists than there are literal ValueSet expansions.
Grahame Grieve (May 09 2019 at 10:30):
not in USA where the habitual practice is distributing expansions
Michael Lawley (May 09 2019 at 10:38):
as I have only recently begun to recognize
Rob Hausam (May 09 2019 at 10:42):
"not in USA where the habitual practice is distributing expansions" is a habitual practice in and around VSAC and quality measures, but I don't think it's really the established practice in general outside of that
Michael Donnelly (May 09 2019 at 12:37):
Michael Donnelly I expect that the statement "your List is really a view of a search result / a dynamically generated thing" will not necessarily be true for all lists you deal with (in fact, it's likely that most legacy systems will have a mix of literal and inferred lists like this)
True.
Michael Donnelly (May 09 2019 at 12:38):
Likely, though, the larger the list the higher the odds that it isn't a single composed entity.
Michael Donnelly (May 09 2019 at 12:39):
As for state-based vs offset-based, how do you feel about the middle ground where the client specifies the entry after which it wants new entries? I believe this gives both sides what they want. @Michael Lawley? @Grahame Grieve?
Michele Mottini (May 09 2019 at 12:59):
@Michael Donnelly : what are your list? From what you write I assume they cannot be created / modified via FHIR
Michael Donnelly (May 09 2019 at 13:21):
What we've done so far is a List of Patients. It's the same patient list that clinicians and administrative staff use in the hospital. This includes:
- A physician's "My List" of patients he or she is attending
- A list of all the patients roomed in a specific unit
- A census of all the patients in the hospital
These are actually two kinds of list. One is built in the system with rule-based criteria (e.g. "patients in the ICU"). The other is a manually managed list to which a user adds and removes patients.
Michael Donnelly (May 09 2019 at 13:22):
The system list will never be modifiable with FHIR. The user-managed list can't be updated with FHIR yet, but we could add that in the future.
Michael Donnelly (May 09 2019 at 13:22):
We haven't gotten requests for it, but if someone in the App Orchard was interested, we would look into it.
Michael Donnelly (May 09 2019 at 13:23):
I haven't done a lot of work in that space, but my gut says that managing it with FHIR would be straightforward. (But don't quote me on that. :slight_smile: )
Michele Mottini (May 09 2019 at 13:24):
So the big one can be the patient in the hospital - some 1000s' ? Just do not implement $getPage - and do not worry about it
Michael Donnelly (May 09 2019 at 13:24):
I think the concept of bookmark-based paging through a list would also apply to valueset expansion.
Michael Donnelly (May 09 2019 at 13:24):
So the big one can be the patient in the hospital - some 1000s' ? Just do not implement $getPage - and do not worry about it
Not a bad point.
Michael Donnelly (May 09 2019 at 13:25):
No client yet has a use case where they need to fetch a list with more than 1,000 entries. I believe today that if a List will contain more than 1,000 entries, we return an error instead of the List.
Michael Donnelly (May 09 2019 at 13:26):
I'm always wary of things like that, though. Too often when I've said, "but nobody would ever need to..." I've later been proven wrong.
Michael Donnelly (May 09 2019 at 13:27):
So in your case, @Michele Mottini, how well would it work to request an offset by referring to the last entry you got rather than a number of entries?
Michele Mottini (May 09 2019 at 13:29):
We will have to search for that entry every time - not great, not too bad - but: that assumes that the same resource appears only once in a List (or Group) - and that is not in the specs
Michael Lawley (May 09 2019 at 13:34):
here's a thought - if the server has a "next token", then the server can keep an association between this token and the corresponding offset-based link, then recognises this in the next request from the client and substitutes the token to avoid re-performing the search.
Specifically:
1. client asks for /List/icu-patients?count=10
2. server performs search, iterates to construct List of Patients 0..9
3. server sets bookmark & generates session token then associates it with /List/icu-patients?offset=10
4. client gets List, does whatever, and then asks for /List/icu-patients?count=10&offset=10
5. server looks up url /List/icu-patients?offset=10 and finds a session token, continues iterating to build List of next 10 Patients, and continues as per step 3
If client instead asks for /List/icu-patients?count=10&offset=11, then server needs to re-perform search, skip over first 11, etc But, use cases suggest this is not a likely case.
Michael Donnelly (May 09 2019 at 13:35):
That would probably work. I'd worry about edge cases it might introduce.
Michael Donnelly (May 09 2019 at 13:36):
We will have to search for that entry every time - not great, not too bad - but: that assumes that the same resource appears only once in a List (or Group) - and that is not in the specs
That's a major flaw in my idea, then. :oh_no:
Michael Donnelly (May 09 2019 at 13:37):
Has there been discussion about why a List can include duplicate entries?
Michael Lawley (May 09 2019 at 13:59):
This is just your option 3, but with implicit next links. The main thing to be concerned about is that the client could re-issue its query with the offset in any place (and in fact reorder all parameters arbitrarily), so you need to do some URL normalisation process as part of the session token lookup
Michael Donnelly (May 09 2019 at 18:05):
Totally.
Michael Donnelly (May 09 2019 at 18:05):
I know we've taken that into account for search, so presumably we'd use the same logic.
Lloyd McKenzie (May 09 2019 at 18:36):
In theory an ordered List could indicate who you're seeing that day - and the same patient might appear more than once. Though you could fix that problem by making it a List of Appointments instead...
Michael Donnelly (May 09 2019 at 20:20):
Let me try a different angle: what need do clients have that would not be met by an opaque bookmark? Note that nothing would stop a server from returning an opaque bookmark that happened to be a numeric offset.
Brian Postlethwaite (Jul 08 2019 at 04:17):
This thread sort of just stopped.
One of the benefits of the next links used in the searching is that if the server doesn't like the page size provided, it can adjust it and continue, whereas the explicit use of count/page pairs just has to error out that the page size it too big.
Last updated: Apr 12 2022 at 19:14 UTC
