FHIR Chat · Operations to manage big List and Group resources · implementers

Stream: implementers

Topic: Operations to manage big List and Group resources

Michele Mottini (Sep 17 2019 at 20:37):

Re. https://gforge.hl7.org/gf/project/fhir/tracker/?action=TrackerItemEdit&tracker_item_id=21650
We are about to implement them in our server .... so some ideas / questions
For $add maybe instead of specify entire Group or List resources as parameters have only the items to add - as parameters with parts
For $remove instead of specifying offset and count use a list of referenced item to be removed

Michele Mottini (Sep 17 2019 at 20:37):

@Grahame Grieve @Nick Robison

Michele Mottini (Sep 17 2019 at 20:37):

(and not have optional offset for $add)

Michele Mottini (Sep 17 2019 at 20:44):

eg this would be a $add payload adding two members to a group:

<Parameters xmlns="http://hl7.org/fhir">
  <parameter>
    <name value="member"/>
    <part>
      <name value="entity"/>
      <valueReference>
        <reference value="Patient/123"/>
      </valueReference>
    </part>
  </parameter>
  <parameter>
    <name value="member"/>
    <part>
      <name value="entity"/>
      <valueReference>
        <reference value="Patient/234"/>
      </valueReference>
    </part>
    <part>
      <name value="period"/>
      <valuePeriod>
        <start value="2015-04-02"/>
      </valuePeriod>
    </part>
    <part>
      <name value="inactive"/>
      <valueBoolean value="true"/>
    </part>
  </parameter>
</Parameters>

Michele Mottini (Sep 17 2019 at 20:45):

And this would be the $remove payload to remove those same members:

<Parameters xmlns="http://hl7.org/fhir">
  <parameter>
    <name value="member"/>
    <part>
      <name value="entity"/>
      <valueReference>
        <reference value="Patient/123"/>
      </valueReference>
    </part>
  </parameter>
  <parameter>
    <name value="member"/>
    <part>
      <name value="entity"/>
      <valueReference>
        <reference value="Patient/234"/>
      </valueReference>
    </part>
  </parameter>
</Parameters>

Michele Mottini (Sep 17 2019 at 20:47):

If the same resource appears multiple time in the Group (or List) $remove removes all those members (items)

Michele Mottini (Sep 17 2019 at 21:17):

@James Agnew just saw in GForge comment that you are going to have a go to getPage - do you already have specific ideas?

Paul Church (Sep 17 2019 at 22:40):

Removing by value instead of offset avoids the problem of the list changing since the client last looked at it. Could solve with etags instead but then the client might have to retry many times.

Does $add insert another copy if one or more of the same member is already in the group?

Michele Mottini (Sep 17 2019 at 22:45):

My thinking was no - if the same member is already there it is left alone - just update period and inactive if the $add operation specified them
Does not seem to make much sense to have duplicates in a group - in our system they are flat out not allowed, and possibly in CMS as well.

Josh Mandel (Sep 17 2019 at 23:23):

From our discussion to day we are going to start with drafting the read operation. We were wondering for writes whether folks have tried this with existing patch operations?

Brian Postlethwaite (Sep 18 2019 at 00:03):

I'd prefer to use values or elementIds

Paul Church (Sep 18 2019 at 00:33):

Is there any consideration of a normal read or search giving back a SUBSETTED resource with only the first page of contents? The alternative may be a failure to return anything if the resource is unreasonably large. With only PUT operations it's difficult to create an unreturnable resource, but with $add it's easy.

I haven't implemented FHIRPath Patch but it looks like it already has add/remove type operators similar to what is being proposed. Has that had much adoption among clients or servers?

Josh Mandel (Sep 18 2019 at 00:48):

We talked about this in FHIR-I Q4 today. Summary:

• Some servers might not be able to return whole (large) resources like Group or List on read or search
• In this case they should not just return first-page results.
• Better to return an OperationOutcome indicating that the client needs to call $getPage (to avoid clients getting surprised/confused).

Josh Mandel (Sep 18 2019 at 00:50):

In GF#21650 we outlined this as:

We should define expectations on what servers should do when servers don't support READ but do support the paging operations.

Paul Church (Sep 18 2019 at 01:28):

My only immediate feedback on the idea of $getPage is that it should be as consistent as possible with other pagination. In the linked tracker it mentions "?offset=X&count=Y", and I think that should be "?offset=X&_count=Y".

Paul Church (Sep 18 2019 at 01:32):

I'd be even happier if there was also a consistent _page or _page_token or _offset or something where only the token is opaque instead of the entire request url being opaque, but that's a broader topic.

Josh Mandel (Sep 18 2019 at 01:40):

We were clear in the room (and I hope in our GForge resolution!) that we'll define this with opaque pagination tokens rather than count/offset.

Josh Mandel (Sep 18 2019 at 01:40):

• We would like to use the same mechanism we have used for paged Bundles (so, opaque next/prev tokens or links)

Brian Postlethwaite (Sep 18 2019 at 02:16):

The CodeSystem resource could also fit into this bucket of large resource manipulation too.

Brian Postlethwaite (Sep 18 2019 at 02:18):

(and also has its own paging style outside the fhir search)

Ewout Kramer (Sep 18 2019 at 02:28):

We also said in the resolution that we want to look at PATCH instead of the $add, $remove. @Michele Mottini have you considered doing that?

Paul Church (Sep 18 2019 at 03:38):

Related problems introduced by having resources too large to return: what about vread and history?

What about the return body from a successful patch operation - I guess the server can choose representation=minimal if it can't do more?

Paul Church (Sep 18 2019 at 03:40):

For vread, can just define $getPage on /_history/id/$getPage.

I don't have any immediate ideas for how history could look.

Michele Mottini (Sep 18 2019 at 12:32):

@Paul Church for resources too big to return we fail them with a 413 and an OperationOutcome suggesting to use _summary

Michele Mottini (Sep 18 2019 at 12:32):

(and we do not have history....)

Michele Mottini (Sep 18 2019 at 12:32):

@Brian Postlethwaite ...and ValueSet

Michele Mottini (Sep 18 2019 at 12:33):

@Ewout Kramer no, we do not currently have any support for path and it seems a lot of work to go from that format to adding / removing elements

Michele Mottini (Sep 18 2019 at 12:34):

(does the .NET library offers anything for PATCH? I don't actually know)

Brian Postlethwaite (Sep 18 2019 at 12:43):

Not at the moment @Michele Mottini

Michele Mottini (Sep 18 2019 at 12:46):

I meant: paging could be useful for ValueSet as well

Lloyd McKenzie (Sep 18 2019 at 12:47):

We have paging already through the expand operation. Are you looking for paging for really long enumerated value sets?

Michele Mottini (Sep 18 2019 at 12:48):

Yes - ValueSets in our system are just flat lists of codes

Yunwei Wang (Sep 20 2019 at 16:58):

@Michele Mottini Do you save those in ValueSet.compose?

Yunwei Wang (Sep 20 2019 at 17:04):

@Josh Mandel This leaves ValueSet.$expand having different behavior. I remember there was a strong push back from Vocab the last time we discussed to use paging token. @Rob Hausam

Michele Mottini (Sep 20 2019 at 18:00):

@Yunwei Wang yes, ValueSet.compose.include

Michele Mottini (Sep 24 2019 at 23:03):

OK, so what I have now is ..../Group/5e8188d4-63d6-e911-b7f2-2816a856e19d/$getPage?element=member&count=2, that returns extensions within meta:

{
    "resourceType": "Group",
    "id": "5e8188d4-63d6-e911-b7f2-2816a856e19d",
    "meta": {
        "extension": [
            {
                "url": "http://careevolution.com/fhirextensions#first",
                "valueUrl": "http://localhost:8080/fhir/Group/5e8188d4-63d6-e911-b7f2-2816a856e19d/$getPage?element=member&query=dd7437e3-23f8-4a26-9716-a43b39040fa6&start=1&count=2"
            },
            {
                "url": "http://careevolution.com/fhirextensions#next",
                "valueUrl": "http://localhost:8080/fhir/Group/5e8188d4-63d6-e911-b7f2-2816a856e19d/$getPage?element=member&query=dd7437e3-23f8-4a26-9716-a43b39040fa6&start=3&count=2"
            }
        ],
        "tag": [
            {
                "system": "http://hl7.org/fhir/v3/ObservationValue",
                "code": "SUBSETTED"
            }
        ]
    },

Michele Mottini (Sep 24 2019 at 23:04):

specifying first / next / previous URLs - similar to a normal search

Michele Mottini (Sep 24 2019 at 23:05):

You have to specify the element you want to page in the operation. Not sure how best to return that info - an extension within the paged element itself?

Paul Church (Sep 25 2019 at 00:11):

Is it useful to specify that in the operation? I assumed there would be only one paged element per resource type and it would be fixed.

Michele Mottini (Sep 25 2019 at 00:37):

Yes, but the notes in GF#21650 say 'where each method would specify which element gets pages'

Paul Church (Sep 25 2019 at 00:42):

I read that as saying that the definition of Group/$getPage would specify which element gets pages, and the definition of ValueSet/$getPage would specify an element on ValueSet, and so on - rather than "which element" being a parameter to the operation.

Josh Mandel (Sep 25 2019 at 14:04):

Indeed, @Paul Church you read our intent correctly (from the working group discussion).

Michele Mottini (Sep 25 2019 at 14:06):

OK, I'll remove that parameter it (easier)

Michele Mottini (Sep 25 2019 at 14:07):

Ideas about 'We would need to find a way to make it clear that a server is not returning the full resource, maybe beyond the subsetted marker (metadata about the element paged on?)' - what about a SUBSETTED or similar extension inside the element itself?

Michele Mottini (Sep 25 2019 at 22:18):

For adding I have POST .../Group/5e8188d4-63d6-e911-b7f2-2816a856e19d/$add

{
  "resourceType": "Parameters",
  "parameter": [
    {
        "name": "member",
        "part":[
            {
              "name": "entity",
              "valueReference": { "reference": "Patient/aefc2155-69c6-4264-8d7e-db32c815fc9f" }
            }
        ]
    },
    {
        "name": "member",
        "part":[
            {
              "name": "entity",
              "valueReference": { "reference": "Patient/d42ffdd2-df47-4b13-9f7e-c56c438a2f67" }
            }
        ]
    }
  ]
}

adds those two patients

Michele Mottini (Sep 25 2019 at 22:19):

and POST ../Group/5e8188d4-63d6-e911-b7f2-2816a856e19d/$remove

{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "entity",
      "valueReference": { "reference": "Patient/aefc2155-69c6-4264-8d7e-db32c815fc9f" }
    },
    {
      "name": "entity",
      "valueReference": { "reference": "Patient/d42ffdd2-df47-4b13-9f7e-c56c438a2f67" }
    }
  ]
}

removes them

Josh Mandel (Sep 25 2019 at 22:33):

Cool! Scanning the discussion above, was there anyone who tried to use PATCH for this?

Michele Mottini (Oct 15 2019 at 18:03):

An implementation of Groups with $add, $remove and $getPage is now available on our sandbox https://fhir.careevolution.com/Master.Adapter1.WebClient/fhir

Brian Postlethwaite (Oct 17 2019 at 09:18):

I'll give that a try later @Michele Mottini, looks good. Are there op definitions for them too?
I might add them to my server too.

Michele Mottini (Oct 17 2019 at 12:44):

Yes, there are - contained in the conformance

Brian Postlethwaite (Oct 18 2019 at 07:31):

Perfect. Thanks.

Brian Forbis (Nov 15 2021 at 15:46):

Hi, is this operation a candidate for being promoted up into the HL7 guides? Wasn't sure if there was a roadmap for this or not.

Last updated: Apr 12 2022 at 19:14 UTC