FHIR Chat · Confused about URN fullUrl's and resource.id's · implementers

Stream: implementers

Topic: Confused about URN fullUrl's and resource.id's

Pieter Edelman (Jan 08 2020 at 13:08):

Going back to this thread: https://chat.fhir.org/#narrow/stream/179166-implementers/topic/Resource.2Eid.20requirements. It talks about servers that support search but not read.

Let's sidestep for a moment the question of how desirable/likely this situation is, and just suppose I have a Bundle with resources that can't be queried individually. To make it possible for resources within the Bundle to reference each other, they all have a UUID based URN as entry.fullUrl (a similar situation might occur when using messaging for example). These UUID's are ephemeral, they are generated when creating the Bundle and not used for anything else.

We now have to take the following three rules into account:

• entry.fullUrl may not disagree with Bundle.resource.id
• entry.resource.id may not be absent
• entry.resource.id may not be a UUID based URN. It may be an unprefixed UUID though.

We're confused about these rules, as they seem to be in conflict. So what should be the proper way to treat entry.resource.id when the Bundle.fullUrl is a UUID based URN:

1. entry.resource.id should be the unprefixed version of the URN?
2. entry.resource.id should be absent?
3. something else?

In my opinion, there is an implicit assumption that resource.id is something that has a meaning in a RESTful context, and it should be able to construct an URL from it. If that's not possible, then it should be absent. Also, an unprefixed ephemeral UUID is not the same as a resource.id. So 2 is the way to go. But maybe this assumption is wrong.

Bonus question: in practice we see systems POSTing Bundles, using (ephemeral) UUID based URN's as fullUrl but also including a resource.id. Sometimes they use the UUID part of the URN, sometimes they use something else. These id's don't have a meaning, but is this allowed?

Lloyd McKenzie (Jan 08 2020 at 16:07):

entry.resource.id should be the unprefixed version of the URN. Within the scope of the Bundle, the UUID acts as the resource id. In theory, you could have multiple versions of the same resource in the same Bundle sharing the same id but with different versions (even though there's no ability to subsequently query/read them due to no proper base URL).

In answer to the second, it depends on the type of Bundle. If it's a transaction and you're doing 'create' actions, there shouldn't be ids. For other types of bundles (e.g. messages, documents), I'd expect the ids to be present.

René Spronk (Jan 09 2020 at 05:19):

..and expect them to be uuids.
I've seen some facades that allow search but not read. The server is simply unwilling to commit to using any 'solid IDs', so it's search + uuids in response bundles. Rather annoying if one has a FHIR mindset, but from the viewpoint of that legacy system I guess it makes sense.

Pieter Edelman (Jan 09 2020 at 07:29):

Ok, thanks for the clarification!

Eric Haas (Jan 09 2020 at 14:42):

So if you have a fullUri why do you need an Id to resolve references within the bundle?

Eric Haas (Jan 09 2020 at 14:43):

In other words can you only have the fullUri and not the I’d in a message bundle?

Eric Haas (Jan 09 2020 at 14:44):

Because it seems easier to omit them

Lloyd McKenzie (Jan 09 2020 at 14:57):

The spec currently says "The only time that a resource does not have an id is when it is being submitted to the server using a create operation." In practice, if you're using a UUID the expectation is that if you send a subsequent message, document, etc. where a resource is identified by the same UUID, then it must be the same resource. So it's possible to use UUIDs when you have a persistent identity and no REST interface. Ensuring there's always an id present means that resources function the same way regardless of where they're from.

Eric Haas (Jan 09 2020 at 19:01):

OK the validator didn't error me when sending message bundles without ids, I assumed that they and transaction integrity since they resolved based on the FullUri... so a $process-message is not a create but an operation so there is some ambiguity in the spec...

Eric Haas (Jan 09 2020 at 19:01):

or the validation

Lloyd McKenzie (Jan 09 2020 at 19:01):

Care to submit a change request to get expectations clarified? :)

Eric Haas (Jan 09 2020 at 19:02):

I will after a little more testing and review

Eric Haas (Jan 13 2020 at 00:46):

@Lloyd McKenzie there is a follow up issue on the messaging channel after looking into id.s vis a vis bundles and messaging... I am trying to understand how to know if the Bundle is truly the same instance when is resent.

I see where the spec states in the resource profile...

The only time that a resource does not have an id is when it is being submitted to the server using a create operation.

sounds like an invariant like statement, but probably not processable. Besides many resources are created on the fly and are only referenced by identifiers since I guess that they never touched a fhir server (yet). So an operation may result in the creation of a resource on a server or the resource may never persist beyond the transaction. so is kind of like a lemma to the above statement. I frankly think adding any more text will only muddy the waters and concede that the best practice is to just add ids (uuid's or whatever) to the Bundles.... I think that the ref servers are pretty generous on this point which led to my laziness in constructing bundles sans ids and may clear up some of my more esoteric validation errors.

Alexander Henket (May 18 2020 at 10:08):

I'd like to think that Resource.id has relevance only when it actually means you may retrieve this exact resource (albeit versioned) for eternity at the given base until deletion. A server SHALL NOT reuse any Resource.id. Any other use of Resource.id, e.g. populate with random uuid or a make-belief string, is ambiguous and undesirable.

So the relationship between fullUrl and Resource.id in my mind is like this:

• If Resource.id than SHALL have fullUrl of type absolute URL.
  • Resource.id MAY be an unprefixed OID (system/root and value/extension or system/root only if no value/extension)
• If Resource.identifier of type OID than MAY have fullUrl of type urn:oid: because OID SHALL be stable and persistent
• If all else fails then SHALL have fullUrl of type urn:uuid:

Lloyd McKenzie (May 18 2020 at 15:19):

I don't understand preferring OIDs to GUIDs. I'd much rather have GUIDs than OIDs because there's no need (or temptation) to try to extract meaning and they're a lot harder to get wrong.

Alexander Henket (May 18 2020 at 16:06):

Preferring OIDs over UUIDs is besides the point on Resource.id, but since you asked: the patient ssn, a national identifier for a doctor, a national identifier for an organization are all very common use cases for OIDs in v3. Most of our FHIR data starts its life a v3 data.

The V3 Class.id is the FHIR Resource.identifier, but serves the same purpose as the Resource.id. It uniquely and persistently identifies the Class instance (Act, Role, Entity, ...). So:

• if you have V3 data identified using an oid based Class.id (normally root + extension for us) and
• if that Class.id root+extension does not violate the FHIR Resource.id limitations of 64 characters and
• if you intend to serve that Class RESTfully based on the Class.id, then
• you MAY use the V3 Class.id as FHIR Resource.id

UUID may also be Resource.id if you intend to support read on it. An ad hoc UUID could not serve that purpose.

Lloyd McKenzie (May 18 2020 at 17:01):

I understand that much of your data might start as OIDs, but you should be converting as many of them as possible to URIs when you port to FHIR (and you MUST convert any of those where FHIR has defined a URI).

Alexander Henket (May 19 2020 at 07:30):

For internal identifiers we don't have system URI's. For a limited number of national ones we do. That leads to a system URI in the Resource.identifier.system, but the Resource.id could still be the original OID, hence:

<entry>
  <fullUrl value="urn:oid:2.16.528.1.1007.3.3.2727"/>
  <resource>
     <Organization>
        <identifier>
           <system value="http://fhir.nl/fhir/NamingSystem/ura"/>
           <value value="00002727"/>
        </identifier>
         ...
     </Organization>
  </resource>
</entry>

or

<entry>
  <fullUrl value="https://example.org/fhir/Organization/2.16.528.1.1007.3.3.2727"/>
  <resource>
     <Organization>
        <id value="2.16.528.1.1007.3.3.2727"/>
        <identifier>
           <system value="http://fhir.nl/fhir/NamingSystem/ura"/>
           <value value="00002727"/>
        </identifier>
         ...
     </Organization>
  </resource>
</entry>

Alexander Henket (Aug 18 2020 at 07:29):

We keep running into problems with the Resource.id for stateless facade situations. The current FHIR spec (+ tickets that have yet to be applied) thinks there SHALL be a Resource.id except for create / conditional updates. But for stateless facades serving for a non-FHIR system, there may or may not be an id.

For those systems it is not impossible to generate an ad hoc id, but normally the Resource.id is also the static RESTful id. Ad hoc conflicts with static.

So my question is: should a system infer RESTful availability from the presence of a Resource.id? If the answer is no we would feel free to follow the FHIR Core spec, and facades would insert an ad hoc id. If the answer is no then we have to infer that stateless FHIR facades cannot exist unless the backend supports RESTful ids. @Grahame Grieve

Grahame Grieve (Aug 18 2020 at 07:35):

I certainly don't understand all of this question. You have 2 'if the answer is no' but I don't understand other bits like inferring restful availability

Alexander Henket (Aug 18 2020 at 07:36):

If a resource carries a Resource.id: does that Resource.id have a to be stable?

Alexander Henket (Aug 18 2020 at 07:37):

If a resource carries a Resource.id: does that mean that I can RESTfull read on Resource/id?

Grahame Grieve (Aug 18 2020 at 07:37):

it's hard to answer that question without more context. If a resource has a different id, it's a different resource

Grahame Grieve (Aug 18 2020 at 07:39):

does that mean that I can RESTfull read on Resource/id

I don't think so, but it's not clear what your assumptions are that lead to that question. I certainly don't think that there's a general answer to the question. For a start, what restful server would you decide you could read from with that id safely?

Alexander Henket (Aug 18 2020 at 07:40):

When you query for data, you get a response Bundle. For a stateless FHIR facade it is not possible to generate stable Resource.id values. Hence it will generate new ids every time you query.

Grahame Grieve (Aug 18 2020 at 07:40):

so this is the result of a search on a restful interface?

Alexander Henket (Aug 18 2020 at 07:41):

Yes. It supports query, but not read based on id

Grahame Grieve (Aug 18 2020 at 07:41):

well, the first thing to say is that the id in that bundle can never be re-used to identify a different resource.

Grahame Grieve (Aug 18 2020 at 07:41):

but we never state anywhere that you have to be able to read a resource that you got in a search response

Grahame Grieve (Aug 18 2020 at 07:42):

I think it's a really unstable design but it's not non-conformant

Alexander Henket (Aug 18 2020 at 07:42):

Ok, so that means we cannot populate Resource.id as we cannot guarantee the same id every time you request the same resource

Grahame Grieve (Aug 18 2020 at 07:42):

that's not what i said.

Grahame Grieve (Aug 18 2020 at 07:43):

I said you can't re use the ids. You can give the resources different ids each time. Bad design, and would break lots of clients. But not non-conformant

Alexander Henket (Aug 18 2020 at 07:43):

Then I don't understand. We can guarantee uniqueness of Resource.id in the Bundle entries, but we cannot guarantee the same Resource.id, upon serving the next query that yields the same resource

Grahame Grieve (Aug 18 2020 at 07:44):

you need to guarantee not to re-use the same id for a different resource. That's not the same thing

Alexander Henket (Aug 18 2020 at 07:45):

Since generated ids are not stored anywhere on the facade, you could unknowingly regenerate the same id, could you not?

Alexander Henket (Aug 18 2020 at 07:46):

If the ids are as meaningless as they get in this case: would not populating them not be more safe?

Grahame Grieve (Aug 18 2020 at 07:46):

well, that depends on how you generate them. a UUID is sufficient. or a serial number.

Grahame Grieve (Aug 18 2020 at 07:47):

I'm assuming that you have to populate them for internal references in the bundle, and I'm also thinking that a search bundle needs ids, though I'm not sure we've specifically said that

Alexander Henket (Aug 18 2020 at 07:48):

The Bundle consistency works fine with fullUrl references without a need for Resource.id.

Grahame Grieve (Aug 18 2020 at 07:48):

if you say so. I'm not sure.

Alexander Henket (Aug 18 2020 at 07:48):

fullUrl is a UUID in these cases. The beauty about fullUrls is that they are not expected to be unique across different Bundles

Grahame Grieve (Aug 18 2020 at 07:49):

really? I'm not sure about that on search results. But why are UUIDs any different in a fullurl as opposed to a resource.id?

Alexander Henket (Aug 18 2020 at 07:49):

Resource.id on the contrary is expected to be persisted and guaranteed across resources over time.

Alexander Henket (Aug 18 2020 at 07:52):

After processing of a response Bundle, the fullUrls have served their purpose and may be forgotten. The Resource.ids live on 'forever'.

Grahame Grieve (Aug 18 2020 at 07:52):

if we say that, maybe we need to review that in regard to search results, but I'm not aware of us saying that

Alexander Henket (Aug 18 2020 at 07:54):

What's the best place to have a proper discussion on the topic? We really need closure on the expectations and boundaries of Resource.id.

Grahame Grieve (Aug 18 2020 at 07:56):

here and then resolution by FHIR-I

Alexander Henket (Aug 18 2020 at 08:04):

Alright, I'm in meetings for the next couple of hours. We've been discussing this issue in various ways before on Zulip and somehow this has not yet gotten to a solution. That must mean we/I have not explained ourselves right. So I'll try to craft something with everything we think we know, maybe as Google doc, and post that here as the basis for further discussion. Would that work or do you have a different proposal?

Grahame Grieve (Aug 18 2020 at 08:06):

I don't have a better proposal

René Spronk (Aug 18 2020 at 08:18):

In general there should not be the expectation that a) if one posts a new resource, that somehow the resource.id listed in it would be persisted/used by the receiving server, nor b) that resource.ids listed in a search result would be persisted by a client. The scope of use for a resource.id value, when talking about persistence, is limited to one single server.

René Spronk (Aug 18 2020 at 08:19):

Where does it say that a server, once it has communicated a resource with a certain resource.id, that it SHALL support reads based on that same resource.id ? It won't be able to guarantee that anyway, as the resource may be deleted at some future point in time.

Alexander Henket (Aug 18 2020 at 08:22):

At the very least you would expect, if you get data, that the data corresponds with the same resource every time. If you generate ids without keeping track of what you generated, say 2 months ago, it is possible that it generates the same id twice.

René Spronk (Aug 18 2020 at 08:24):

With UUIDs or the inclusion of a time-stamp as part of the id that's not very likely. But yes, one should not re-use one and the same resource.id for two semantically different resources. That's a fairly low bar for a facade.

Alexander Henket (Aug 18 2020 at 08:25):

But more importantly: the facades I'm talking about do not support read at all based on [type]/[id]. Why would you even need a Resource.id if not for RESTful read? The Resource.identifier is a different matter.

René Spronk (Aug 18 2020 at 08:27):

The Facade use case is not the 80%. Just populate resource.id with the fullURL minus its initial part. fullURL and id (=leafURL) are tightly related anyway.

Alexander Henket (Aug 18 2020 at 08:28):

The Resource.id, as generated by a face, should have added value to a receiver. I'm arguing there is no value. It is an ad hoc, fire and forget, value with no relationship to the backend system. Why does a facade have to populate data that has no value?

René Spronk (Aug 18 2020 at 08:34):

Because it's useful in most situations. The alternative, which is to make resource.id optional, would cause a lot more issues than the one faced by facades like yours.

Alexander Henket (Aug 18 2020 at 08:35):

Useful how? Only if RESTful read is possible? Or also in other cases?

René Spronk (Aug 18 2020 at 08:38):

In most FHIR implementations there is consistency in the use of resource.id as long lived identifier of a particular resource instance. If you start to question that, then you're questioning one of the basic principles of REST. You can, but this is hardly the forum to discuss the merits of one of the key REST principles.

Alexander Henket (Aug 18 2020 at 08:39):

I'm arguing that if you do RESTful read, you SHALL have persistent/unique Resource.id. I'm also arguing that if you cannot support RESTful read/persistent/unique Resource.ids you SHALL NOT populate them

Alexander Henket (Aug 18 2020 at 08:40):

The current spec requires me to populate Resource.ids regardless of my situation.

Alexander Henket (Aug 18 2020 at 08:42):

Note that Resource.id already is optional. It's the text around it that tries to convey the conditions on when you should/shall use it. (currently just POST and conditional update)

René Spronk (Aug 18 2020 at 08:44):

I can see your position, but the price we'd pay would be way to high IMHO to make such a change, whereas the price of the Facade generating some dummy ids is very very low. Feel free to create an issue with your detailed arguments ;-)

Pieter Edelman (Aug 18 2020 at 09:41):

@René Spronk, I don't really understand what the price to pay is.

At first glance, it may seem like a nice consistent rule to state that all resources in a searchset Bundle should have .id populated. However, its value in this context can never be interpreted in isolation since there is no indication of the _kind_ of id you're using. It might be an actual logical id that is persistent on the server, or it might be something temporary for the Bundle only. You will always need to inspect the corresponding fullUrl to figure this out.

So there's no actual _need_ for .id in searchset Bundles, and IMO it only leads to confusion if you allowing different kinds of id's in the .id field. It also breaks the expectation that the .id is "The logical id of the resource, as used in the URL for the resource. Once assigned, this value never changes." I think it would make much more sense to say that you only populate the .id field IF you have a logical, persistent id. Otherwise, you just use fullUrl for inter-Bundle references and forget about the whole .id field.

René Spronk (Aug 18 2020 at 09:56):

"you only populate the .id field IF you have a logical, persistent id" - but that condition is not testable. As such a client won't have a clue (ever) whether or not they'll receive an id. Lazy servers will omit the id by default. Inconsistency rules. That's the price I'm talking about.

Pieter Edelman (Aug 18 2020 at 10:17):

Yes, but the a client will never need to look at .id anyway (in searchset Bundle's), because it can only be interpreted in the context of .fullUrl. And .fullUrl doesn't need .id because it already contains all the information needed.
If the server always includes .id's, a lazy client might make the naïve assumption that they can simple be used as logical ids for RESTful reads, without bothering with .fullUrl's.

Alexander Henket (Aug 18 2020 at 11:33):

And don't forget that servers do not enter the network without vetting. At vetting time we inspect how they operate. Not having a Resource.id because of laziness will be detected and is not allowed. We firmly steer towards having Resource.id that is meaningful, but if a facade is unable to populate it, and we have no use case for 'read' we don't expect the facade to fake a Resource.id on the resources.

1. The client would otherwise never know what the value of the Resource.id is (if it truly identifies the object on the origin server or not)
2. The server may inadvertently be guilty of the biggest sin of all, which is to issue the same Resource.id twice on different resources (of the same type)

Alexander Henket (Aug 18 2020 at 11:37):

All in all the condition on Resource.id is testable, just not in an automated way. The upside for clients is that if they encounter a Resource.id, they know what it stands for and trust it.

Josh Mandel (Aug 18 2020 at 13:32):

This is really good discussion, and I look forward to seeing Alexander's proposal. I agree with premise here that populating resource.id sometimes provides no value, and can be hard for servers to do properly.

That said, @Alexander Henket, if you can share some more details about your facade architecture I would find that quite helpful. Often a facade sits in front of some other database or interface where data objects have some kind of internal identifier like a primary key -- which the facade can combine with some representation of the source system and hash to create a unique + stable fhir Resource.id. It would be great if you can share a few details about your system/constraints, because I bet it is quite different from what I described here :-)

Vassil Peytchev (Aug 18 2020 at 14:03):

One thing that is not clear to me, is whether the facade that only supports search, but not read, does not implement search using _id (i.e. GET [base]/resource?_id=33adne5528340)

Josh Mandel (Aug 18 2020 at 14:14):

I think if it is able to support stable ids, then it could choose to support this search parameter. But in Alexander's example, by definition the server could not and so would not support the search parameter.

René Spronk (Aug 18 2020 at 14:24):

It could support it, but there would (by definition) never be a match :-)

René Spronk (Aug 18 2020 at 14:28):

The Dutch use case (which Alexander refers to) is an aggregator, which by law isn't allowed to store any data itself. Receives a query from a client, sends queries (potentially using standards other than FHIR) to multiple applications (a selection of all Dutch national infrastructure enabled applications), aggregates the responses in a FHIR response bundle. The aggregator has no knowledge of the storage layer of the responding systems.

Michele Mottini (Aug 18 2020 at 16:29):

Don't see how things would work without ids - it is central to references, how you access data, pretty much everything in FHIR

Vassil Peytchev (Aug 18 2020 at 17:18):

Exposing the read operation is not the same as storing any data. Going though the effort to enable search and the ability to return search bundles, but not exposing a simple read sounds strange to me. Assuming that there are identifiers in the resources returned, the id can be a hash based on system and value.

Alexander Henket (Aug 18 2020 at 21:44):

The backends for these types of facades are V3. In V3 you have identifiers, but not ids. The aggregation that Rene refers to, could happen but doesn't at present: data comes from exactly 1 system. The facade is bridging a V3 messaging infrastructure that support some queries but not every type that FHIR theoretically could support. Some objects may be queried by identifier, but a V3 identifier may or may not be compatible with the special Resource.id requirements especially in STU3. A valid V3 identifier could look like this:

<id root="2.16.528.1.1007.3.3.112233445.1" extension="235"/>

This could be turned into a Resource.id by concatenation: 2.16.528.1.1007.3.3.112233445.1.235 carrying the same semantics in terms of persistency and uniqueness. A V3 identifier could also look like this:

<id root="2.16.528.1.1007.3.3.112233445.1" extension="b615a4a2-5fe1-4d6f-9411-81139ef7f2a6"/>
or this:
<id root="2.16.528.1.1007.3.3.112233445.1" extension="202009_abcd"/>

This could not be turned into a Resource.id because it exceeds 64 characters or the allowed character set respectively. But even more importantly in all three cases: how do I map a Resource.id back into a meaningful V3 query?

Suppose FHIR read of [base]/Observation/[id] ... how do I ever know whether I should issue a POOB_IN990001NL message with [id] split back into @root and @extension, or into a POLB_IN354001NL03 message? The first yields generic observations like length/height, the second yields lab observations.

Based on a read of Observation there could be 10-20 applicable different V3 queries. So even if I was able to construct a valid FHIR Resource.id from a V3 identifier... I would still be left in the dark. Should I just fire them all in V3 and hope one yields data?

Alexander Henket (Aug 18 2020 at 21:49):

@Vassil Peytchev Similar to read we cannot support _id in these situations. We never specify use of _id anyway and only talk about a subset of query parameters that FHIR supports. While systems are allowed to support more (within the use case boundaries), they are only tested for the specified set. We specify for use cases, not for general purpose FHIR servers.

Michele Mottini (Aug 18 2020 at 23:30):

How are you going to handle references between resource if there are no ids? eg how does a DiagnosticReport references its Observation, or an Observation its Patient or Encounter?

Josh Mandel (Aug 19 2020 at 01:54):

@Michele Mottini Within a bundle this is done using a resolution rules based on bundle entry full URL. Like this example; spec at http://build.fhir.org/bundle.html#references. For Alexander's use case there is no need to create references that go beyond the boundaries of a bundle.

Grahame Grieve (Aug 19 2020 at 04:56):

this is not an unusual situation when you are putting a pseudo-restful facade in front of a service orientated architecture that has no place for consistent identification. All the facade can do is pass the problem on downstream to the client, which can't consistently identify data between calls, for the same reason the facade can't. (such efforts start out looking each but always turn into a sisyphean task - https://www.cartoonstock.com/cartoonview.asp?catref=CX903792)

Vassil Peytchev (Aug 19 2020 at 05:44):

Well, maybe. The example seems to be wrong, because Reference.reference is defined as a URL, yet in at least one instance, it contains a URN.

Here is the short description of Reference.reference:

Literal reference, Relative, internal or absolute URL

Everything directly describing and defining Reference.reference insists that the string must be a URL.

Here is the definition of Bundle.entry.fullURL

URI for resource (Absolute URL server address or URN for UUID/OID)

Further explanation on fullURL

[...] each entry in a Bundle must have a fullUrl which is the identity of the resource in the entry. Note that this is not a versioned reference to the resource, but its identity. Where a resource is not assigned a persistent identity that can be used in the Bundle, a UUID should be used (urn:uuid:...).

Only then come the resolution rules for resources within a Bundle, and here we have a discrepancy, as there is an implication that Reference.reference can be a URI that is not a URL, and the example that shows reference containing a URN. Since this seems to contradict the definition of Reference.reference, I think the implication needs a technical correction, and the example needs to be fixed.

The way the normative content is written, to me it seems that the following should be true:
If the fullUrl is a URN, it can't be used in Reference.reference, only reference.identifier can be used.

For Alexander's case, using reference.identifier will allow for proper resolution. I think, however, that there is also value to provide Resource.id for the resources, by simply calculating the SHA1 or SHA256 of the root+extension, as that will be a consistent .id. This way you are not completely designing yourself into a dead end...

Grahame Grieve (Aug 19 2020 at 07:39):

I don't think that this interpretation is correct, actually. One of the problems here is that the definitions that W3C have provided are incomplete, inconsistent, and vague.

My working definition is:
uri: A sequence of characters that identifies an abstract or physical resource
url: a URI that can be resolved to a physical resource
urn: a URI that is globally unique and persistent even when the resource ceases to exist or becomes unavailable

From this I draw 2 conclusions:

• FHIR RESTful URLs are very definitely URLs and they are also URNs
• A URI can be a URL if it is well understand how it resolves to a physical resource. Whether it starts with urn: or not is not the point.

Hence: Reference.reference does need to be a URL, not a URI. It is intended to resolve. But it can be, in and only in the context of a bundle, a URL that starts with urn:

Alexander Henket (Aug 19 2020 at 12:31):

It took me a little more time than anticipated, but I've drafted a document with the issue at hand. It should be world writable:

https://docs.google.com/document/d/1RKuvZ5ttRnpaTOG4JntcpRz7BJFiHRTIRyOTwZlo0PY/edit?usp=sharing

Let me know if it is missing arguments. What I hope we land at, is an update in FHIR Core for Resource.id that either:

• Sets unambiguous, realistic rules for when a Resource.id SHALL be present and acknowledges that this may mean that are also cases where Resource.id is _not_ present, other than post/conditional update (preferred)

or

• Requires Resource.id to be present as it currently does, and acknowledges that Resource.id in this case may or may not have RESTful value (not preferred)

Michele Mottini (Aug 19 2020 at 12:35):

@Josh Mandel I do not understand. If a client search for Observation it gets back a bundle of observations - in-bundle resource references are not going to help to get to the encounters that those observation references, they are not in that bundle (or even if they are, the client would not expect them to be there - it asked for observations)

Alexander Henket (Aug 19 2020 at 12:36):

A server may include whenever it feels it needs to, regardless of client query. We normally anticipate clients having to request _include though

Alexander Henket (Aug 19 2020 at 12:38):

But yes... in certain corner cases the reference.identifier is being used (logical reference) which leaves systems with no other option than to marry Encounter.identifier in a different query to the logical reference.

Michele Mottini (Aug 19 2020 at 12:41):

.Yes, so it needs to be a client written specifically for that server behavior, a generic FHIR client won't work, that was Grahme point I think

Michele Mottini (Aug 19 2020 at 12:42):

I thing the specs should forbid that if they don't already do that

Alexander Henket (Aug 19 2020 at 12:43):

Logical references in general are always a dead end, but that was just the design I believe. A regular FHIR client would not know what to do there

Alexander Henket (Aug 19 2020 at 12:47):

But note: logical references are not the highlight of the topic. I'm talking mostly about Bundle with Observation that references an Encounter that is present in that same Bundle and referred to by means of its fullUrl. Any regular FHIR client would get that today. However as long as the Encounter.id is not present: it is considered non conformant

John Moehrke (Aug 19 2020 at 12:51):

This is more urgent in healthcare than it is in the general REST community due to the fact that healthcare is made up of thousands and thousands of servers, with data on any one patient split across hundreds. So we MUST be better able to deal with this. logical id is fine in a monolithic environment, but fails in healthcare. This is not to say that I want to eliminate logical id, I love it... but we must recognize that we are dealing with patients that are getting coordinated care (eg. carePlan) that is NOT on a monolithic REST server.

Alexander Henket (Aug 19 2020 at 12:52):

Understood. Logical reference has purpose, but should be avoided if possible because it suggests custom resolving.

Jens Villadsen (Aug 19 2020 at 13:07):

@Alexander Henket i totally follow and forsee that i have the same needs as you

Josh Mandel (Aug 19 2020 at 13:15):

@Grahame Grieve I am having trouble finding references that support your conclusion that a fhir restful URL can be considered a "URN". https://www.w3.org/TR/uri-clarification/ provide some context, but as far as I can tell, a necessary condition for something to be a URN is starting with the urn prefix.

Josh Mandel (Aug 19 2020 at 13:21):

@Michele Mottini I agree with your point that clients would not be able to follow normal references to get more data than they had originally queried for. There are probably circumstances where this is a reasonable trade-off. But it would be a big problem if any fhir server could choose to inflict this pain on clients -- like in the scope of patient API access in the US, if it suddenly became okay for a server to just break all links by omitting resource IDs, that would be a total fai

Grahame Grieve (Aug 19 2020 at 13:21):

well, I feel as though that document doesn't help, in that what you are referring to is "URN Namespaces" which is not quite the same as "a URN".

Josh Mandel (Aug 19 2020 at 13:23):

@Michele Mottini I agree with your point that clients would not be able to follow normal references to get more data than they had originally queried for. There are probably circumstances where this is a reasonable trade-off. But it would be a big problem if any fhir server could choose to inflict this pain on clients -- like in the scope of patient API access in the US, if it suddenly became okay for a server to just break all links by omitting resource IDs, that would be a total failure. Getting the conformance language right here is critical, and if we relax it in the base specification then many implementation guides and they need to make updates to tighten it downstream.

Josh Mandel (Aug 19 2020 at 13:24):

In practice it gets pretty painful to manage!

Grahame Grieve (Aug 19 2020 at 13:26):

the earlier RFC says that it's a name if it is reliably unique.

Vassil Peytchev (Aug 19 2020 at 13:32):

Here is RFC 8141 that defines URNs

The W3C link Josh shared, while not clarifying everything about URIs, seems to confirm my understanding that basically, URI is the union of URL and URN, and URLs and URNs are disjoint sets.

Jens Villadsen (Aug 19 2020 at 13:36):

Correct me if I'm wrong - but it is absolutely crucial to the spread of FHIR that it is usable as a façade layer in front of existing legacy infrastructure as @Alexander Henket describes

Jens Villadsen (Aug 19 2020 at 13:36):

which means that there are many situations where the Resource.id has little / no value

Grahame Grieve (Aug 19 2020 at 13:37):

that doesn't mean that it will always be possible to do it without pain

John Moehrke (Aug 19 2020 at 13:38):

when used as an API as a facade... I think it is important that Resource.id management is an expectation. It might not be easy, but it is necessary.

Jens Villadsen (Aug 19 2020 at 13:40):

That makes very little sense

Jens Villadsen (Aug 19 2020 at 13:42):

We've had similar discussions about this before with eg. how terminology services sort of differ from how clinical servers work

Jens Villadsen (Aug 19 2020 at 13:42):

And now there is a case for facades as well

Josh Mandel (Aug 19 2020 at 13:50):

For what it's worth: I actually think it is a specific (small?) subset of facade servers that have this problem; most facade servers I have come across at least are wrappers around existing structured data stores that just happen happen not to use fhir; they are perfectly capable of generating stable IDs and resolving references.

Jens Villadsen (Aug 19 2020 at 13:52):

Imaging an on-demand service behind a FHIR facade

Jens Villadsen (Aug 19 2020 at 14:02):

Or a service from which you only get sensitive business identifiers which may not be part of an URL in a request as it will be logged across all kind of different passing proxies. That is a real concern and a real problem of mine

Vassil Peytchev (Aug 19 2020 at 14:30):

At the bottom of the Logical ID section, we find the following (key phrase is in bold):

In some contexts, resources are not associated with location on a RESTful server, either because they are only created transiently for transfer between systems, or the systems are not using RESTful servers. In these cases, resources may be assigned some kind of location anyway, for purposes of consistency, or they might not have an assigned logical id, and they are identified based on other kinds of identifiers. See Resolving references in Bundles for one method of using resources not associated with RESTful servers.

We definitely need to clarify the language, even if we agree on the intent...

Alexander Henket (Aug 20 2020 at 06:53):

Just for the record: I agree with everybody who says we should push FHIR Servers to do what they can to arrange for Resource.ids. In fact that is what we already do. However: in a facade situation where the backend is not the system itself, but some form of messaging like V3 where there is no reasonable expectation that a Resource.id is persistent and unique for that Resource ... we currently allow the Resource.id to be omitted, and never expect a system to generate ad hoc Recource.ids just to comply with having one.

Jens Villadsen (Aug 20 2020 at 06:56):

Jens Villadsen said:

Or a service from which you only get sensitive business identifiers which may not be part of an URL in a request as it will be logged across all kind of different passing proxies. That is a real concern and a real problem of mine

• and in that case, issuing GET on the Resource/[id] is not an option. For the same reasons, the searches are conducted using the POST method with application/x-www-form-urlencoded parameters

Alexander Henket (Aug 20 2020 at 06:56):

Josh Mandel / Michele Mottini I agree with your point that clients would not be able to follow normal references to get more data than they had originally queried for.

I also agree with that. We require that servers that cannot support read actions on essential data attached to the focal resources, SHALL include this data with the response Bundle, thus eliminating the need for read actions. Inclusion of data is server discretion for as far as the FHIR spec goes, so a client does not have to ask for it.

Vassil Peytchev (Aug 20 2020 at 13:23):

In such cases, is it useful to pretend there is a RESTful search interface? Why not have these as operations?

John Moehrke (Aug 20 2020 at 13:31):

Or just present it as what it is, a messaging interface.

Jens Villadsen (Aug 20 2020 at 13:58):

Its not messaging as it aint async

Alexander Henket (Aug 20 2020 at 14:49):

In such cases, is it useful to pretend there is a RESTful search interface? Why not have these as operations?

I fail to see how that would help. Operations require far more work on the server side than simple queries. Nothing on the network would compare to anything in the outside world. It would do nothing for Resource.id. Why do you think it helps?

Michele Mottini (Aug 20 2020 at 14:50):

It makes clear to the client that they are not really dealing with a REST interface and that they are expected to do things in a different way

Alexander Henket (Aug 20 2020 at 14:52):

Sorry but no. We support RESTful query here, just not (always) RESTful read. It makes no sense at all do ditch RESTful just because we have a few limitations for some systems. The system works fine. It works fine with out-of-the-box reference frameworks. What you are asking here is to make it a proprietary network (messaging/operations). No can do

John Moehrke (Aug 20 2020 at 15:12):

REST is more about the resource id, than it is about query / search models. I think you are confusing REST with http GET

Michele Mottini (Aug 20 2020 at 15:17):

My thinking (and @Vassil Peytchev ?) is that it does things so differently from 'normal' FHIR server that is already proprietary - so better be upfront about it

John Moehrke (Aug 20 2020 at 15:22):

we tend to push complexity to servers... so either accept the need for resource.id and do extra work to de-duplicate and track things on the server for the benefit of clients not needing to do something different (meaning clients can query and retrieve). OR make it clearly different using an operation with more of a messaging pattern.

Alexander Henket (Aug 20 2020 at 17:20):

Ok, if you put it like that: could FHIR Core please remove the requirement for Resource.id based on the fact that there are more uses for FHIR than RESTful?

René Spronk (Aug 20 2020 at 17:24):

That's an interesting suggestion. Note that 'Bundles' (even if transported via HTTP) are strictly taken not RESTful ..

Alexander Henket (Aug 20 2020 at 17:36):

So does search even count as RESTful then? Every query yields a Bundle.

John Moehrke (Aug 20 2020 at 18:15):

Alexander Henket said:

Ok, if you put it like that: could FHIR Core please remove the requirement for Resource.id based on the fact that there are more uses for FHIR than RESTful?

quite possibly the right question. The model so far has been dominated by REST. It is possible some requirements that should be limited to REST have leaked in as absolutes.

Vassil Peytchev (Aug 20 2020 at 20:13):

I am repeating myself, but here it is:

At the bottom of the Logical ID section, we find the following (key phrase is in bold):

In some contexts, resources are not associated with location on a RESTful server, either because they are only created transiently for transfer between systems, or the systems are not using RESTful servers. In these cases, resources may be assigned some kind of location anyway, for purposes of consistency, or they might not have an assigned logical id, and they are identified based on other kinds of identifiers. See Resolving references in Bundles for one method of using resources not associated with RESTful servers.

Jens Villadsen (Aug 20 2020 at 20:13):

Well - the description on logical Id's as @Vassil Peytchev pointed out allows it

Jens Villadsen (Aug 20 2020 at 20:13):

oh snap ... double post!

Grahame Grieve (Aug 20 2020 at 21:10):

catching up on this thread:

@Alexander Henket I don't think that changing to an operation from a search is a big a deal as you made it out to be; it would effectively be changing from [base]/[type]?params to [base]/[type]/$query?params, but everything would otherwise be the same.

Michele's proposal to do this would make sense because:

• it makes clear that you are not dealing with the search obligations
• Clients won't accidentally get themselves in trouble; if they get in trouble with this approach (they will anyway!) you can at least be clear that they should've paid attention

OTOH, there's several reasons I can think of why it wouldn't be a good idea to do it.

Grahame Grieve (Aug 20 2020 at 21:10):

could FHIR Core please remove the requirement for Resource.id based on the fact that there are more uses for FHIR than RESTful

I'm not familiar with this requirement you are referring to here.

René Spronk (Aug 21 2020 at 07:08):

@Alexander Henket Whilst quite a few REST APIs use some sort of bundle principle for search results, most other bundles in FHIR are effectively an attempt to do non-RESTful things in a HTTP-transport context (messaging, documents, collections, transactions, .. - in all of those bundles it's easy to see that one may not have resource.ids, and that fullURLs are sufficient to establish linkages).

Note that resources in bundles could have a resource.id, which means that a receiving application can use that id to fetch additional (related) resources or that it could subscribe to updates.

The text cited by Vassil already allows for resource.ids to be omitted. If that wording is limited to bundles (of any kind) then certainly I'd agree with the wording.

Alexander Henket (Aug 21 2020 at 07:20):

@Grahame Grieve / @Michele Mottini I don't think operations make sense because it does not change anything for the FHIR resources you retrieve. They are still required to have the Resource.id's that we can't provide ... in some cases.

Also it means that the majority of systems that behave RESTfully, serve the exact same use cases, need different ways of accessing that information. So if I want my patient summary from system A I need to jump through hoop A and if want that very same patient summary from system B I need to jump through hoop B.

And finally: we have had the hardest time getting all systems to implement the few operations we have (like MedicationOverview). Not one reference framework has ever had any trouble implementing standard query behavior. From an economical perspective it does not make sense to do operations at scale. We do operations if it cannot be done otherwise.

There are really only two ways out of this is, imo:

• to lift the RESTful bias from the Resource.id, or
• to accept that Resource.id is not a guaranteed access key to a RESTful endpoint

I know what I would choose: reliability of what Resource.id means. If it exists, it SHALL be clear. If it does not: that's clear too.

Alexander Henket (Aug 21 2020 at 07:30):

@René Spronk

Note that resources in bundles could have a resource.id, ...

No that is not correct. FHIR spec says Resource.id SHALL be present except for POST and there is an accepted but not yet applied ticket from Ewout that argues "conditional updates". Any other Resource regardless it being retrieved by operation or query, sent in a Bundle, SOAP, messaging or contained SHALL have a Reource.id.

René Spronk (Aug 21 2020 at 07:37):

Contrary to my initial position I'm now suggesting to add such wording: that resource.id in Bundles (of any type) may not be present.

Jens Villadsen (Aug 21 2020 at 07:56):

In addition to support @René Spronk and @Alexander Henket a HL7 standardized extension to the CapabilityStatement resource could be to flag the behavioural pattern eg. and extension that clearly states that Resource.id is not to be expected in the bundles or alike (a hierarchy of extensions could also be suitable if eg. certain behaviour can be established - eg. an extension saying this service acts as a façade (whatever fixed behaviour that means) ).

Vassil Peytchev (Aug 21 2020 at 13:38):

FHIR spec says Resource.id SHALL be present except for POST and there is an accepted but not yet applied ticket from Ewout that argues "conditional updates".

I am confused. Where does it say that?

Grahame Grieve (Aug 22 2020 at 21:18):

@Alexander Henket I've had a chance to read your document. In that document, you write:

We also think that if a Resource.id is present, and the origin server is a RESTful server at all, it SHALL be capable of serving the resource by means of its Resource.id

This is ambiguous. Are you saying that this is a rule that you want to make? Well, you can, but I want to be clear that this is not something that the core spec says, not will it ever say this.

What is still not clear to me is how much your whole argument is based on this position

Grahame Grieve (Aug 22 2020 at 21:26):

There's been several references in this thread to an unapplied task from Ewout about this, but I'm not sure which task this is. J#20390 ?

Grahame Grieve (Aug 22 2020 at 21:27):

ah no it's J#20651. The resolution hardly helps us here

Grahame Grieve (Aug 22 2020 at 21:29):

I don't think operations make sense because it does not change anything for the FHIR resources you retrieve

well, specifically, the spec says that 'results from operations may include resources that are not identified'. Just so you know that

Grahame Grieve (Aug 22 2020 at 21:32):

I also want to point that out since the definition of fullUrl says:

The Absolute URL for the resource. The fullUrl SHALL NOT disagree with the id in the resource

I think for this reason that the uniqueness requirements for fullUrl are not less than for Resource.id.

Grahame Grieve (Aug 22 2020 at 21:33):

final comment: yes, there's some theoretical prospect that if your systems run for several million years, a duplicate UUID might occur. But generating a UUID is sufficient for conformance.

Grahame Grieve (Aug 22 2020 at 21:35):

However, none of that deals with your core issue. And it's tricky. See, if we accept that that the normative specification does say that a resource.id must be populated in a search result, we also accept that this isn't something we can fix, because it's a normative requirement.

If, on the other hand, we accept that it's ambiguous, then we accept that (a) it can be clarified but (b) you can leave it out without being conformant (so we don't need to fix it)

Lloyd McKenzie (Aug 23 2020 at 22:31):

I don't understand the desire to leave out the id given that you have to have the fullUrl and the fullUrl and id are expected to align. Having/not having the id in no way implies whether you can/can't do things you'd expect to do with the id - whether that's read it, update it, retrieve a history for it or reference it in resources created elsewhere. Whether you can do most of those things (read, history, update, etc.) are governed by the CapabilityStatement of the server from which you retrieved the results. If it's a facade, it's fine for it to support search but none of the other operations. Where things get dicey is when you retrieve a resource in a search and then create an instance somewhere that references a resource you saw in a search. Nothing stops anyone from doing that. However, a reference to a resource that isn't retrievable via a 'read' isn't terribly useful, so it might be worth making a statement that it should not be done.

In essence, a facade that's incapable of maintaining a persistent id or a resource is generating a new transient resource instance on the fly every time it executes the query. That resource has an id - the same as in the fullUrl. It's redundant, yes, as the id and fullUrl are always intended to be redundant. However, that redundancy is there for expediency of processing. There are times when you want to process using the fullUrl and there are times you want to process using the id and the pain of getting from one to the other is sufficient that having both present was deemed appropriate and necessary.

I agree there should be absolute clarity about whether the id can be expected to be present (I argue for 'yes' because it makes life easier for processors - and is why we have both it and fullUrl there in the first place). But I don't think we should be ascribing any meaning to the presence of a fullUrl and absence of an id. The fact a resource can't be 'read' is conveyed by the CapabilityStatement, not any characteristic of the resource itself.

Alexander Henket (Aug 24 2020 at 05:19):

The position is this:

FHIR Core requires Resource.id except for a few cases. The few cases do not match circumstances for some or our systems. Resource.id does not have purpose outside being an endpoint for RESTful calls or when a resource is contained.

As long as long FHIR keeps Resource.id required, we have no alternative but to generate the Resource.id. So apparently generating a meaningless Resource.id is better than omitting it. I don't like it, but I'm not sure how to break the RESTful bias here.

Having/not having the id in no way implies whether you can/can't do things you'd expect to do with the id - whether that's read it, update it, retrieve a history for it or reference it in resources created elsewhere.

If RESTful read/update/delete and search by _id is governed by Resource.id and just about any client is taught to work with Resource.id that way: does it not become the de facto purpose of the Resource.id? Why would I have a Resource.id I cannot support, and mitigate/hide that fact by means of a CapabilityStatement?

Grahame Grieve (Aug 24 2020 at 06:37):

I don't like it, but I'm not sure how to break the RESTful bias here

I'd feel bad about that, except that you're actually implementing the RESTful API here. If, for instance, you were doing some service thing that was using bundles in some other context, and complaining about RESTful orientation, then I'd be inclined to give it a little more weight

René Spronk (Aug 24 2020 at 06:54):

Hmm. Transactions aren't REST, yet we require resource.id in such bundles (whether transported using HTTP or not). Messages aren't REST, yet we require resource.id in such bundles (whether transported using HTTP or not). Documents aren't REST, yet we require resource.id in such bundles (whether transported using HTTP or not).
In many of these paradigms resource.ids don't make a lot of sense - e.g. most messaging systems won't expose a RESTful endpoint, so resource.ids in messaging are useless (we'll need fullURL though for referencing).

Grahame Grieve (Aug 24 2020 at 06:54):

I think that transactions are indeed RESTful - they are completely defined in terms of the REST Api.

Grahame Grieve (Aug 24 2020 at 06:55):

Do we require Resource.id in a message? where does it say that?

René Spronk (Aug 24 2020 at 07:00):

http://build.fhir.org/resource.html#id (we only have the high level requirement) - there is no special wording about messaging and resource.ids AFAIK.

Grahame Grieve (Aug 24 2020 at 07:01):

I'm pretty sure that we have a task to clarify that those words about Resource.id are applicable in the context of the RESTful api, and other rules apply outside that

René Spronk (Aug 24 2020 at 07:02):

As for transactions being restful - what we have seems like a patch to me. What % of APIs have some sort of transaction functionality ? Does the highrise API support transactions ?

Grahame Grieve (Aug 24 2020 at 07:07):

I haven't seen transactions elsewhere.

Grahame Grieve (Aug 24 2020 at 07:07):

I have seen batches elsewhere

René Spronk (Aug 24 2020 at 07:11):

Transaction feels like an operation to me. Operations (services) aren't REST. As such "other rules may apply to that". Anyways, I hope there already is a workitem to clarify the current wording.

Grahame Grieve (Aug 24 2020 at 07:12):

right, indeed other rules might apply to an operation. but since the details of transaction are defined by the RESTful API, the rules around whether a resource.id must be present or not are defined by the RESTful API. e.g. there are conditions where Resource.id must not be present

Alexander Henket (Aug 24 2020 at 12:04):

Resource.id is defined here: http://build.fhir.org/resource-definitions.html#Resource.id

it says:

Definition: The logical id of the resource, as used in the URL for the resource. Once assigned, this value never changes.
Cardinality 0..1
Type id
Summary true
Comments: The only time that a resource does not have an id is when it is being submitted to the server using a create operation.

This does not list any other exception than create. It does not list operations, messages, conditional updates (yet) or anything else.

On this page http://build.fhir.org/operations.html#response I read the exception you are referring to

The resources that are returned by the operation may be retained and made available in the resource repository on the operation server. In that case, the server will provide the identity of the resource in the returned resources. When resources that are not persisted are returned in the response, they will have no id property.

This is most interesting as it goes directly against the comment on resource-definitions.html. At the very least there is a consistency problem here. I would argue to move/copy the operations exception to the main definition for Resource.id as a single of truth.

But coming back to:

If, for instance, you were doing some service thing that was using bundles in some other context, and complaining about RESTful orientation, then I'd be inclined to give it a little more weight

I don't see how this position holds unless you feel that search always is a RESTful action. I'd say search can be RESTful. RESTful search and not-so-RESTful search could coexist perfectly, and Resource.id is the key to that.

Vassil Peytchev (Aug 24 2020 at 13:09):

I don't see how the Comment on Resource.id has some overall normative hold that overrides the whole section in the content.

Search is a RESTful interaction by definition, since it is defined as part of the RESTful API

Alexander Henket (Aug 24 2020 at 14:03):

@Vassil Peytchev from your Logical Id link

In some contexts, resources are not associated with location on a RESTful server, either because they are only created transiently for transfer between systems, or the systems are not using RESTful servers. In these cases, resources may be assigned some kind of location anyway, for purposes of consistency, or they might not have an assigned logical id, and they are identified based on other kinds of identifiers

Emphasis applied by me. Do I read what I want when I say this supports what we do?

Alexander Henket (Aug 24 2020 at 14:04):

I think I can also read in this fragment that Resource.id and RESTful are explicitly interlinked.

Richard Braman (Aug 26 2020 at 22:12):

(deleted)

Alexander Henket (Sep 21 2020 at 13:31):

Okay, say a server generates Resource.id based on UUID and I request data twice. Different MedicationRequest.id (because of generated nature), same MedicationRequest.identifier (because in V3 this is the unique identifier for it)

What would you expect from a receiver? I'm fully expecting loads of virtual duplicates because they distinguish based on Resource.id which is generated and does not reflect something on the source server.

Alexander Henket (Sep 21 2020 at 14:06):

When a server generates a new uuid each time it serves a resource: does that not constitute a change in the resource.id and as such a direct violation of this part of the spec:

Once assigned by the server, the id is never changed.

Vassil Peytchev (Sep 21 2020 at 14:10):

I see two points here:

• How to generate Resource.id: If you have a unique identifier, is it not possible to "generate" Resource.id based of Resource.identifier?
• What is expected from the receiver: The whole premise is that the resources don't exist at the expected REST endpoints. The receivers need to treat the resources as business objects, and not FHIR resources. Yes, this breaks down several premises that make FHIR so compelling, but that is due to the decision that the facade in front of the existing system is not going to be sophisticated enough to provide the expected RESTful API .

Alexander Henket (Sep 21 2020 at 14:30):

I've filed FHIR#28587

Alexander Henket (Sep 21 2020 at 14:33):

How to generate Resource.id: If you have a unique identifier, is it not possible to "generate" Resource.id based of Resource.identifier?

Sometimes, but that is not guaranteed. Resource.id is really short and disallows most characters ([A-Za-z0-9\-\.]{1,64})

Alexander Henket (Sep 21 2020 at 14:35):

Whenever possible we will definitely encourage servers to generate a relevant Resource.id. We even development mappings for V3 ourselves for production use that go to great lengths to guarantee stable/unique Resource.ids. That being said: it turns out it is just not feasible in all cases

Vassil Peytchev (Sep 21 2020 at 14:37):

A SHA-1 or SHA-256 hash on a v3 II data type, or on the Resource.identifier content fits the restrictions of Resource.id - isn't that trivial to generate at run time?

Lloyd McKenzie (Sep 21 2020 at 14:42):

If you don't have a reliable id, FHIR REST isn't going to work well. That's just the reality of REST. It relies on resources have a fixed identity.

Alexander Henket (Sep 21 2020 at 14:43):

Sure, but the current requirements talk about requirements for "Resources". You are saying "if you are using them in a RESTful way". That is not the same

Alexander Henket (Sep 21 2020 at 14:44):

We fully acknowledge that without a Resource.id these resources will not play nice in a RESTful read/update/delete/patch, but... they work excellently in search and create.

Lloyd McKenzie (Sep 21 2020 at 14:46):

Something you search for that has its id change each time you search or something that you create that isn't at the same place you look isn't "excellent" in my opinion...

Alexander Henket (Sep 21 2020 at 14:46):

It has the very same Resource.identifier every time.

Lloyd McKenzie (Sep 21 2020 at 14:47):

Resource.identifier doesn't matter for most use

Alexander Henket (Sep 21 2020 at 14:48):

It has done so for the last 15 years of V3.

Lloyd McKenzie (Sep 21 2020 at 14:48):

If the 'id' isn't reliable, then I don't think you can say that search or create with REST are behaving 'well'.

Alexander Henket (Sep 21 2020 at 14:48):

Observation.id in V3 SHALL be unique, persistent and SHALL NOT be changed once assigned.

Lloyd McKenzie (Sep 21 2020 at 14:48):

V3 was doing documents and messaging

Alexander Henket (Sep 21 2020 at 14:48):

We do both.

Lloyd McKenzie (Sep 21 2020 at 14:49):

If you run a search multiple times and you get different ids each time, then that's essentially "changing" the id

Alexander Henket (Sep 21 2020 at 14:49):

Ok, so ad hoc generated ids are out of the question then.

Lloyd McKenzie (Sep 21 2020 at 14:49):

Systems doing RESTful search would not expect to have to de-dup using identifier

Alexander Henket (Sep 21 2020 at 14:50):

They have no choice. It is not possible to do so based on id

Lloyd McKenzie (Sep 21 2020 at 14:50):

Not out of the question, but be aware that they're going to cause issues, so all client systems need to be aware of the limitations.

Lloyd McKenzie (Sep 21 2020 at 14:50):

A standard RESTful client isn't going to have a great experience if it's not prepared.

Alexander Henket (Sep 21 2020 at 14:51):

A standard RESTful client is too limited for use then. In a MedicationOverview any system will include everything it knows, wherever it came from even if not from its own system. So: two systems will potentially respond with the same object. This object, say MedicationRequest, would have the same identifier, but a different id.

Alexander Henket (Sep 21 2020 at 14:52):

RESTful solves a technical communication issue on getting/sending objects, but it does not solve all business requirements

Alexander Henket (Sep 21 2020 at 14:54):

All I'm asking is: make FHIR useful outside RESTful use.

Lloyd McKenzie (Sep 21 2020 at 14:55):

You can't make REST do something it isn't designed to do. Certainly it's possible for a system to expose 30 AllergyIntolerance instances reflecting different views of a patient's penicillin allergy - that's one of the reasons we have List - to provide a filtered view. However List isn't going to work if the AllergyIntolerance ids aren't constant

Lloyd McKenzie (Sep 21 2020 at 14:56):

If you've got a 'stable' .identifier, your best solution is to have the infrastructure map that to a stable .id

Vassil Peytchev (Sep 21 2020 at 14:57):

All I'm asking is: make FHIR useful outside RESTful use.

Isn't that what operations are for? I don't understand how using RESTful search is outside RESTful use...

Alexander Henket (Sep 21 2020 at 14:57):

I'm not making REST do anything. I'm trying to use FHIR resources but the RESTful bias prohibits it.

If you've got a 'stable' .identifier, your best solution is to have the infrastructure map that to a stable .id

Would you like me to open a ticket for lifting the extreme limitations on Resource.id in that case? I would need 128 + 64 characters and no restrictions on characters

Alexander Henket (Sep 21 2020 at 14:58):

Operations don't help: they are about the SAME resources that REQUIRE id

Alexander Henket (Sep 21 2020 at 14:59):

And: if 90% of the network is perfectly fine with Resource.id and we could have a way to have RESTful systems and non-RESTful systems coexist and all it needs is to lift the SHALL on Resource.id.... why would I architect two different networks?

Alexander Henket (Sep 21 2020 at 15:00):

Its a bad value proposition.

Lloyd McKenzie (Sep 21 2020 at 15:00):

You can use FHIR resources for messaging and documents. But if you want to use them with REST, then you need to make them behave in a RESTful way.

Lloyd McKenzie (Sep 21 2020 at 15:01):

Resource.id syntax is normative and isn't going to change

Alexander Henket (Sep 21 2020 at 15:01):

We already established that search is not REST (I think). Nonetheless the FHIR spec requires Resource.id.

Lloyd McKenzie (Sep 21 2020 at 15:02):

If you want to take a messaging/document system and expose it via a RESTful interface, then you have to add additional logic and infrastructure that handles mapping to a reliable id

Vassil Peytchev (Sep 21 2020 at 15:02):

OK, simple question - why is sha256(Resource.identifier) not possible? What are the issues that prevent this from working?

Alexander Henket (Sep 21 2020 at 15:04):

Generated ids don't resolve to anything and are thus generated solely for the satisfaction of a RESTful requirement we don't have. It also weakens the experience for RESTful systems as they never know if the Resource.id actually means anything.

Vassil Peytchev (Sep 21 2020 at 15:15):

I am missing something. How will the following work:

I would need 128 + 64 characters and no restrictions on characters

yet sha256(string[128]+string[64]) will not?

Alexander Henket (Sep 21 2020 at 15:15):

OK, simple question - why is sha256(Resource.identifier) not possible? What are the issues that prevent this from working?

The most elementary answer there is: there is no XSLT2 solution for sha256 and lots of the mappings are written in XSLT2 without any expectation of Saxon-EE that may include sha256 support as extension. The second part is tat FHIR Resources slice the world differently than V3 does. Hence not all FHIR Resources will get a V3 id. The third part is that in number of cases certain V3 objects never needed an id hence there is no data to base the sha256 hash on.

Alexander Henket (Sep 21 2020 at 15:16):

So to summarize: we might be able to generate stable Resource.ids in more cases than currently possible, based on sha256 hashing. It will not be a 100% solution

Alexander Henket (Sep 21 2020 at 15:18):

But still: what you have is stable yet meaningless ids that would not serve any read/update/delete/patch purpose. Why? Because RESTful says so? But what if I don't need that functionality? What if all I need is search?

Lloyd McKenzie (Sep 21 2020 at 15:28):

If a RESTful client executes search, they're going to treat the 'id' as the reliable identifier - because in REST, that is always the primary and reliable identifier. If it searches again, it's going to presume everything with a new 'id' is a new resource instance - because that's what a new 'id' means in REST. 'id' isn't meaningless - it's essentially the primary key of the record. Trying to use REST without a reliable id is like trying to use SQL when you don't have a primary key. It's not going to go well...

Lloyd McKenzie (Sep 21 2020 at 15:30):

If you have limitations on what you can create reliable ids for, then at least produce them everywhere possible. "Most" is better than "None". If XSLT can't do what you need, then seriously consider going beyond XSLT. The importance of reliable ids is (or ought to be) higher than the importance of not changing the mapping technology.

Alexander Henket (Sep 21 2020 at 15:34):

I thought we landed on search not being RESTful? As said we will definitively strive for the best possible coverage of Resource.ids but as long as the requirement to have one stays and as long we can't fully comply, we will always be "FHIR-like".

Grahame Grieve (Sep 21 2020 at 15:39):

@Alexander Henket this discussion is getting tiresome. This stuff is normative. Use RESTful, RESTfullty. Or don't. SEARCH is part of the restful interface, and we wouldn't make it non-restful if we could.

Grahame Grieve (Sep 21 2020 at 15:40):

There's a serious misconception is what you write above:

Grahame Grieve (Sep 21 2020 at 15:40):

When a server generates a new uuid each time it serves a resource: does that not constitute a change in the resource.id and as such a direct violation of this part of the spec:

Once assigned by the server, the id is never changed.

Grahame Grieve (Sep 21 2020 at 15:41):

There answer is: yes, you cannot change the Resource.id. But the fact that you cannot change the Resource.id means that you have phrased the question wrongly.

Grahame Grieve (Sep 21 2020 at 15:41):

the correct phrasing is "Does FHIR require that a system can only have one resource for a given business concept", and the answer is: no, we never said that

Alexander Henket (Sep 21 2020 at 15:43):

That last bit means that as a receiver you have to assume discrete copies where the source system has 1 object.

I'm tired about it too, but that does not resolve the issue. So if you feel you don't want to discuss it anymore I'll have to talk to leadership here that in certain places we can never be FHIR compliant. I don't know where that ends.

Grahame Grieve (Sep 21 2020 at 15:44):

I don't know, from what you've said, where what you have done is not FHIR complaint, unless you insist on wording it wrongly.

Grahame Grieve (Sep 21 2020 at 15:44):

I agree about consequence. it's ugly, but not non-conformant

Alexander Henket (Sep 21 2020 at 15:54):

Don't know what to say. Please formally close FHIR#28587 and we'll see from there. I cannot break iron with my bare hands (Dutch expression).

I 'really' don't know what not solving this will do. In any case I'm really disappointed about all this.

Grahame Grieve (Sep 21 2020 at 15:55):

well, I can't close it personally, but the committee can not do anything but close it as out of scope because it's normative. The tiresome part for me is that after all your explanations, I still don't understand why this is a problem

Alexander Henket (Sep 21 2020 at 15:56):

I asked for in-person discussion, but I'm not sure what new thing I can say if all the above doesn't help.

Grahame Grieve (Sep 21 2020 at 15:59):

what you have established clearly is that

1. the standard requires you to populate Resource.id
2. you choose not to populate with an id that will be persistent (to avoid re-engineering your backends or an onerous solution on the facade)
3. under these conditions, the Resource.id brings no value

What you have not showed is that something about this is non-conformant

Gino Canessa (Sep 21 2020 at 16:08):

According to FHIR Resource.id must be unique, but it does not say there can only ever be a single id. So, if you generate a new id every time a resource is returned, that is conformant (but ugly*). The issue is that you must ensure they are unique - a client will assume it is the same resource if the same id is seen twice. Alexander, it seems this is what you would like changed (e.g., have this use some combination of identifiers). Without getting into a discussion about pros and cons, that concept is normative and it cannot realistically be changed (it would, for example, require every single FHIR implementer to agree - as one step).

*Years ago there was a large commercial DICOM implementation that did this (responded to every query with newly-generated ids). It worked, but everyone else had to do extra work to interact with their system (e.g., manually checking MRNs in addition to the IDs). This sounds like the same case: if you do the mapping to create a stable id, then nobody else has to. If you don't, everyone else will need to.

Alexander Henket (Sep 21 2020 at 16:08):

Ok.... that is a different unclarity than I thought you were on. To avoid adding large texts again: what we do in some cases (I keep stressing this), i.e. not giving back a Resource.id, is "ugly but conformant."

If that is shared opinion then I'll take it, because that is a fair statement. The notion that what we do is "non conformant" was fuelled by what I kept getting back here from people and the FHIR spec. If all that is a big misunderstanding, then I'm glad we cleared that up.

Grahame Grieve (Sep 21 2020 at 16:10):

I think that's the case: it's ugly but conformant.

It will invite people to think it's non-conformant if you keep using the langauge 'changing the id' since you can't. Instead, use the language 'creating a new resource which doesn't persist'

Alexander Henket (Sep 21 2020 at 16:15):

@Gino Canessa I'm not advocating generation of a new id every time. In cases where that is the only option, there would not be Resource.id at all to avoid people thinking it is that stable id that Resource.id is supposed to be. The net effect is that a RESTful client is not misled.

It will invite people to think it's non-conformant if you keep using the langauge 'changing the id' since you can't. Instead, use the language 'creating a new resource which doesn't persist'

Point taken.

Gino Canessa (Sep 21 2020 at 16:23):

That is fine - the result is functionally equivalent (where Resource.id is optional).

I will note though, that while the field is optional, many clients assume Resource.id will be present. Excluding it means that you (or someone on your staff) will likely have this discussion with many implementors, many times. You may want to document this behavior thoroughly =).

Lloyd McKenzie (Sep 21 2020 at 16:26):

And that's only where discussions are possible. For third-party tools, there won't be opportunity for discussion. They just won't work.

Alexander Henket (Sep 21 2020 at 17:52):

You may want to document this behavior thoroughly
And that's only where discussions are possible. For third-party tools, there won't be opportunity for discussion. They just won't work.

Evidence so far shows no issues other than deduplication questions. We will have to beef up there on our documentation no doubt, but that we can do. Thanks to you all for sticking with me on this.

Grahame Grieve (Sep 21 2020 at 18:02):

I think that it won't be so much a matter of 'they don't work' as 'their efficiency mechanisms will not work'. At least, a well designed client should continue to work, since resource ids still have meaning.

Alexander Henket (Sep 21 2020 at 19:01):

... and we can help those implementers on how to work with the data they will get on the network. Make them aware of Resource.id behavior, duplicate detection strategies and maybe more.

Alexander Henket (Sep 21 2020 at 19:04):

Even with Resource.id in place they might still find the 'same' MedicationRequest coming from two different sources (different Resource.id) with MedicationRequest.identifier as linking pin. This is because MedicationOverview includes what you have, even if that is obtained from a third party.

Grahame Grieve (Sep 21 2020 at 19:19):

right.

René Spronk (Sep 22 2020 at 06:59):

Most non-trivial implementations will require a de-duplication strategy. Is there any cross-industry guidance on such strategies?

Last updated: Apr 12 2022 at 19:14 UTC