FHIR Chat · Epic has IDs that are more than 64 characters · implementers

Stream: implementers

Topic: Epic has IDs that are more than 64 characters

Sean McIlvenna (Jul 30 2021 at 14:32):

I'm using HAPI libraries to communicate with an Epic FHIR servers and store the responding data from HAPI on a HAPI FHIR server. In a couple cases, it has responded with IDs that are 80+ characters long, which causes HAPI to throw an exception because FHIR has a requirement that IDs are no longer that 64 characters. Does anyone have any ideas on how to work around this? Does hapi-fhir-jpaserver-starter have any configurations or options to work around this issue?

John Silva (Jul 30 2021 at 15:00):

Just a thought --- what if you stored the Epic id as one of the identifier elements and then created your own (UUID?) id for the resource. Of course then you'd have to fix up all FHIR references though so that might not be ideal. Another, related thought it so have an algorithm that converts the long ids into 64 chars (or less) for the Id. You'd still have the resource referencing problem though. (One advantage of putting the Epic Id in the identifier array is that you have traceability back to the original Epic id.)

Josh Mandel (Jul 30 2021 at 16:24):

a couple cases, it has responded with IDs that are 80+ characters long, which causes HAPI to throw an exception because FHIR has a requirement that IDs are no longer that 64 characters. Does anyone have any ideas on how to work around this?

@Cooper Thompson we saw some related issues with IDs in Epic's SMART Scheduling Links implementation. Is this on your radar?

Josh Mandel (Jul 30 2021 at 16:28):

@Sean McIlvenna of course this is broken server behavior and should be fixed by the server.

That said: even in general with correctly behaving servers it can be hard to re-use an external server's IDs, because if you talk with >1 server, you might see an ID like Patient/123 being used differently across them.

One pattern you can use is to explicitly remap (server, resourceType, resourceId) tuples into your own IDs, and keep track of the mappings you create. A sub-pattern is to accomplish this mapping by feeding the tuple into a hash function, like

 $ echo -n "https://server.example.org/Patient/123" | openssl dgst -sha256
 6cebb4ef2ce81ed08cbc094ee54e5513ebc2463a02bf2d73024004e6b7572704

(This works, of course, even if the source IDs are too long.)

Sean McIlvenna (Jul 30 2021 at 16:57):

So, I think all roads lead to "change the id and update references to it"...
Does anyone know if there is any functionality in the HAPI libraries to aid in doing this? Or, do I need to create some custom logic to go through Java objects and find any instances of "Reference" class, and check if the reference needs to be updated with a new ID...

Cooper Thompson (Jul 30 2021 at 17:35):

@Josh Mandel The 64-character ID limit is separate than the issues we ran into with SMART Scheduling Links. We've been aware of the 64-charactacter issue for a while, but there really just isn't a practical, straightforward way for us to address it. I think we have a project on our backlog to see what we can do about it, but we've been kept busy by a bunch of regulatory related FHIR dev recently :grinning_face_with_smiling_eyes: .

Cooper Thompson (Jul 30 2021 at 17:37):

This has come up before in industry discussions, and I think there were Jira (or gforge) tickets asking if we could increase the ID length. If there were tickets, they were not accepted, or it might have just been chat.fhir discussion that ended up in a "no - not going to change the length in the FHIR spec" answer.

Josh Mandel (Jul 30 2021 at 18:29):

Fascinating. Does this come up as an issue in Inferno testing today?

Stephen MacVicar (Jul 30 2021 at 18:43):

Inferno uses the hl7 validator, so we'll catch that error if the validator catches it. Some initial testing is leading me to believe that the validator doesn't, though, which seems odd...

Josh Mandel (Jul 30 2021 at 18:56):

Huh @Stephen MacVicar you're quite right -- not only does the validator happily accept ids >64 chars, but it also accepts ids >1M chars (i.e., longer than any FHIR string is allowed to be -- the data type that id specializes).

Josh Mandel (Jul 30 2021 at 19:00):

I added comments on https://github.com/hapifhir/org.hl7.fhir.core/issues/513 which raises the broader issue.

David deRoode (Aug 02 2021 at 14:19):

aversable

Rick Geimer (Aug 02 2021 at 15:46):

I don't think it makes sense to keep an Epic to UUID mapping table. Epic is publishing these IDs as FHIR IDs, so they need to be compliant with the spec. They are welcome to create an identifier or extension if they want to share their 80+ character internal ID in addition to a legal FHIR ID.

Rick Geimer (Aug 02 2021 at 15:48):

I think once the validator is updated per Josh's comment we should see some movement because their data will start failing validation.

Rick Geimer (Aug 02 2021 at 15:54):

It's already failing when trying to load into HAPI.
@Vassil Peytchev @Michael Donnelly any thoughts here?

Josh Mandel (Aug 02 2021 at 15:54):

Both things are true: IDs should be spec compliant and systems consuming resources from >1 external source will need to remap (or use some full URI representation, not just bare IDs), else conflicts arise.

Rick Geimer (Aug 02 2021 at 15:57):

Agreed Josh. In our initial use case we are assuming a single tenent thus no ID conflicts, but if/when we move to multi-tenent we will need to deal with mappings.

Cooper Thompson (Aug 02 2021 at 16:59):

The other options is that we could update the spec to allow longer IDs. I know of at least a few non-Epic system that have also run into issues with the 64-character limit.

Cooper Thompson (Aug 02 2021 at 17:01):

And to pre-empt any comments about that being normative content, at some point, FHIR is going to have to sort out how to change normative content. It wouldn't apply retroactively to R4 of course, but there is an increasingly large list of normative content that needs updating.

Cooper Thompson (Aug 02 2021 at 17:01):

Also, for reference, some previous threads about ID length:

• https://chat.fhir.org/#narrow/stream/179166-implementers/topic/Resource.2Eid.20Data.20Type
• https://chat.fhir.org/#narrow/stream/179166-implementers/topic/Confused.20about.20URN.20fullUrl's.20and.20resource.2Eid's

Lloyd McKenzie (Aug 02 2021 at 17:05):

At this point, the id type is normative. Changing it to allow larger ids would be a breaking change and would be a very challenging thing to do at this point in the core spec's life cycle.

Cooper Thompson (Aug 02 2021 at 17:15):

At some point, we're going to have to figure out how to make updates to normative content when the spec isn't meeting the needs of the industry. ID length maybe isn't on the top of the list, but there are other areas both in infrastructure and PA where I know we'll need to look at normative updates.

David Pyke (Aug 02 2021 at 17:16):

The problem is that it's been 64chars for a long time and changing to a longer limit would break a lot of systems. We need to create not only an update but a transition plan for systems that rely on that 64char limit

Sean McIlvenna (Aug 02 2021 at 17:42):

@Lloyd McKenzie , I understand your point about breaking changes... But, I don't think loosening the rule is the same as tightening the rule when it comes to breaking changes. To implementers that are following the rule, it doesn't break anything for them.

Sean McIlvenna (Aug 02 2021 at 17:44):

Of course, if they consume FHIR resources, they would need to account for longer IDs, but my main concern of "breaking changes" would be for those that create resources and distribute the resources to others.

Gino Canessa (Aug 02 2021 at 17:50):

I'm going to have to agree that changing the max length is definitely a breaking change. If a database currently has a field with a 64-character limit (since that's what it is defined to be), this change will be incompatible. Same thing for any system doing validation outside of the official FHIR validator (e.g., checking field lengths, regex expressions, etc.).

Note that I'm not arguing for or against right now, just noting that changing it will potentially break any existing system.

Lloyd McKenzie (Aug 02 2021 at 17:52):

Breaking change is any change that could cause systems to fail when consuming conformant data. We've defined the types of changes that systems are expected to be able to handle between versions (e.g. new elements, new codes, new resources, new reference types). Relaxing invariants wasn't one of those. I'm not saying it's impossible, just that it will be very hard - so we'd need to be convinced that workarounds weren't possible or that the pain of workarounds to the community would be higher than the pain of making the change.

Cooper Thompson (Aug 02 2021 at 17:57):

One variable you can take into consideration, is that any system that works with Epic already can support longer FHIR IDs :joy:

Eric Prud'hommeaux (Aug 02 2021 at 17:57):

Sometimes the best way to make a change like this is through grass-roots civil disobedience. If you can invent a label for the new variant and get a bunch of vendors to use your test suite, you can reduce the FUD for early adopters. They will still have to go through their tool chain with a fine-toothed comb to see if some field in a database or a fixed memory allocation in a misc script somewhere will break their data flow, but that's stuff that would have to happen anyways if v5 were to suddenly announce longer IDs.

David Pyke (Aug 02 2021 at 17:58):

That's a great way to lose your rights to call yourself FHIR Conformant.

Eric Prud'hommeaux (Aug 02 2021 at 18:00):

the data is not FHIR but the tools still are (unless everyone's expected to reject IDs that are too long)

Eric Prud'hommeaux (Aug 02 2021 at 18:01):

and if that goes well, you can make that case for changing the FHIR definition to adopt the new industry practice

Eric Prud'hommeaux (Aug 02 2021 at 18:02):

if it doesn't, you probably weren't going to convince the world to change anyways

Eric Prud'hommeaux (Aug 02 2021 at 18:05):

if you add a must-understand extension, you also provide a nice way to weed out the FHIR+longIDs from the FHIR instance data.

Chris Moesel (Aug 02 2021 at 18:15):

We just need to create a FHIR bit.ly service that can create short ids that expand into longer ids. ;-)

Daniel Venton (Aug 02 2021 at 18:28):

A hexadecimal representation of a guid, takes 32 characters to assign a unique value to every grain of sand on earth and have values left over. If you use something like base64 it takes even less. How many unique values does one EHR need for a given resource? How you cross 64 characters is by doing things like compositing all the attribute values into making a PK, "Epic-Condition-Site-Client-Provider-ConditionCode-Patient-number" rather than assigning a guid as a primary key and be done with it.

Cooper Thompson (Aug 02 2021 at 18:32):

The issue (which I've seen non-Epic systems run in to as well), is that sometimes it is very useful to have reversible IDs rather than mapped IDs. That means you need to encode enough data in the ID such that when decoded, it has enough info for you to resolve it to the data you are addressing.

Daniel Venton (Aug 02 2021 at 19:55):

If you have a reversible id, one that is calculated and can be decoded. Take my example above. I used the condition code in my id value let's just assume that the condition was 12345678. Turns out I made a typo and the actual condition code was supposed to be 12345679. Now that code attribute has been fixed to 12345679 but the decode of the id still has 12345678 in it. What is the correct code value? Since the id value of a given resource can never change, then in the case of generated id's, the components that comprise the id shall never change. Otherwise you end up with conflicts. My opinion: the id attribute should only ever have 1 definition, "the PK of the resource".

Cooper Thompson (Aug 02 2021 at 20:44):

I don't think we need to get in to the details of how the "encoded" FHIR IDs get generated. I'll make the claim that reasonable and competent developers can do it in a way that works, and is not afflicted by the issue you describe, though not necessarily in a way that fits into 64 characters.

Cooper Thompson (Aug 02 2021 at 20:46):

Though the example of the composite FHIR server might help. Say you have a FHIR facade that is the front for two backend FHIR servers. The facade might generate a FHIR ID their own outward-facing FHIR ID that is just the composite of the namespace and the FHIR ID of a backend FHIR server.

John Moehrke (Aug 02 2021 at 20:51):

See a presentation Grahame gave at DevDays November 2020 on this topic -- https://youtu.be/stFGtk-YKPQ

John Moehrke (Aug 02 2021 at 20:52):

useful conclusion he gives on the 64 character limit... essentially... no matter what limit we give, someone will come up with a need for more. Thus we stick with 64 character...

John Moehrke (Aug 02 2021 at 20:53):

Thus. It is those that might produce issues with more than 64 characters that must come up with a solution to their own problem. The limit is normative.

Sean McIlvenna (Aug 02 2021 at 20:56):

makes sense. so, I'm fine with that normative limit, generally speaking...
one thing I'd like to know is if there is standard guidance on how to handle IDs when taking resources from one FHIR server and putting them on another?

John Moehrke (Aug 02 2021 at 21:09):

that is a task left to a systems design engineer... get a good one.

Cooper Thompson (Aug 02 2021 at 21:16):

I'm fine if the industry settles on a 64-character limit. But I don't accept the "it is normative now and forever" argument. That comes up more and more in FHIR, and if FHIR can't evolve, then it will become obsolete. I am not saying the 64-character limit is an essential evolution point, but if something is not right in the spec, we need a path to get it updated, normative or not.

Cooper Thompson (Aug 02 2021 at 21:25):

John Moehrke said:

that is a task left to a systems design engineer... get a good one.

"Good" is relative in engineering, as all engineering disciplines are about evaluating tradeoffs. One the one hand you have strict spec compliance, and on the other hand you have lower system cost, faster time to market for the API (so patients can get their data sooner), and in some cases, significantly simpler system design (for example, the facade server in front of other FHIR servers), which leads to lower system maintenance.

John Silva (Aug 02 2021 at 21:26):

Following ...
The notion of trying to 'encode info' in an id seems like a bad idea. [This might be the 'root cause' of Epic's long FHIR IDs -- remember, behind the facade is a MUMP string-based 'database' system.] I remember reading articles about NOT doing that for coding systems, and I think the same applies for FHIR IDs.

The DB world frequently deals with 'schema changes', which this is in essence. The process is not cheap but it's typically a migration process that includes not only the schema migration but also the data migration, tooling, client code. Definitely not easy but can be done if there is value in doing it.

(not saying that is the case here but for system integrators that do not have the ability to change what Epic (or other system) gives them they need to work around this to get the job done, even if the spec says "not supported".)

Cooper Thompson (Aug 02 2021 at 21:27):

John Silva said:

Following ...
The notion of trying to 'encode info' in an id seems like a bad idea.

I feel like the notion of composite IDs is fairly standard. https://en.wikipedia.org/wiki/Composite_key

John Silva (Aug 02 2021 at 21:34):

Ok, yes, in DBs they have primary keys that are composites of multiple columns. It just seems like exposing that composite key as an external identifier is problematic. What if the composite key is 5 columns -- the length becomes unmanageable -- and fragile if the DB owner decides to change the schema design.

Tejvir Saggu (Aug 02 2021 at 21:44):

Hi, Need help on the authorization for backend systems. Getting client invalid error...
What is the resolve on that?

Michele Mottini (Aug 02 2021 at 21:46):

@Tejvir Saggu please create a new topic. Also, we'll need some more details. Also also: probably better to ask to whoever support the server you are trying to connect to

Michele Mottini (Aug 02 2021 at 21:49):

It just seems like exposing that composite key as an external identifier is problematic. What if the composite key is 5 columns -- the length becomes unmanageable -- and fragile if the DB owner decides to change the schema design.

Being Epic maybe maybe they actually know what they are doing and they have good reason to do things that way

Tejvir Saggu (Aug 02 2021 at 21:49):

@Michele Mottini Thanks for the response. Will do that

Cooper Thompson (Aug 02 2021 at 21:50):

I'm half arguing on behalf of Epic, but half on behalf of other systems I know have the same problem. Epic may actually adjust our approach in the future. I know some other systems that will have a harder time.

John Silva (Aug 02 2021 at 22:14):

@Cooper Thompson - if Epic has this "problem" of needing long IDs, and as you're suggesting, other systems do too, then maybe there's something to be learned from this. It seems like the internals of whatever system is providing a FHIR Facade (I assume that's what Epic and like-minded systems are doing) have this problem then there seems like there must be a way to provide an intermediary mechanism to implement a mechanism that creates FHIR IDs that fit the 64 char limit and are not dependent on composite keys, or at least not with the IDs they expose to FHIR clients.

Cooper Thompson (Aug 03 2021 at 14:19):

@John Silva if you add an intermediary, then you are trading overall system complexity for the 64-character length, since you are now adding a mapping table to the integration. In some cases (like Epic), we could implement that intermediary mapping internally. However other systems I know of would need an external mapping layer, which means you need a database for the mapping table, DBA to maintain that database, network engineers to maintain the two new network connects (public->mapping tier and mapping tier -> backend server), and a web server with enough code to execute the mapping and proxy the request, and then another admin to maintain and patch that web server.

Many of those features could be incorporated into a product (API Gateway) of course, similar to the bit.ly pseudo-suggestion. But products aren't free, so that cost always lands somewhere.

John Silva (Aug 03 2021 at 14:32):

@Cooper Thompson - yes, there would need to be an intermediary, which brings extra complexity and cost. However, when an integrator needs to get the job done and does NOT have the ability to change the backend system (e.g. Epic or other that isn't following the 64-char limit) what other alternatives does one have? I think the better question, which I think was already raised, is WHY would a system choose to be non-conformant to start with?

Cooper Thompson (Aug 03 2021 at 14:48):

@John Silva The vast majority of our IDs are shorter than 64-characters - only a few IDs in a few resource types get too long. We may look at revisiting our ID strategy at some point, but now we have to deal with backwards compatibility, so the tradeoff is "breaking backwards compatibility" vs. "be strictly compliant with the spec". For us, making sure FHIR integrations continue to work in real world healthcare settings is more important than strict spec compliance.

For other systems I know of, their choice is driven by a desire to keep system complexity low. Healthcare IT is expensive, and keeping system complexity low keeps costs low. So for them, the trade off is just cost vs. "strict spec compliance". Even though I'm a standards nerd, I respect that engineering choice. Being too much of a purist can be very impractical (and expensive).

Paul Church (Aug 03 2021 at 14:57):

Is a FHIR integration actually "working" if it produces data that other conformant systems can't consume? What is the point of a FHIR integration?

Michele Mottini (Aug 03 2021 at 15:48):

We have been happily consuming data from Epic for over five years....

John Silva (Aug 03 2021 at 16:09):

Any FHIR client, especially those used by integrators, depend on the spec. If a system doesn't comply then they have to 'special case' around that; assuming they have the ability to do that. Having 'outlier systems' that do not conform makes the work of integrators more complex and problematic. [Does Epic's FHIR API docs specify these 'resource with long/non-standard length IDs', I haven't seen this on the API pages.].

Michele Mottini (Aug 03 2021 at 16:20):

Agree in principle, but of all the possible issues consuming FHIR data ID length is really not that big of a deal

John Moehrke (Aug 03 2021 at 16:31):

buffer overflows often don't cause trouble... until they do.

Cooper Thompson (Aug 03 2021 at 16:46):

Paul Church said:

Is a FHIR integration actually "working" if it produces data that other conformant systems can't consume? What is the point of a FHIR integration?

In my mind, a FHIR integration is "working" if the data provided by the integration is being used to improve patient health, whether that is a FHIR app used by clinical staff to improve the care they provide, or by a patient access their own data for whatever purpose they choose.

Cooper Thompson (Aug 03 2021 at 16:47):

Also, "FHIR Conformant" is a harmful myth IMO. We surveyed a bunch of systems that implemented a "FHIR Conformant" scheduling API, but none of them were remotely interoperable.

Sean McIlvenna (Aug 03 2021 at 16:57):

that is a task left to a systems design engineer... get a good one.

LOL @John Moehrke, I could design a solution. But, before I do that, I was hoping there would be some guidance for the general community and/or FHIR spec itself.
Seems like this is not an uncommon issue, but should have common guidance.

John Moehrke (Aug 03 2021 at 17:14):

I am not saying it should not be asked about. I am saying that it is also right to be left to systems design.

John Moehrke (Aug 03 2021 at 17:15):

an interop standard needs to focus on interop. it might include guidance on systems design, but systems design should be allowed

Paul Church (Aug 03 2021 at 17:23):

So a FHIR integration is "working" if it successfully interoperates with other Epic systems or apps that were designed to handle the quirks of Epic data. Sounds like the Judy Faulkner definition of interoperability.

Lloyd McKenzie (Aug 03 2021 at 17:29):

Conformant and interoperable are different things. Ideally conformance aids in interoperability, but it certainly doesn't guarantee it. That said, conformance is what tends to be evaluated by regulation and certification. Any system that sends ids over the limit is at risk in those processes - and also carries the risk of not interoperating with systems that depend on the length limit being respected.

Lloyd McKenzie (Aug 03 2021 at 17:30):

(@Paul Church, let's avoid the use of names and keep the conversation technical.)

Cooper Thompson (Aug 03 2021 at 18:14):

@Paul Church - Epic does consider this a bug, but we have a lot of FHIR dev on our plate (ONC stuff, but also a bunch of other countries have their own FHIR regs now), so we just can't prioritize a fix right now. I've tried to make it clear that part of why I'm pushing back here is because I'm aware of other (non-Epic) systems that have the same problem and they don't have an easy solution. This is not just an Epic issue.

Daniel Venton (Aug 03 2021 at 19:05):

If it's ok that the length can be exceeded, then I suppose it's ok for the id to have a % character in it, if it's ok for a % char, then I suppose it's ok to not do X, if that's ok then it's probably ok to do Y...... Eventually you get interfaces that don't interface.

John Silva (Aug 03 2021 at 20:51):

"aware of other (non-Epic) systems that have the same problem" ...

Are these also facades around EHR 'backends' (w/o naming them)? It seems like a facade should be able to deal with this because it's already the intermediary between the backend and the "FHIR server face" it's providing. As folks more knowledgable than me pointed out, a 64-char id can represent a huge number but if Ids are generated by concatenating strings in the 'backend' then it's susceptible to this problem.

Oh well, it is what it is ...

Last updated: Apr 12 2022 at 19:14 UTC