Stream: hapi
Topic: hapi fhir protect agains race conditions
Pengyu Wang (May 24 2021 at 22:10):
Hi, we're running into an issue with HAPI FHIR where multiple resources are created with the same identifier, even though we're doing a condition create with the header If-None-Exist. We believe the issue is because of a race condition where we're sending multiple create requests simultaneously, so HAPI is not able to reliably synchronously find the existing resource for a given identifier, and thus ends up creating multiple duplicates.
We've made changes to the code so it's sending the requests synchronously to prevent this particular issue. But the root cause of the issue is still not solved because there could potentially be multiple processes writing to the HAPI server and we can't guarantee the order of operations.
What is the best practice for protecting against race conditions? Is it possible to manually enforce uniqueness constraints at the db level in HAPI?
Joel Schneider (May 24 2021 at 23:33):
The hapi-fhir documentation describes several mechanisms for control of Search Result Caching.
https://hapifhir.io/hapi-fhir/docs/server_jpa/configuration.html
I'd suggest trying the options mentioned there, such as using myDaoConfig.setReuseCachedSearchResultsForMillis(null); or setting a Cache-Control: no-cache HTTP request header.
Pengyu Wang (May 25 2021 at 04:45):
Joel Schneider said:
The hapi-fhir documentation describes several mechanisms for control of Search Result Caching.
https://hapifhir.io/hapi-fhir/docs/server_jpa/configuration.html
I'd suggest trying the options mentioned there, such as usingmyDaoConfig.setReuseCachedSearchResultsForMillis(null);or setting aCache-Control: no-cacheHTTP request header.
Thank you for the reply. But I think you misunderstood the problem. The issue was not caused by hapi-fhir caching the search results. I'm not manually checking the existence of a resource before creating it.
Craig McClendon (May 25 2021 at 13:12):
Not speaking for HAPI, but from my own general experience - this is a hard problem to solve on the server side without severe performance implications such as serializing all persistence operations.
It's not really any different than if your client was directly searching the database for existing records on multiple threads - each thread could find no record existing and create it and since the Identifier "attribute" doesn't have a unique constraint it would not be caught.
It is the client that has the real context of what constitutes a duplicate in this case.
I have a similar client/problem and what I do is use a striped locking mechanism on the identifiers I am using for uniqueness to ensure no two threads process the same resource at the same time without preventing other resources from being processed in parallel.
Craig McClendon (May 25 2021 at 13:49):
... I suppose the server could use a similar striped locking mechanism on the contents of the 'If-None-Exist' header when it is present, but there would still be holes in it such as if a different client creates the resource without the header. It would tighten the window of opportunity, but not eliminate the possibility.
Jens Villadsen (May 25 2021 at 23:39):
If you wan't to guard the server against such race conditions there's a feature in HAPI using SearchParameters (yes, search parameters - keep reading). What you can do is that you can create a SearchParameter instance on the server that uses the magic of https://github.com/hapifhir/hapi-fhir/blob/2ba100576263f07e8733b733d90ca8046ae33da4/hapi-fhir-base/src/main/java/ca/uhn/fhir/util/HapiExtensions.java#L102 that basically creates a unique index in the database on whatever datafield you would like. With this enabled, the database will actually guard you against such duplicates. Using that, you are on your own, but it works with a minimal effort. It won't however be portable onto other FHIR solutions but it will work for any HAPI FHIR-based solutions that runs on top of the HAPI FHIR JPA stack
Pengyu Wang (Jun 19 2021 at 05:27):
@Jens Villadsen Thank you for the answer. I found this link (https://www.smilecdr.com/our-blog/custom-search-parameters-in-hapi-fhir) on how to create a custom search parameter. Am I understanding this correctly that I need to:
- Create an extension with the
EXT_SP_UNIQUEurl - Create a SearchParameter and add the extension to it
- Do the above when initializing the server
I'm using the hapi fhir jpaserver starter and I added the following to JpaRestfulServer.java in the initialize method:
@Override
protected void initialize() throws ServletException {
super.initialize();
// Add your own customization here
Extension ext = new Extension();
ext.setUrl(HapiExtensions.EXT_SP_UNIQUE);
SearchParameter customSearchParam = new SearchParameter();
customSearchParam
.addBase("Appointment")
.setCode("uniqueIdentifier")
.setType(Enumerations.SearchParamType.TOKEN)
.setTitle("Unique Identifier")
.setStatus(Enumerations.PublicationStatus.ACTIVE)
.addExtension(ext);
IGenericClient client = getFhirContext().newRestfulGenericClient("http://localhost:3000");
client.create().resource(customSearchParam).execute();
}
But when I start the server I got the error
2021-06-18 22:24:23.716 [main] ERROR o.a.c.c.C.[Tomcat].[localhost].[/] [DirectJDKLog.java:175] Servlet.init() for servlet [jpaRestfulServer] threw exception
ca.uhn.fhir.rest.server.exceptions.ResourceNotFoundException: HTTP 404
What am I missing here?
Jens Villadsen (Jun 19 2021 at 06:44):
Looks like you are starting a client within your server before the service is actually started
Jens Villadsen (Jun 19 2021 at 10:41):
instantiate your client outside the bootstrappning of your server and you should be fine
Pengyu Wang (Jul 12 2021 at 22:01):
@Jens Villadsen i created a custom search parameter on the patient identifer and it seems to be working as intended when i bulk inserted multiple patients into a single partition (I have partitioning enabled on the server). However, the uniqueness constraint seemed to be applied across partitions as well. When I tried to create a patient resource in a different partition but with the same identifier, the server doesn't allow me to do it and I got the error Can not create resource of type Patient as it would create a duplicate unique index matching query. Is there a way to make this constraint only apply to resources within a partition? Meaning I should be able to create Patient with identifier 123 in multiple partitions, but in each partition there should only be one such patient? Detaild description of this issue is posted here
Jens Villadsen (Jul 13 2021 at 08:44):
I don't believe that is a feature of HAPI. You could proceed the hack'ish way and make a compound identifier consisting of the partition Id and the patient identifier.
Joe Atterberry (Jan 07 2022 at 21:18):
Thanks @Jens Villadsen and @Pengyu Wang I was able to successfully create a custom search parameter and it does guard against race conditions as was noted, however when I try to retrieve any resource using an http GET of the form http://localhost:3111/my-fhir/v1/Patient?identifier=1555667 zero records are returned where one was returned before introducing the search parameter.
I use the following:
{
"resourceType": "SearchParameter",
"meta": {
"lastUpdated": "2021-05-02T10:43:02.162-04:00",
"tag": [ {
"system": "https://my.health.com/x/Onc1C",
"code": "34a800fa-bba9-4cb9-77ed-0aa4d0eecef2",
"display": "FHIR Implementation Team"
} ]
},
"extension": [
{
"url": "http://hapifhir.io/fhir/StructureDefinition/sp-unique",
"valueBoolean": true
}
],
"url": "http://health.com/fhir/SearchParameter/uniq-patient-identifier",
"name": "unique-patient-identifier",
"status": "active",
"description": "A unique patient identifier",
"code": "unique-patient-identifier",
"base": [
"Patient"
],
"type": "token",
"expression": "Patient",
"component": [
{
"definition": "http://hl7.org/fhir/SearchParameter/patient-identifier",
"expression": "Patient"
}
]
}
In order to retrieve any resource I have to include the system with the identifier. So the GET looks like http://localhost:3111/my-fhir/v1/Patient?identifier=urn:uuid:2.16.840.1.113883.4.349|1555667
Is there anyway to change the search parameter so that the resource can be retrieved as it was before?
Thanks.
Pengyu Wang (Jan 10 2022 at 19:37):
@Joe Atterberry is it safe to assume that the patients only have one identifier therefore you don't care about the system? if so have you tried adding an expression to the search parameter? try adding "expression":"Patient.identifier.value" to the unique patient sp
Joe Atterberry (Jan 13 2022 at 23:24):
@Jens Villadsen @Pengyu Wang Yes, that is correct. I tried using what you suggested. I substituted "expression":"Patient with "expression":Patient.identifer.value" and I still need to include the system in order to retrieve the record. I did check the HFJ_IDX_CMP_STRING_UNIQ table where the expressions are kept and IDX_STRING column includes the system.
Patient?identifier=urn%3Auuid%3A2.16.840.1.113883.4.349%7C1555667
Last updated: Apr 12 2022 at 19:14 UTC
