Stream: hapi
Topic: create new search parameter for HAPI
Vibin_chander (May 10 2020 at 14:04):
Hi All,
This might sound as a dumb question.
I need to add a incoming message's date as a Extension to all of my resources. If i've added the extension with the date value coming in the incoming message, then How can I make this date as a new "date" search parameter?
Vibin_chander (May 10 2020 at 14:07):
The ultimate aim of this question is, currently we are quering the HAPI repository using "_lastUpdatedDate" we want to get rid of querying the last updated date generated by the HAPI. And use the date of incoming message. But this date should be searchable in all the resources like _lastUpdatedDate
Vibin_chander (May 10 2020 at 14:15):
Im asking this question based on the Stackoverflow comment i saw
Vibin_chander (May 10 2020 at 14:15):
René Spronk (May 10 2020 at 15:19):
An alternative may be to create Provenance resources, the 'date' thereof can be searched by default. Assuming that each message causes the creation of multiple FHIR resources, one could create one single Provenance resource and associate it with all of the newly created resources. When extracting data from messages the use of Provenance will be useful anyway for the tracking of when which things were imported/extracted. yes, a bit of overhead when compared to an extension on each and every resource, but flexibility has its price.
Igor Sirkovich (May 10 2020 at 15:42):
@Vibin_chander If you do need this new extension and an ability to search by it, create and post a new SearchParameter (after posting this to your server, HAPI will take a few minutes to index the data before you can use this new search parameter). Have a look at the following example: https://www.hl7.org/fhir/searchparameter-example-extension.json.html
Vibin_chander (May 11 2020 at 03:27):
@René Spronk and @Igor Sirkovich Thanks for you inputs. Let me try the Search Parameter once
Vibin_chander (May 11 2020 at 05:26):
Im trying the Search Parameter but not sure how it works.
I'm trying like this , in the search parameter resource I've made like this
"name": "newurl",
"type": "reference",
"expression": "Coverage.extension('http://myUrlName"),
"xpath": "f:Coverage/f:extension[@url='http://myUrlName']",
"target": [
"Coverage"
]
In the Coverage Resource Im defining my extension like this
"extension" : [
{
"url" : "http://myUrlName",
"valueCodeableConcept" : {
"coding" : [
{
"system" : "http://myUrlName",
"code" : "HL7V2-SOURCE-DATE",
"display" : "2020-04-30T15:19:11"
}
]
}
}
]
Which means I can search the Coverage Resource by using "newurl" as search parameter right?
But searching Coverage?newurl=http://myUrlName does not return anything. Let me know if y understanding is wrong here
Igor Sirkovich (May 11 2020 at 05:50):
Try adding "code": "myextension" in the SearchParameter and replacing "target" with "base". In the Coverage extension, "system" should point to the CodeSystem URL, e.g. "system": "http://www.test.com/MyCodeSystem". Then you should be able to search Coverage?myextension=http://www.test.com/MyCodeSystem|HL7V2-SOURCE-DATE. Of course you can use your own values instead of "myextension" and "http://www.test.com/MyCodeSystem".
Igor Sirkovich (May 11 2020 at 05:53):
Also replace "type": "reference" with "type": "token" since your extension is of CodeableConcept type
Vibin_chander (May 11 2020 at 05:57):
replacing "target" with "base" meaning both should be of same resource name? or only either of the one should exist?
Vibin_chander (May 11 2020 at 06:04):
Im getting an error saying this Unknown search parameter \"myextension\"
Vibin_chander (May 11 2020 at 06:11):
Here is the whole JSON im using for search parameter
"resourceType": "SearchParameter",
"id": "example-extension",
"url": "http://hl7.org/fhir/SearchParameter/example-extension",
"name": "newurl",
"status": "draft",
"experimental": true,
"publisher": "Health Level Seven International (FHIR Infrastructure)",
"contact": [
{
"telecom": [
{
"system": "url",
"value": "http://hl7.org/fhir"
}
]
}
],
"description": "Search by patient for a MSH.7 agreement, which is stored in a any resource",
"code": "myextension",
"base": [
"Coverage"
],
"type": "token",
"expression": "Coverage.extension('http://www.caringly.net/incoming-message-date')",
"xpath": "f:Coverage/f:extension[@url='http://www.caringly.net/incoming-message-date']",
"xpathUsage": "normal",
"target": [
"Coverage"
]
}```
Vibin_chander (May 11 2020 at 06:14):
I was searching by Coverage?myextension=http://www.caringly.net/incoming-message-date
Igor Sirkovich (May 11 2020 at 06:19):
You don't need target (it's only needed for a SeacrhParameter of a reference type. Is http://www.caringly.net/incoming-message-date the URL of your extension? Is this extension defined as a StructureDefinition on your HAPI server? What's the CodeSystem URL you use in the actual Coverage instance extension? In the search you need to us a token of CodeSystem and code separated by "|", e.g. Coverage?myextension=http://www.test.com/MyCodeSystem|HL7V2-SOURCE-DATE
Vibin_chander (May 11 2020 at 06:22):
Im using that URL in the extension of the coverage resource
"extension" : [
{
"url" : "http://www.caringly.net/incoming-message-date",
"valueCodeableConcept" : {
"coding" : [
{
"system" : "http://www.caringly.net/incoming-message-date",
"code" : "HL7V2-SOURCE-DATE",
"display" : "2020-04-30T15:19:11"
}
]
}
}
]
I will remove the target and try once
Vibin_chander (May 11 2020 at 06:23):
Is it Must to define the Extension in structure definition resource ?
Igor Sirkovich (May 11 2020 at 06:25):
URL of the extension should be different from the CodeSystem URL. I would recommend to play with the public HAPI server: hapi.fhir.org. Try to create the SearchParameter resource, StructureDefinition resource for the extension and an instance of a Coverage resource using this extension. Then try searching for this Coverage resource using the new SearchParameter.
Vibin_chander (May 11 2020 at 06:26):
ok sure
dsh (May 11 2020 at 09:40):
@Igor Sirkovich could you please share an example of creating StructureDefinition resource for Extension ?
Igor Sirkovich (May 11 2020 at 14:31):
Have a look at https://www.hl7.org/fhir/extensibility-registry.html. There are tons of extensions of various types.
dsh (May 11 2020 at 17:45):
@Igor Sirkovich thanks. That helped a lot. I do have a working SD, SP and search.
Now I am trying to create SearchParameter
with base
= Resource
so that it is available on all resources. But when I do that Patient
resource is no longer searchable by SearchParameter
{
"resourceType": "SearchParameter",
"id": "10025450",
"meta": {
"versionId": "1",
"lastUpdated": "2020-05-11T16:48:00.413+00:00",
"source": "#wWpWb92nquQITjqi"
},
"url": "http://www.caringly.tech/fhir/SearchParameter/SearchByMsgDateParameter",
"version": "1.0",
"name": "SearchByMsgDateParameter",
"status": "active",
"description": "Search by patient for a MSH.7 agreement, which is stored in a any resource",
"code": "SearchByMsgDate",
"base": [
"Resource"
],
"type": "date",
"expression": "*.extension('http://www.caringly.tech/SearchByMsgDate')",
"xpathUsage": "normal"
}
dsh (May 11 2020 at 17:45):
any ideas ?
dsh (May 11 2020 at 18:40):
I have even tried the following
{
"resourceType": "SearchParameter",
"id": "10025450",
"meta": {
"versionId": "1",
"lastUpdated": "2020-05-11T16:48:00.413+00:00",
"source": "#wWpWb92nquQITjqi"
},
"url": "http://www.caringly.tech/fhir/SearchParameter/SearchByMsgDateParameter",
"version": "1.0",
"name": "SearchByMsgDateParameter",
"status": "active",
"description": "Search by patient for a MSH.7 agreement, which is stored in a any resource",
"code": "SearchByMsgDate",
"base": [
"Resource"
],
"type": "date",
"expression": "extension('http://www.caringly.tech/SearchByMsgDate')",
"xpath": "f:*/f:extension('http://www.caringly.tech/SearchByMsgDate')",
"xpathUsage": "normal"
}
to no avail. What am I missing ?
Igor Sirkovich (May 11 2020 at 20:15):
Did you create a StructureDefinition for the extension. Can you show this StructureDefinition?
dsh (May 11 2020 at 20:16):
yes, just a sec
dsh (May 11 2020 at 20:17):
here it is https://gist.github.com/a9e98fc8b64e0fff3c1a93d46b342345 @Igor Sirkovich
dsh (May 11 2020 at 20:21):
I have tried this SearchParameter def
{
"resourceType": "SearchParameter",
"id": "10025450",
"meta": {
"versionId": "1",
"lastUpdated": "2020-05-11T16:48:00.413+00:00",
"source": "#wWpWb92nquQITjqi"
},
"url": "http://www.caringly.tech/fhir/SearchParameter/SearchByMsgDateParameter",
"version": "1.0",
"name": "SearchByMsgDateParameter",
"status": "active",
"description": "Search by patient for a MSH.7 agreement, which is stored in a any resource",
"code": "SearchByMsgDate",
"base": [
"Resource"
],
"type": "date",
"expression": "extension('http://www.caringly.tech/SearchByMsgDate')",
"xpath": "f:*/f:extension('http://www.caringly.tech/SearchByMsgDate')",
"xpathUsage": "normal",
"comparator": ["eq", "ge", "le", "gt", "lt", "ne"]
}
but when I query Patient I get
{
"resourceType": "OperationOutcome",
"text": {
"status": "generated",
"div": "<div xmlns=\"http://www.w3.org/1999/xhtml\"><h1>Operation Outcome</h1><table border=\"0\"><tr><td style=\"font-weight: bold;\">ERROR</td><td>[]</td><td><pre>Unknown search parameter "SearchByMsgDate". Value search parameters for this search are: [_id, _language, active, address, address-city, address-country, address-postalcode, address-state, address-use, birthdate, death-date, deceased, email, family, gender, general-practitioner, given, identifier, language, link, name, organization, phone, phonetic, telecom]</pre></td>\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t</tr>\n\t\t</table>\n\t</div>"
},
"issue": [
{
"severity": "error",
"code": "processing",
"diagnostics": "Unknown search parameter \"SearchByMsgDate\". Value search parameters for this search are: [_id, _language, active, address, address-city, address-country, address-postalcode, address-state, address-use, birthdate, death-date, deceased, email, family, gender, general-practitioner, given, identifier, language, link, name, organization, phone, phonetic, telecom]"
}
]
}
Igor Sirkovich (May 11 2020 at 20:25):
Maybe we need to start over. What is the purpose of your extension and search by this extension? Do you want to store a message date/time from MSH.7 in each resource that was in the message and then search for resources of a specified type by this message date/time?
dsh (May 11 2020 at 20:26):
yes that is the purpose
dsh (May 11 2020 at 20:26):
so that if hl7 message needs to be reprocessed then I am not relying on _lastUpdated
which will change on each update
dsh (May 11 2020 at 20:27):
@Igor Sirkovich
as I said if I declare base
as [ "Patient" ]
and expression
as Patient.extension('http://www.caringly.tech/SearchByMsgDate')
i.e.
{
"resourceType": "SearchParameter",
"id": "10025450",
"meta": {
"versionId": "1",
"lastUpdated": "2020-05-11T16:48:00.413+00:00",
"source": "#wWpWb92nquQITjqi"
},
"url": "http://www.caringly.tech/fhir/SearchParameter/SearchByMsgDateParameter",
"version": "1.0",
"name": "SearchByMsgDateParameter",
"status": "active",
"description": "Search by patient for a MSH.7 agreement, which is stored in a any resource",
"code": "SearchByMsgDate",
"base": [
"Patient",
"Coverage"
],
"type": "date",
"expression": "Patient.extension('http://www.caringly.tech/SearchByMsgDate')",
"xpathUsage": "normal"
}
then SearchByMsgDate
on Patient
resource only works fine but does not work on Coverage
, so I need something universal like on Resource
Igor Sirkovich (May 11 2020 at 20:27):
Did you consider using Provenance (https://www.hl7.org/fhir/provenance.html) instead?
Igor Sirkovich (May 11 2020 at 20:28):
So your expression only looks at a Patient resource: "Patient.extension('http://www.caringly.tech/SearchByMsgDate')"
dsh (May 11 2020 at 20:29):
@Igor Sirkovich to be fair Provenance
seemed like an overkill .... when only one cross cutting attribute needs to be added to get the work done. But to be honest I don't know much about Provenance so I may be wrong in my analysis.
dsh (May 11 2020 at 20:30):
Igor Sirkovich said:
So your expression only looks at a Patient resource: "Patient.extension('http://www.caringly.tech/SearchByMsgDate')"
Yes it does so I changed it to *.extension('http://www.caringly.tech/SearchByMsgDate')
or extension('http://www.caringly.tech/SearchByMsgDate')
and base
to Resource
but it does not help
Igor Sirkovich (May 11 2020 at 20:32):
I believe that "base": [ "Resource" ] and "expression": "extension('http://www.caringly.tech/SearchByMsgDate')" is supposed to work
dsh (May 11 2020 at 20:34):
that's what I thought too but not, let me try and remove xpath
just in case :(
dsh (May 11 2020 at 20:35):
no luck :unamused:
dsh (May 11 2020 at 20:36):
Igor Sirkovich said:
I believe that "base": [ "Resource" ] and "expression": "extension('http://www.caringly.tech/SearchByMsgDate')" is supposed to work
that's what I thought
Igor Sirkovich (May 11 2020 at 20:36):
Also ensure you don't have 2 SearchParameter resources with the same "code".
dsh (May 11 2020 at 20:38):
@Igor Sirkovich there is only one https://gist.github.com/651444647118b68cb9768acc0d335cd4
dsh (May 11 2020 at 20:41):
I wonder looking at this http://hapi.fhir.org/baseR4/SearchParameter?code=_id if SP on Resource is supposed to work only on simple dataTypes
Igor Sirkovich (May 11 2020 at 20:41):
Try to add "xpath": "f:*/f:extension[@url='http://www.caringly.tech/SearchByMsgDate'']",
dsh (May 11 2020 at 20:41):
ok
Igor Sirkovich (May 11 2020 at 20:42):
Daya Sharma said:
I wonder looking at this http://hapi.fhir.org/baseR4/SearchParameter?code=_id if SP on Resource is supposed to work only on simple dataTypes
No, it's supposed to work on all data types
dsh (May 11 2020 at 20:43):
Igor Sirkovich said:
Try to add "xpath": "f:*/f:extension[@url='http://www.caringly.tech/SearchByMsgDate'']",
nope that did not help
{
"resourceType": "SearchParameter",
"id": "10025450",
"meta": {
"versionId": "16",
"lastUpdated": "2020-05-11T20:42:37.444+00:00",
"source": "#wWpWb92nquQITjqi"
},
"url": "http://www.caringly.tech/fhir/SearchParameter/SearchByMsgDateParameter",
"version": "1.0",
"name": "SearchByMsgDateParameter",
"status": "active",
"description": "Search by patient for a MSH.7 agreement, which is stored in a any resource",
"code": "SearchByMsgDate",
"base": [
"Resource"
],
"type": "date",
"expression": "extension('http://www.caringly.tech/SearchByMsgDate')",
"xpath": "f:*/f:extension[@url='http://www.caringly.tech/SearchByMsgDate']",
"xpathUsage": "normal",
"comparator": [
"eq",
"ge",
"le",
"gt",
"lt",
"ne"
]
}
Igor Sirkovich (May 11 2020 at 20:45):
I can only suggest to try this on hapi.fhir.org to see this might be related to your local HAPI setup.
dsh (May 11 2020 at 20:47):
great suggestion @Igor Sirkovich will try and keep you posted
Igor Sirkovich (May 11 2020 at 20:50):
@James Agnew, I just wanted to confirm with you whether HAPI supports searching by an Extension using a custom SearchParameter that is defined on a Resource, i.e. is supposed to work on all resources.
Igor Sirkovich (May 11 2020 at 20:53):
@Daya Sharma, you can also try "base": [ "Patient", "Coverage" ], while keeping the current expression and xpath. If this works, then probably the issue is with HAPI support for "base": [ "Resource" ], which would require indexing all FHIR resources on the server as opposed to indexing specific resources.
dsh (May 11 2020 at 20:53):
ah that makes sense
dsh (May 11 2020 at 20:58):
@Igor Sirkovich it only works on Patient but not on Coverage , the resource that should be fetched is https://gist.github.com/5ca45a8f2b366cbb012559e08d0ac9c8 and the query is Coverage?SearchByMsgDate=>=2020-04-03
and SP is
{
"resourceType": "SearchParameter",
"id": "10025450",
"meta": {
"versionId": "17",
"lastUpdated": "2020-05-11T20:54:06.085+00:00",
"source": "#wWpWb92nquQITjqi"
},
"url": "http://www.caringly.tech/fhir/SearchParameter/SearchByMsgDateParameter",
"version": "1.0",
"name": "SearchByMsgDateParameter",
"status": "active",
"description": "Search by patient for a MSH.7 agreement, which is stored in a any resource",
"code": "SearchByMsgDate",
"base": [
"Patient",
"Coverage"
],
"type": "date",
"expression": "extension('http://www.caringly.tech/SearchByMsgDate')",
"xpath": "f:*/f:extension[@url='http://www.caringly.tech/SearchByMsgDate']",
"xpathUsage": "normal",
"comparator": [
"eq",
"ge",
"le",
"gt",
"lt",
"ne"
]
}
Igor Sirkovich (May 11 2020 at 21:06):
It's funny. Maybe try searching again in a few minutes to see if HAPI is still indexing the resources. Otherwise, can you try adding another resource type, e.g. Organization with this extension, add "Organization" to the "base" of the SearchParameter and see if this would work? This would help understand if this issue is specific to Coverage. Also, you might try to change the order of the resources in "base" to see if only the first one gets indexed.
dsh (May 11 2020 at 21:09):
@Igor Sirkovich I had the same thought, so I added SP extension to Consent
and it does not fetch the result so possible an indexing problem, but the good thing is both Coverage and Consent now do not complain about unknown parameter SearchByMsgDate
dsh (May 11 2020 at 21:10):
this begs the question, how do I know if the indexing has finished .... obviously doing this on scale in production requires insight
dsh (May 11 2020 at 21:27):
@Grahame Grieve does HAPI support searching by an Extension using a custom SearchParameter
that is defined on a Resource
as opposed to individually listed resources. And is there a way to know if indexing is of those resources with that SearchParameter
is finished or pending ?
Grahame Grieve (May 12 2020 at 00:33):
that's not my area of code - I don't know
dsh (May 12 2020 at 04:13):
@James Agnew any insight into SearchParameter with base
set to Resource
why does it not work in above ^ case ?
Vibin_chander (May 12 2020 at 08:19):
In this example it is mentioned "Resource" as Base. Is it because _id is a common search parameter for all resources? https://www.hl7.org/fhir/searchparameter-example.json.html
Vibin_chander (May 12 2020 at 08:20):
I'm trying to make base["Resource"] in HAPI and my search is not getting recognized
Vibin_chander (May 12 2020 at 08:21):
My parameter here is SearchByMsgDate
Vibin_chander (May 12 2020 at 09:34):
I tried making the search parameter resource as bundle for each resource. Still only one resource Patient works and the search parameters for the other resources does not work
Vibin_chander (May 12 2020 at 09:36):
Patient?SearchByMsgDate=>=2020-04-22 This returns value and Coverage?SearchByMsgDate=>=2020-04-22 does not. Which means even if the search parameter resource is sent as bundle with expression:Patient.extension('http://www.caringly.tech/SearchByMsgDate')
in the base as "Patient" & expression:Coverage.extension('http://www.caringly.tech/SearchByMsgDate')
in the base as "Coverage" it takes only the Patient and not for the coverage resource
Vibin_chander (May 12 2020 at 09:43):
@James Agnew I have same question as @Daya Sharma has. Is it possible to create search parameter for base ["Resource"] meaning that parameter will be applied for all the resources?. on HAPI
Vibin_chander (May 12 2020 at 11:31):
Ok I can make it work with bundles
Vibin_chander (May 12 2020 at 11:31):
it takes sometime and new message flow for the process to set in
James Agnew (May 12 2020 at 11:45):
No, that is not possible currently. You need to explicitly specify the base resource(s) you want the SearchParameter to apply to.
Vibin_chander (May 12 2020 at 11:47):
ok
Vibin_chander (May 12 2020 at 11:47):
Im doing like that only
Vibin_chander (May 12 2020 at 11:47):
Which means for each resource i need to create a search parameter right?
James Agnew (May 12 2020 at 11:48):
yup
hopefully will adress this in a future release
dsh (May 12 2020 at 14:26):
@James Agnew Thanks for chiming in. FYI this
{
"resourceType": "SearchParameter",
"id": "10025450",
"meta": {
"versionId": "17",
"lastUpdated": "2020-05-11T20:54:06.085+00:00",
"source": "#wWpWb92nquQITjqi"
},
"url": "http://www.caringly.tech/fhir/SearchParameter/SearchByMsgDateParameter",
"version": "1.0",
"name": "SearchByMsgDateParameter",
"status": "active",
"description": "Search by patient for a MSH.7 agreement, which is stored in a any resource",
"code": "SearchByMsgDate",
"base": [
"Patient",
"Coverage"
],
"type": "date",
"expression": "extension('http://www.caringly.tech/SearchByMsgDate')",
"xpath": "f:*/f:extension[@url='http://www.caringly.tech/SearchByMsgDate']",
"xpathUsage": "normal",
"comparator": [
"eq",
"ge",
"le",
"gt",
"lt",
"ne"
]
}
works for Patient not Coverage even though both are explicitly declared.
dsh (May 12 2020 at 14:31):
@Igor Sirkovich FYI, we ended up creating Bundle of SearchParameter https://gist.github.com/daya/80f7d8fbc516f2e6225a0aca82d2a831 where each SP is for one resource. This seems like a brute force but actually works while declaring multiple resources explicitly
"base": [
"Patient",
"Coverage"
],
does not work. To me that seems like a bug.
dsh (May 12 2020 at 14:50):
@James Agnew could you please confirm if ^ seems like a bug to you?
James Agnew (May 12 2020 at 15:07):
Yup, I'd call that a bug. Can you replicate the issue on hapi.fhir.org?
dsh (May 12 2020 at 15:08):
@James Agnew further it seems that _id
works on Resource
but not a custom SearchParamater that also seems like a bug. What do you think ?
dsh (May 12 2020 at 15:08):
I will try and replicate it there
James Agnew (May 12 2020 at 15:09):
_id
is a built-in search parameter (as are all the SPs that start with _
), it doesn't rely on the definition at all
dsh (May 12 2020 at 15:10):
ok than one cannot declare SP on Resource
is that a short coming or part of the spec ?
dsh (May 12 2020 at 15:12):
@James Agnew ^
James Agnew (May 12 2020 at 15:13):
i know for sure that hapi doesn't support it, but i don't know if the spec allows it
James Agnew (May 12 2020 at 15:14):
i guess one way of looking at it is that if someone subimtted a pull request that made base=Resource work, it would likely be accepted
dsh (May 12 2020 at 15:15):
ok thanks for confirming. I will let you know what hapi.fhir.org replication of this bug reveals
dsh (May 12 2020 at 19:35):
James Agnew said:
Yup, I'd call that a bug. Can you replicate the issue on hapi.fhir.org?
We are using 4.1 and hapi.fhir.org is on 5.0 snapshot, and I don't see this bug in 5.0 snapshot. But Resource
still cannot be used as base
I will see about submitting a PR
Vibin_chander (May 14 2020 at 18:42):
Im trying to understand the duplicate error we are getting in few resources
"diagnostics": "Failed to UPDATE resource with match URL \"RelatedPerson?identifier=ouridentifier" because this search matched 2 resources"
Vibin_chander (May 14 2020 at 18:43):
Any idea why this would happen. Our understanding is that, we are using the conditional PUT, meaning if the same data with the same identifier exist already it should simply update.If not it has to create one new
David Meyers (Jan 17 2022 at 15:13):
Hey FYI: We are facing the same issue with hapi 5.5 and a postgresql database. Theres a high likelihood that this is the result of multithreading importing.
Update: Same issue with mysql db
Last updated: Apr 12 2022 at 19:14 UTC