Stream: hapi
Topic: Performance as FHIR server
Binu DGIT (Nov 01 2016 at 10:35):
I tried to use HAPI server as FHIR server. I have a request queue placed for the interaction from the client. HAPI server fetch each message from the request queue and process it and successful messages will be moved to response queue. But it is found that HAPI server occupies entire processor load as well as the memory. It is almost 12 minutes to process 1000 request. I am using Oracle database to store FHIR messages. It is not scalable in this way. Can you please give me some suggestions to make the processing bit faster..
Karl M. Davis (Nov 01 2016 at 12:01):
Binu, there are a whole host of issues to consider, so apologies for all of the incoming questions:
1. What version of HAPI are you using?
2. Do you need to support full text search or terminology services?
3. What is the ratio of reads to writes in those 1000 requests?
4. How many requests per second do you need to support?
5. Are you using HAPI's JPA database layer, or is HAPI just wrapping some other backend?
Binu DGIT (Nov 02 2016 at 04:31):
Karl, Thank you very much for the response. Please find my response to your queries
1. What version of HAPI are you using?
-- Version 1.4
2. Do you need to support full text search or terminology services?
-- Not necessary
3. What is the ratio of reads to writes in those 1000 requests?
-- Currently 2 minutes is taken to process 100 requests, want to achieve the same in 1 second.
4. How many requests per second do you need to support?
--100 requests / second
5. Are you using HAPI's JPA database layer, or is HAPI just wrapping some other backend?
--I am using HAPI JPA database layer storing data in Oracle database..
what I found today that the bottleneck is with FHIRValidator using the XML file.. The file is huge and it occupies more memory. Further not enough memory is available for the process to utilized. Is there any mechanism to cache the XML and do the validation? or which is the better mechanism to make validation faster?
Binu DGIT (Nov 03 2016 at 04:14):
what I found today that the bottleneck is FHIRValidator using the XML file.. The file is huge and it occupies more memory. Is there any mechanism to cache the XML and do the validation? or which is the better mechanism to make validation faster?
Grahame Grieve (Nov 03 2016 at 06:00):
@James Agnew where did we get to talking about that? The validator is not thread safe, but it's the context that matters - the validator doesn't load or cache it's own stuff.
James Agnew (Nov 03 2016 at 10:48):
Hi Binu,
Assuming that you're keeping an instance of HAPI's DefaultProfileValidationSupport in memory, and reusing it for all validation activities? That's the class that loads the profile StructureDefinitions into memory and it will keep a cache going as long as you are reusing it. You should see lines like the following in your server logs only once:
06:46:03.353 [main] INFO o.h.f.d.h.v.DefaultProfileValidationSupport - Loading structure definitions from classpath: /org/hl7/fhir/instance/model/dstu3/profile/profiles-resources.xml 06:46:04.146 [main] INFO o.h.f.d.h.v.DefaultProfileValidationSupport - Loading structure definitions from classpath: /org/hl7/fhir/instance/model/dstu3/profile/profiles-types.xml 06:46:04.188 [main] INFO o.h.f.d.h.v.DefaultProfileValidationSupport - Loading structure definitions from classpath: /org/hl7/fhir/instance/model/dstu3/profile/profiles-others.xml
Binu DGIT (Nov 03 2016 at 11:13):
Thank you James for the timely response..
I am using HAPI 1.4 and DSTU2 in place. It is found that profile resources are spread among many XML files than the three mentioned above in DSTU3 It seems the mistake which I have done that each time new instance of the validator is initiated. I ll have a detailed look into this and let you know the root cause..
James Agnew (Nov 03 2016 at 11:27):
Sounds good.
If at all possible, I'd also recommend looking at upgrading HAPI to the latest version (2.0). HAPI FHIR 2.0 still supports the exact same FHIR structures for DSTU2 but has lots of fixes including performance fixes that may help you.
Jasmine Ameerdeen (Nov 06 2016 at 09:17):
Actually, DefaultProfileValidationSupport instance is created every time the FhirResourceDaoDstu2.validate() is invoked. May be this is causing the issue.
FhirInstanceValidator val = new FhirInstanceValidator();
val.setBestPracticeWarningLevel(BestPracticeWarningLevel.Warning);
val.setValidationSupport(new ValidationSupportChain(new DefaultProfileValidationSupport(), myJpaValidationSupport));
validator.registerValidatorModule(val);
Is it okay to autowire this instance?
James Agnew (Nov 06 2016 at 12:49):
Oh yeah, you definitely want to keep a single instance of that class loaded and reuse it. That will certainly cause bad performance otherwise.
Last updated: Apr 12 2022 at 19:14 UTC
