Stream: hapi
Topic: Write performances
Jason Paumier (Jan 31 2022 at 15:42):
Hello!
I'm currently working on an ETL to load data into a hapi fhir jpa server. Basically, the "load" part is a kafka consumer based on https://github.com/hapifhir/hapi-fhir-jpaserver-starter. What it does is that it consumes an event which contains a json, builds a resource and then I dao.update() to upsert it into the DB. Everything is on GitHub here (https://github.com/arkhn/fhir-river/blob/main/hapi-loader/src/main/java/com/arkhn/hapi/loader/ResourceConsumer.java) if you want to have a look at it.
The problem I have is that it will only process around 30K events per minute (500 events per second). I've seen this benchmark (https://www.smilecdr.com/benchmarking-smile-cdr) where 12K resources are processed per seconds and tried to tweak the parameters I could but it's still way slower.
Do you have any idea about how I could improve that?
James Agnew (Feb 01 2022 at 16:03):
A few quick thoughts:
- The biggest bang for your buck is to bundle resources together into FHIR transactions. Your total resources ingested per second will be vastly higher if you are performing FHIR transactions with 1000 resources each than it will be if you ingest them one at a time.
- Make sure your kafka consumer has mlutiple processing threads. You want to totally saturate your CPU and IO if you can.
- Depending on the patterns of your source data, using Kafka partition keys can be very helpful in making sure that you aren't ingesting conflicting data concurrently, which really hurts performance (e.g. two threads trying to create the same patient at the same time)
- Look at the suggested performance settings linked to from the blog post. They are all described in terms of Smile CDR settings, but they are all just setting config options on HAPI FHIR DaoConfig and ModelConfig settings under the hood. Thngs like mass ingestion mode, and disabling deletes can be a big help for performance.
- Tune your search parameters. No point in leaving SPs enabled if you know you're never going to use them.
- Try taking a thread dump so see what your system is actually doing during ingestion. It can be informative to discover that (for example) you are only actually using one processing thread when you thought you had more, or most of your threads are blocked waiting for DB connections because the pool is too small, or everyone is validating and validation is slow, etc
Jason Paumier (Feb 07 2022 at 10:21):
Thanks a lot, I'll try all of that!
Last updated: Apr 12 2022 at 19:14 UTC
