FHIR Chat · use of elasticsearch · hapi

Stream: hapi

Topic: use of elasticsearch

Jens Villadsen (Feb 17 2020 at 08:41):

@James Agnew
we see errors when trying to do valueset expansions (4.2.0) while using elasticsearch (6.8.6):

Caused by: org.hibernate.search.exception.SearchException: HSEARCH400002: Lucene query 'myCode:"(message notification advice note)"' cannot be transformed into equivalent Elasticsearch query
    at org.hibernate.search.elasticsearch.impl.ToElasticsearch.fromLuceneQuery(ToElasticsearch.java:268)
    at org.hibernate.search.elasticsearch.impl.ToElasticsearch.convertBooleanQuery(ToElasticsearch.java:302)
    at org.hibernate.search.elasticsearch.impl.ToElasticsearch.fromLuceneQuery(ToElasticsearch.java:206)
    at org.hibernate.search.elasticsearch.impl.ElasticsearchLuceneQueryTranslator.convertLuceneQuery(ElasticsearchLuceneQueryTranslator.java:41)
    at org.hibernate.search.engine.impl.ImmutableSearchFactory.createQueryDescriptor(ImmutableSearchFactory.java:308)
    at org.hibernate.search.engine.impl.ImmutableSearchFactory.createHSQuery(ImmutableSearchFactory.java:291)
    at org.hibernate.search.engine.impl.MutableSearchFactory.createHSQuery(MutableSearchFactory.java:138)
    at org.hibernate.search.impl.FullTextSessionImpl.createFullTextQuery(FullTextSessionImpl.java:77)
    at org.hibernate.search.impl.FullTextSessionImpl.createFullTextQuery(FullTextSessionImpl.java:52)
    at ca.uhn.fhir.jpa.term.BaseTermReadSvcImpl.expandValueSetHandleIncludeOrExclude(BaseTermReadSvcImpl.java:614)
    at ca.uhn.fhir.jpa.term.BaseTermReadSvcImpl.lambda$expandValueSet$0(BaseTermReadSvcImpl.java:457)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
...

• is this related to a malformed configuration/incorrect spring bindings on our side?

Jens Villadsen (Feb 17 2020 at 09:17):

its triggered by the fact that org.apache.lucene.search.MultiPhraseQuery's are not currently mapped

Jens Villadsen (Feb 17 2020 at 09:30):

Which happens here: https://github.com/jamesagnew/hapi-fhir/blob/c5c1e3196b776c8b5cb6ddbf70845c024a73148f/hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/term/BaseTermReadSvcImpl.java#L603

Jens Villadsen (Feb 17 2020 at 12:50):

@Patrick Werner - have you stumbled across this problem?

Patrick Werner (Feb 17 2020 at 15:58):

@Jens Villadsen not yet, we are still using Lucene

James Agnew (Feb 17 2020 at 18:48):

Currently, ElasticSearch 2.x is required as this is the version supported
by the current GA release of Hibernate Search. They are seemingly very
close to releasing a new major release though which will add support for
ES5/6.

Cheers,
James

James Agnew (Feb 17 2020 at 18:53):

Sorry- I take that partially back.

It's 6.x that isn't supported, but 5.x is. So you should be able to do this using a 5.x release of ES.

Jens Villadsen (Feb 17 2020 at 20:41):

Awesome! Thx.

Jens Villadsen (Feb 17 2020 at 20:42):

I dont know why I insisted on using 6.x ...

Jens Villadsen (Feb 17 2020 at 20:42):

Jens Villadsen (Feb 17 2020 at 21:28):

@James Agnew - just tried with ES 5.6.16 ...no luck

Jens Villadsen (Feb 17 2020 at 21:32):

it is caused by https://github.com/hibernate/hibernate-search/blob/e7e19ed2a64fffce8d161802e7664438b8d53340/legacy/elasticsearch/src/main/java/org/hibernate/search/elasticsearch/impl/ToElasticsearch.java#L267 (maybe not the entirely correct branch on hibernate search) - the use of a MultiPhraseQuery seems to end there - where hibernate throws an exception

Jens Villadsen (Feb 17 2020 at 21:34):

the error is thrown before there is any interaction with ES

James Agnew (Feb 18 2020 at 12:12):

Whoa, interesting.

What does the ValueSet that is triggering this look like?

Jens Villadsen (Feb 18 2020 at 15:22):

https://docs.ehealth.sundhed.dk/latest/ig/ValueSet-message-category.html

Jens Villadsen (Feb 18 2020 at 15:23):

backing codesystem: https://docs.ehealth.sundhed.dk/latest/ig/CodeSystem-ehealth-message-category.html

Jens Villadsen (Feb 18 2020 at 15:33):

@James Agnew there you go

Jens Villadsen (Feb 18 2020 at 18:53):

Loading that and calling $expand on it

James Agnew (Feb 18 2020 at 20:24):

@Jens Villadsen yup, I see this same issue with your test case. Thanks.

Fix has gone in: https://github.com/jamesagnew/hapi-fhir/pull/1717

James Agnew (Feb 18 2020 at 20:24):

(for review, not merged yet)

Jens Villadsen (Feb 18 2020 at 20:50):

@James Agnew out of curiosity, would you in any way consider my use of valuesets and codesystems exotic?

James Agnew (Feb 18 2020 at 20:54):

Heh nope, I'd say that's about as "hello world" of expanding valuesets as it gets :)

James Agnew (Feb 18 2020 at 20:55):

Although.. Technically for your specific example you could just not include any codes at all in the VS, and only include the system if your intent is to just include all codes

Jens Villadsen (Feb 18 2020 at 21:27):

hmmm ... yes that is true... can't say why that valueset ended up like that ... i don't believe thats is the general picture of the valuesets in that IG though ;)

Jens Villadsen (Feb 19 2020 at 16:31):

@James Agnew any change that this and the update of the hl7 core validation component could go into a 4.2.1?

Jens Villadsen (Feb 19 2020 at 16:36):

as an alternative (I guess/hope) I could make it my self from https://github.com/jamesagnew/hapi-fhir/pull/1719 and https://github.com/jamesagnew/hapi-fhir/pull/1717 given that the changes are not conflicting with the rest of 4.2.0

Jens Villadsen (Feb 22 2020 at 20:56):

once done, that is

Last updated: Apr 12 2022 at 19:14 UTC