Stream: ibm
Topic: re-indexing
Sridhar S (Nov 16 2020 at 06:13):
where can i find the bucket jar file fhir-bucket-4.5.0-cli.jar for re-indexing the existing resources
Lee Surprenant (Nov 16 2020 at 16:07):
hi Sridhar, fhir-bucket is pretty new and so we didn't have it in our release steps to copy the built artifact to the GitHub release yet
Lee Surprenant (Nov 16 2020 at 16:07):
can you access it from https://bintray.com/ibm-watson-health/ibm-fhir-server-releases/fhir-bucket#files/com/ibm/fhir/fhir-bucket/4.5.0 ?
Lee Surprenant (Nov 16 2020 at 16:08):
I'll go ahead and copy that into our github release too...thanks for pointing it out
Lee Surprenant (Nov 16 2020 at 16:10):
done...you'll now find it under "Assets" at https://github.com/IBM/FHIR/releases/tag/4.5.0
Sridhar S (Nov 20 2020 at 15:52):
I am facing some issue while Reindex resources
Exception in thread "main" com.ibm.fhir.database.utils.api.UndefinedNameException: org.postgresql.util.PSQLException: ERROR: relation "resource_types" does not exist
Position: 45
at com.ibm.fhir.database.utils.postgresql.PostgreSqlTranslator.translate(PostgreSqlTranslator.java:100)
at com.ibm.fhir.bucket.persistence.ResourceTypesReader.run(ResourceTypesReader.java:50)
at com.ibm.fhir.bucket.persistence.ResourceTypesReader.run(ResourceTypesReader.java:25)
at com.ibm.fhir.database.utils.common.CommonDatabaseAdapter.runStatement(CommonDatabaseAdapter.java:475)
at com.ibm.fhir.bucket.scanner.DataAccess.init(DataAccess.java:98)
at com.ibm.fhir.bucket.app.Main.scanAndLoad(Main.java:765)
at com.ibm.fhir.bucket.app.Main.process(Main.java:749)
at com.ibm.fhir.bucket.app.Main.main(Main.java:834)
Caused by: org.postgresql.util.PSQLException: ERROR: relation "resource_types" does not exist
Position: 45
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2533)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2268)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:313)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:159)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:109)
at com.ibm.fhir.bucket.persistence.ResourceTypesReader.run(ResourceTypesReader.java:41)
... 6 more
Exception in thread "Thread-4" java.lang.NullPointerException
at com.ibm.fhir.bucket.app.Main.shutdown(Main.java:699)
at com.ibm.fhir.bucket.app.Main.lambda$scanAndLoad$1(Main.java:759)
at com.ibm.fhir.bucket.app.Main$$Lambda$1/000000000831B060.run(Unknown Source)
at java.lang.Thread.run(Thread.java:823)
Sridhar S (Nov 20 2020 at 15:53):
can anyone help with this
Sridhar S (Nov 20 2020 at 15:53):
java -Djava.util.logging.config.file=logging.properties -jar fhir-bucket-4.5.0-cli.jar --fhir-properties fhir.properties --tenant-name demo1 --max-concurrent-fhir-requests 200 --no-scan --reindex-tstamp 2020-11-20 --reindex-resource-count 10 --reindex-concurrent-requests 20 --db-type postgresql --db-properties postgresql.properties --cos-properties cos.properties
John Timm (Nov 20 2020 at 16:00):
@Robin Arnold have you seen anything like this? :point_up:
Robin Arnold (Nov 20 2020 at 16:09):
This comes from the schema used internally by fhir-bucket. This schema is used when fhir-bucket is used to load data from object storage - it shouldn't be needed if all you are doing is reindexing, but it must be still looking for it. I'll take a look at the code-path. If it's needed, the simplest thing would be to run the create schema step. Let me get back to you.
Robin Arnold (Nov 20 2020 at 16:20):
Unfortunately with this version you'll need run the above fhir-bucket command once with the --create-schemaoption. Make sure that your postgresql.properties file is pointing to the database you want to use. This doesn't have to the same database as the FHIR server, but it certainly can be. Make sure you have currentSchema=FHIRBUCKET in that properties file too to direct the code to create the tables in the FHIRBUCKET schema.
Sorry about this - the intention was that the schema wouldn't be required for the reindexing function. We have an open issue to tidy up the CLI, so I'll make a note about this.
Sridhar S (Nov 20 2020 at 16:35):
After following the step even though the column "resource_type_id" is not null i am getting this error
Sridhar S (Nov 20 2020 at 16:35):
Exception in thread "main" com.ibm.fhir.database.utils.api.DataAccessException: org.postgresql.util.PSQLException: ERROR: null value in column "resource_type_id" violates not-null constraint
Detail: Failing row contains (null, Appointment).
at com.ibm.fhir.database.utils.postgresql.PostgreSqlTranslator.translate(PostgreSqlTranslator.java:104)
at com.ibm.fhir.bucket.persistence.MergeResourceTypesPostgres.run(MergeResourceTypesPostgres.java:59)
at com.ibm.fhir.database.utils.common.CommonDatabaseAdapter.runStatement(CommonDatabaseAdapter.java:459)
at com.ibm.fhir.database.utils.postgresql.PostgreSqlAdapter.runStatement(PostgreSqlAdapter.java:266)
at com.ibm.fhir.bucket.app.Main.bootstrapDb(Main.java:653)
at com.ibm.fhir.bucket.app.Main.process(Main.java:747)
at com.ibm.fhir.bucket.app.Main.main(Main.java:834)
Caused by: org.postgresql.util.PSQLException: ERROR: null value in column "resource_type_id" violates not-null constraint
Detail: Failing row contains (null, Appointment).
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2533)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2268)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:313)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:159)
at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:125)
at com.ibm.fhir.bucket.persistence.MergeResourceTypesPostgres.run(MergeResourceTypesPostgres.java:54)
... 5 more
Sridhar S (Nov 20 2020 at 16:35):
Screenshot-2020-11-20-at-22.03.30.png
Robin Arnold (Nov 20 2020 at 16:41):
I'll have to check where this is coming from. It looks like a regression. But if the schema was generated (fhir_bucket.resource_types exists), then you should now be able to run the reindex operation.
Robin Arnold (Nov 20 2020 at 16:46):
What is your currentSchema set to in your postgresql.properties file? You must not use the FHIRDATA schema here - it has to be a different schema name for example FHIRBUCKET.
Sridhar S (Nov 21 2020 at 01:05):
(deleted)
Sridhar S (Nov 21 2020 at 04:22):
yes after changing currentSchema=FHIRBUCKET the above issue got resolved thank you
Sridhar S (Nov 21 2020 at 04:23):
can someone share a default fhir.properties
Sridhar S (Nov 23 2020 at 16:50):
(deleted)
Sridhar S (Nov 24 2020 at 12:25):
2020-11-24 09:07:13.133 00000025 INFO .reindex.DriveReindexOperation Reindex - all done
After i got the message that reindex is all done the process is still been running for hours is it right behaviour or am i missing something
Sridhar S (Nov 24 2020 at 12:25):
2020-11-24 12:23:47.560 00000024 INFO .fhir.bucket.scanner.COSReader allocateJobs[NDJSON]: free = 1
2020-11-24 12:23:47.561 00000024 INFO .fhir.bucket.scanner.COSReader Allocated job count[NDJSON]: 0
2020-11-24 12:23:47.561 00000024 INFO .fhir.bucket.scanner.COSReader Jobs inflight[NDJSON] 0, just allocated: 0
2020-11-24 12:23:57.561 00000024 INFO .fhir.bucket.scanner.COSReader allocateJobs[NDJSON]: free = 1
2020-11-24 12:23:57.562 00000024 INFO .fhir.bucket.scanner.COSReader Allocated job count[NDJSON]: 0
2020-11-24 12:23:57.562 00000024 INFO .fhir.bucket.scanner.COSReader Jobs inflight[NDJSON] 0, just allocated: 0
2020-11-24 12:24:07.562 00000024 INFO .fhir.bucket.scanner.COSReader allocateJobs[NDJSON]: free = 1
2020-11-24 12:24:07.563 00000024 INFO .fhir.bucket.scanner.COSReader Allocated job count[NDJSON]: 0
2020-11-24 12:24:07.563 00000024 INFO .fhir.bucket.scanner.COSReader Jobs inflight[NDJSON] 0, just allocated: 0
2020-11-24 12:24:17.563 00000024 INFO .fhir.bucket.scanner.COSReader allocateJobs[NDJSON]: free = 1
2020-11-24 12:24:17.564 00000024 INFO .fhir.bucket.scanner.COSReader Allocated job count[NDJSON]: 0
2020-11-24 12:24:17.565 00000024 INFO .fhir.bucket.scanner.COSReader Jobs inflight[NDJSON] 0, just allocated: 0
2020-11-24 12:24:27.565 00000024 INFO .fhir.bucket.scanner.COSReader allocateJobs[NDJSON]: free = 1
2020-11-24 12:24:27.566 00000024 INFO .fhir.bucket.scanner.COSReader Allocated job count[NDJSON]: 0
2020-11-24 12:24:27.566 00000024 INFO .fhir.bucket.scanner.COSReader Jobs inflight[NDJSON] 0, just allocated: 0
guokun zhang (Nov 26 2020 at 18:11):
Dear IBM team,
we are also trying to update our local ibm fhir server from version 4.4.2 to version 4.5.0.
As suggested from here (https://github.com/IBM/FHIR/releases) and also from here (https://github.com/IBM/FHIR/tree/master/fhir-bucket), we need to do "Database Migration" which includes "Schema Migration" and "Reindex resources". We have done "Schema Migration" but are stuck with "Reindex resources". Would you please help us with the following questions:
-
Is this "Reindex resources" definitely necessary for the updates from 4.4.2 to 4.5.0 with existing fhir resources? Without doing this the existing resources will not be searched correctly?
-
The command includes "--fhir-properties fhir.properties", it is the connection properties file for the FHIR server. But how exact it should be? Is there a template available?
-
The command includes "--cos-properties cos.properties", it is the connection properties file for COS. Is it only optional? I am asking because we are not using IBM COS bucket and don't understand why we need this for reindex. But running withou this returned:
Exception in thread "main" java.lang.IllegalArgumentException: No COS properties at com.ibm.fhir.bucket.app.Main.checkConfig(Main.java:473) at com.ibm.fhir.bucket.app.Main.main(Main.java:832)
Many thanks!
Lee Surprenant (Nov 27 2020 at 12:43):
Is this "Reindex resources" definitely necessary for the updates from 4.4.2 to 4.5.0 with existing fhir resources?
yes
Without doing this the existing resources will not be searched correctly?
correct
Lee Surprenant (Nov 27 2020 at 12:44):
however, fhir-bucket is not required for invoking the $reindex operation...it was just for convenience
Lee Surprenant (Nov 27 2020 at 12:47):
I recently drafted https://github.com/IBM/FHIR/pull/1730 to explain the operation a little bit better. Basically you just need to repeatedly POST to this endpoint with a tstamp parameter that is later than the time in which the db schema migration was applied.
Lee Surprenant (Nov 27 2020 at 13:23):
If you do decide to use the fhir-bucket utility, the best documentation for that is in the README for that module: https://github.com/IBM/FHIR/tree/master/fhir-bucket#configuration
Lee Surprenant (Nov 27 2020 at 13:23):
there you will find templates for both fhir.properties and cos.properties
Lee Surprenant (Nov 27 2020 at 13:25):
however, one thing we recently found is that there is a bad bug in the 4.5.0 reindex implementation which is preventing it from properly re-indexing certain search parameters. That issue is https://github.com/IBM/FHIR/issues/1742 and we'll be rolling a 4.5.1 ASAP to get that addressed (hopefully monday)
guokun zhang (Dec 09 2020 at 11:04):
(deleted)
Last updated: Apr 12 2022 at 19:14 UTC
