Stream: pascal
Topic: SNOMED CT Importer "Out of memory" errors
Rob Hausam (Feb 09 2021 at 03:23):
It looks like we have a rather serious problem with the the latest version(s) of the SNOMED CT Importer that are included in the FHIR Console. After that work was done, on Dec 7 I tried to import the Swedish Edition, but that failed with an "Out of memory" error on tx.fhir.org. The question was raised about possibly increasing the available RAM on the the server, but instead @Grahame Grieve suggested doing the import using a local instance. And, Grahame's statement at the time that "the server is more of a ram consumer than it used to be" turns out possibly to be the biggest understatement of the year (even in the year of COVID-19!). :) When I finally tried doing that, I found that the imports (this time of the International Edition) were still failing with "Out of memory", even on Windows 10 VMs with increasingly large amounts of RAM - up until my attempt today which failed with 256 GB (see attached screenshots)! This is obviously completely unworkable, and it makes no sense to try anything further. So at this point I can't import and load the 20210131 International Edition, which is now available and I was asked for last week, or probably any other SNOMED CT Edition. So we need to get this fixed (I assume I should submit a GH issue?). Screen-Shot-2021-02-08-at-18.35.03.png Screen-Shot-2021-02-08-at-18.41.23.png
Grahame Grieve (Feb 09 2021 at 19:54):
where's the easiest place for me to get the source for the international edition?
Rob Hausam (Feb 09 2021 at 20:54):
@Grahame Grieve It's on tx.fhir.org in D:\download\SnomedCT_InternationalRF2_PRODUCTION_20210131T120000Z (the complete release).
Grahame Grieve (Feb 10 2021 at 01:36):
well, I can reproduce the problem. That's using crazy amounts of memory
Rob Hausam (Feb 10 2021 at 01:38):
Yes. At least you're seeing the same thing - that should be the first step in getting it fixed.
Grahame Grieve (Feb 10 2021 at 03:04):
http://healthintersections.com.au/snomed_20210131_intl.zip but I'll have to do a new release in order to use it
Rob Hausam (Feb 10 2021 at 03:27):
Looks like you're getting there?
Rob Hausam (Feb 10 2021 at 03:41):
And primarily you're removing (commented out) some of the sorting code?
Grahame Grieve (Feb 10 2021 at 03:53):
I reduced the amount of memory allocated to undertanding the relationships. not sure why it was consuming so much memory
Grahame Grieve (Feb 10 2021 at 04:53):
ok it's all released now
Grahame Grieve (Feb 10 2021 at 04:54):
I'll upgrade tx.fhir.org shortly
Rob Hausam (Feb 12 2021 at 15:15):
@Grahame Grieve I assumed "all released" would mean that it is updated on the Tools Release page, but the version there is still v0.0.369 dated 2020-12-18. I still haven't finished working out the openssl issue for launching my compiled version, so if I can get the updated binary for now that will be good.
Grahame Grieve (Feb 17 2021 at 20:15):
.370 for me - you have a caching problem?
Rob Hausam (Feb 17 2021 at 22:35):
@Grahame Grieve I don't think so. On the 12th when I posted that it was still .369 - but now today (before I saw your reply above) it's .370. So, with that, I was finally able to get a successful cache file generated for the 2021-01-31 International Edition and now that is loaded on tx.fhir.org. But it's still using large amounts of RAM - though fortunately somewhat smaller than before (screen shot attached). So I was able to do the import with the 256 GB VM, and it looks like I should be able to scale it back and it would work (barely) with 128 GB - but that's still pretty massive, and much more than we ever needed in the past. Screen-Shot-2021-02-17-at-14.42.55.png
Last updated: Apr 12 2022 at 19:14 UTC
