Stream: cql
Topic: Android Versions of DBCG libs
Vitor Pamplona (Dec 02 2021 at 19:06):
Has anybody worked on Android ports of the https://github.com/DBCG/cql_engine and https://github.com/DBCG/cql-evaluator libs?
It feels like the project could benefit from a pure Android-based dependency stack to reduce the chance of runtime errors between the multiple versions of Android out there. I saw the SampleApp from the DBCG team and even that Sample code is not that stable on multiple android devices. :(
Vitor Pamplona (Dec 02 2021 at 19:09):
For additional information, Moxy has a dependency on java.awt.Image which is not part of Java anymore and if an Android project is compiled with the latest Android Studio, which Java 11 is used by default, it causes a RuntimeError when trying to unmarshal an XML Library because java.awt is not there. (Don't ask why an XML lib needs AWT... )
Vitor Pamplona (Dec 02 2021 at 19:12):
At the same time, Context has a dependency on ZonedDateTime, which is not included in some Android <=7.0 devices which are still widely used :(
Vitor Pamplona (Dec 02 2021 at 19:23):
Since the CQL evaluator and engine don't actually need java.awt and the ZonedDateTime evaluationDateTime field on Context doesn't seem to be that useful, a dependency clean up + field deletion could improve the stability of the lib without affecting it's utility.
JP (Dec 02 2021 at 19:27):
the main is the use of JAXB for serialization and serialization of ELM, and of the ModelInfo in the cql-translator. There are a few issues tracking that, such as these:
https://github.com/cqframework/clinical_quality_language/issues/640
https://github.com/DBCG/cql_engine/issues/405
JP (Dec 02 2021 at 19:28):
Short version is that running on Android is a goal, but there's some legwork to do before we can get there.
Vitor Pamplona (Dec 02 2021 at 19:46):
Nice! Sorry, what's the difference between the XML from the cql-translator and the XML from the engine? Issue 405 speaks about both as if they were different, but they seem to be calling the same JAXB processor. Are they the same?
JP (Dec 02 2021 at 21:08):
The XML content itself is identical, but the two projects are independent so both implement JAXB serialization / deserialization separately. There are a couple cases where the behavior isn't consistent between the two.
Vitor Pamplona (Dec 03 2021 at 13:40):
If we could put some resources to help you out on this process, where or which tasks would you ask us to look into first? :)
JP (Dec 03 2021 at 21:04):
I'll try to give an overview here but I'm happy to jump on a call with you and your team to talk through some considerations if that's helpful. Also, @Jing Tang is working on an Android SDK so he may also be interested in this.
A a high level, CQL (human-readable) is translated by the cql-translator to produce ELM, which is machine-readable. The ELM is the executable code. Technically, the cql-engine and cql-evaluator do not need CQL at all. They can run directly on ELM assuming that ELM is available in the execution environment (the cql-evaluator will do CQL translation on the fly if CQL is available and the execution environment supports it).
ELM has a number of different serialization formats, which are what are being discussed in this issue:
https://github.com/DBCG/cql_engine/issues/405
JSON = JSON serializaed by JAXB
XML = XML serialized by JAXB
JXSON = JSON serialized by Jackson
JAXB is (generally) unsupported by Android, so only the JXSON format can be used.
The CPG IG defines a few profiles that constrain Clinical Reasoning resources such that ELM is always present, such as CPGExecutableLibrary. An overview of that is here:
https://build.fhir.org/ig/HL7/cqf-recommendations/profiles.html#artifact-profiles
So, if you generate ELM ahead of time, and if you serialize it as JXSON, and it you attach that to your FHIR resources (or provide some other means of loading it), and if you have a fairly recent version of Android, the cql-engine will work. That's a lot of ifs, but some of that is already automated by the IG publisher tooling or cqf-tooling projects.
On issue #405, we've come to the consensus that we can phase out JSON/XML support in the cql-engine and support only the JXSON format. So, the work to do there is create a PR that removes JAXB and the associated dependencies entirely. You'll still have to pregenerate the ELM, and you'll still have to serialize it as JXSON, but at least it should work on more versions of Android.
JP (Dec 03 2021 at 21:11):
An approach I'd actually prefer is to provide some mechanism such that the JAXB-related dependencies can be loaded dynamically, such as putting them in their own module, marking them "optional" in the pom.xml, and providing some way to detect those on the classpath. The core HAPI libraries use such an approach to support Android:
(they swap-out HTTP client implementations)
I haven't sat down and thought through exactly what the design for this should be, or sketched out what the interface for a serialization format "module" would look like. If you or your team wanted to take a stab at that, it'd be helpful and probably pay dividends on the cql-translator side as well.
JP (Dec 03 2021 at 21:13):
That gets us running ELM consistently across most versions of Android, I think.
JP (Dec 03 2021 at 21:19):
The next step is to be able to do CQL translation on Android so that you don't necessarily have to pregenerate the ELM. There are a couple of related issues on the cql-translator side that need to be addressed to make that possible. First, the serialization of the translated ELM which effectively needs the same "optional JAXB modules" solution.
Second, the cql-translator internally uses a "ModelInfo" to store meta-data about the data model. It loads these whenever it encounters a "using" statement in CQL:
library Test
using FHIR version 3.0.0
or
using QDM version 5.6
or whatever.
JP (Dec 03 2021 at 21:22):
The structure of the ModelInfo is documented here:
https://cql.hl7.org/elm.html
And there are a number of examples here:
https://github.com/cqframework/clinical_quality_language/tree/master/Src/java/quick/src/main/resources/org/hl7/fhir
You'll notice they are all XML. This is because the cql-translator uses JAXB to read those. We need the equivalent of JXSON-based serialization for those ModelInfos in order to be able to use them on Android.
JP (Dec 03 2021 at 21:26):
The FHIR ModelInfos are generated based on the StructureDefintions, so they are pretty easily recreated. The generation tooling will need to write those out as JXSON (or whatever) and then the cql-translator will need to be able to load those. The code that currently does that is here:
https://github.com/cqframework/clinical_quality_language/blob/68e6861b38603e6d87d822b938cc1a5ed354d7a7/Src/java/cql-to-elm/src/main/java/org/cqframework/cql/cql2elm/FhirModelInfoProvider.java
JP (Dec 03 2021 at 21:32):
To summarize, full CQL to ELM to execution support on Android needs:
- The ELM JAXB dependencies removed (or made optional) in the cql-engine
- The ELM JAXB dependencies removed (or made optional) in the cql-translator
- The ModelInfo JXSON format added to the ModelInfo generation tooling
- The ModelInfo JAXB dependencies removed (or made optional) in the cql-translator
- The ModelInfo JXSON format added to the cql-translator
JP (Dec 03 2021 at 21:33):
It's not a ton of work but it requires digging around in the internals of those projects a bit.
Vitor Pamplona (Dec 03 2021 at 22:11):
Wow! Thank you so much for this detailed description. How confident are we to use the current translator to produce the Jackson files correctly? I saw some fears of incorrectly translating it on the referred issue links. Do you think this XML to Jackson conversion will be a one and done event or something that we will need to keep updating as an upstream updates itself?
Vitor Pamplona (Dec 03 2021 at 22:12):
Flagging for @Carl Leitner
Bryn Rhodes (Dec 03 2021 at 22:18):
For a target here, do we have a minimum Android version we want to make sure we support? I see <= 7.0 above, but does that really mean all the way back to 1.0?
Vitor Pamplona (Dec 03 2021 at 22:32):
My initial goal was to go down to Android 5.0, but any gains are welcome.
Vitor Pamplona (Dec 03 2021 at 22:35):
For the project, the goal has to be a clear definition of which versions of Android the lib would work with a single line to be added to the gradle file (no gradle setup shenanigans).
Bryn Rhodes (Dec 03 2021 at 22:38):
For context on the issues with translation, they are all around the "annotations" which we use to include debug/source information in the output ELM. The core ELM translation is fine, and we have an issue to change how we do annotations so that it's not an issue going forward. Ultimately, we want to be able to remove the JAXB dependencies entirely.
JP (Dec 03 2021 at 23:19):
Right, we are fairly confident in JXSON overall with the exception of the annotations. We've used the JXSON pretty extensively at this point. We are tracking of couple issues to improve the test coverage but I don't foresee any real changes with respect to the actual executable content.
JP (Dec 03 2021 at 23:25):
As far as the minimum Android version supported... I'll have to go back and figure out what's possible. I did some testing last year and as I recall there were some other issues that prevented supporting an API level below ~24, which is ~75% of Android devices. Apparently I didn't write it down in my notes though. Getting down to 20 would give us ~98% of the Android ecosystem.
Paul Denning (Dec 06 2021 at 14:20):
Note that CQL to ELM translations can use cql-options.json so any on-the-fly translation of CQL to ELM needs to consider those options.
for example https://github.com/cqframework/ecqm-content-r4/blob/master/input/cql/cql-options.json
Also tools which expect JAXB ELM to do CQL highlighting may break if given JXSON
Vitor Pamplona (Dec 07 2021 at 00:13):
Quick report on how this is going. I was able to remove most of the JAXB dependencies (Moxy is not there anymore) and change the source and test cases to save and load JXSON. There are some failing tests remaining (20 or so) but we should be able to get it ready soon.
Vitor Pamplona (Dec 07 2021 at 00:14):
Code is here: https://github.com/Path-Check/cql_engine I will do a PR a soon as we feel comfortable with these changes
Vitor Pamplona (Dec 07 2021 at 16:18):
FYI, I had to force
CqlTranslator.getJxsonMapper().setSerializationInclusion(Include.NON_NULL);
before calling translator.toJxson everywhere because the JXSON translator ignores empty ('') strings in the default CqlTranslator.getJxsonMapper() behavior.
Vitor Pamplona (Dec 07 2021 at 16:19):
Other than that, all tests pass with the JXSON conversion instead of the old XML/JAXB methods
Vitor Pamplona (Dec 07 2021 at 20:45):
@JP do we like big PRs or small PRs? I think I am ready to do the first one that addresses your first point:
1. The ELM JAXB dependencies removed (or made optional) in the cql-engine
JP (Dec 07 2021 at 20:45):
Smaller is better.
Vitor Pamplona (Dec 07 2021 at 21:12):
Ok, PR up: https://github.com/DBCG/cql_engine/pull/518
Vitor Pamplona (Dec 08 2021 at 01:26):
@JP @Bryn Rhodes I am assuming the next step is to move the XML files here to JSON. Were these XML files manually generated or is there a tool doing it inside the repo? (I can't find it)
JP (Dec 08 2021 at 17:47):
The code for the FHIR ModelInfo generation is in the cqf-tooling project:
https://github.com/cqframework/cqf-tooling/blob/002fbccbd76e0f9d4c7a871173829cb80d25e5c9/src/main/java/org/opencds/cqf/tooling/modelinfo/StructureDefinitionToModelInfo.java#L45
We've thought about moving that to the cql-translator project to keep them collocated. I don't think I've made an issue for that yet though, so I'll do that (and it doesn't need to be done as part of this effort, IMO, unless that's an easier approach for you).
JP (Dec 08 2021 at 17:49):
The QDM ModelInfoes are created separately. As long as we retain the ability to load those as XML I don't think you need to worry about them for your current use case:
https://github.com/cqframework/clinical_quality_language/tree/master/Src/java/qdm/src/main/resources/gov/healthit/qdm
JP (Dec 08 2021 at 17:51):
As far as your engine PR, thanks for the contribution! I'll review it as soon as I can. It might take me a couple days to get started on it. Unless @Bryn Rhodes can jump on it first... :smile:
Vitor Pamplona (Dec 08 2021 at 22:06):
Nice! Do you mind sharing what is on your FHIR-Spec folder? I don't know how to download all specs for all versions easily.
JP (Dec 08 2021 at 22:37):
Unfortunately there's not a way to easily download all the specs or every version including US/QICore, but here's a link to my FHIR-Spec folder:
https://drive.google.com/file/d/1YsJd2AIg1DijCQTtmsqN592JcdQjb2S9/view?usp=sharing
JP (Dec 08 2021 at 22:39):
(collected from downloads like this one: https://www.hl7.org/fhir/downloads.html)
Vitor Pamplona (Dec 08 2021 at 23:00):
Há! Thank you @JP
Carl Leitner (Dec 10 2021 at 17:17):
Beyond the above discussion, how much is left to do to get $apply fully working on android? Is that spec'ed out anywhere? I didn't see it on the kanban. @JP @Bryn Rhodes
JP (Dec 10 2021 at 17:30):
There’s a some work in progress here:
https://github.com/DBCG/cql-evaluator/pull/42
It’s probably 3/4 complete. The caveats due to the JAXB issues means it only works for a very narrow range of content and Android devices.
Carl Leitner (Dec 10 2021 at 17:43):
OK. So if combined with https://github.com/DBCG/cql_engine/pull/518 does that get us most of the way there? Is the other 1/4 something someone can pick-up?
JP (Dec 10 2021 at 17:48):
There’s feedback on the PR that needs to be addressed. Specifically, the operation API is not aligned correctly with the CPG $apply specification. It’d be easiest for the original dev to make those changes and address any bit rot, but they are currently occupied with some other work. If you want to jump in there or know someone that can you are welcome to. If not, I’ll get to it “soon”.
Carl Leitner (Dec 10 2021 at 17:50):
JP said:
There’s feedback on the PR that needs to be addressed. Specifically, the operation API is not aligned correctly with the CPG $apply specification. It’d be easiest for the original dev to make those changes and address any bit rot, but they are currently occupied with some other work. If you want to jump in there or know someone that can you are welcome to. If not, I’ll get to it “soon”.
@Vitor Pamplona ?
JP (Dec 10 2021 at 17:50):
@Vitor Pamplona - Bryn’s working on some translator changes that should clean up the engine PR a bit:
https://github.com/cqframework/clinical_quality_language/issues/702
Vitor Pamplona (Dec 10 2021 at 19:08):
Nice! I was going to do a PR on the cqframework/clinical_quality_language but that project is way too crazy for me :)
Vitor Pamplona (Dec 10 2021 at 19:11):
But I am getting ready to open a PR for the CQL-Evaluator to remove JAXB-based things from the evaluator.engine and evaluator.fhir
Vitor Pamplona (Dec 10 2021 at 19:11):
After changes to the engine and evaluator, I was able to successfully open a JSON CQL on Android
Vitor Pamplona (Dec 12 2021 at 02:33):
App is here if anyone wants to see how it worked: https://github.com/Path-Check/who-verifier-app
Vitor Pamplona (Dec 12 2021 at 03:31):
Next up, we will be investigating why it takes so long to load and run Jackson-based CQL on Android (4-5 seconds).
JP (Dec 13 2021 at 14:44):
There’s a few places that benefit from caching, if you’ve not already done so:
In particular the ModelResolvers use quite a bit of reflection. I think the ideal end goal for these is a code gen step in the cql-engine build, but in the meantime this may help.
JP (Dec 13 2021 at 14:45):
There’s where you’d use the caching decorator.
Vitor Pamplona (Dec 13 2021 at 19:08):
Any strong feelings about the SLF4J dependency? It's quite useless for Android.
JP (Dec 13 2021 at 19:28):
The api or the simple-logger? No opinion on the simple-logger and it should be marked as optional.
For the api, we're supporting a number of platforms besides Android and the slf4j-api allows us to use whatever logging implementation is available on the platform. If slf4j-api is not supported on Android we need _some_ logging solution that works "everywhere". What's the recommended approach on Android?
A quick google search suggests there are slf4j binding available for android. I have no idea how mature they might be though:
https://github.com/nomis/slf4j-android
https://github.com/tony19/logback-android
Vitor Pamplona (Dec 17 2021 at 00:31):
@JP @Bryn Rhodes I just opened another PR to update the Branch of the previous PlanDefinition/$apply PR: https://github.com/DBCG/cql-evaluator/pull/74
Vitor Pamplona (Dec 17 2021 at 00:32):
This solves most of the easy requests by @JP. However, the test suite is not complete and the code is still quite verbose/ugly.
Vitor Pamplona (Dec 17 2021 at 00:34):
I recommend merging PR 74 with PR 42 and merging PR 42 into master nevertheless. With the code on master, we can start to improve both the tests and the current implementation with new PRs.
Last updated: Apr 12 2022 at 19:14 UTC
