cqframework / clinical-reasoning Goto Github PK

I've been wondering about the maximum performance possible with cql_engine for use over populations. The case I'm looking at is evaluating a single CQL library (along with it's included libraries and ValueSets) against a population of patient FHIR Bundles.

I was having issues with the evaluator Dagger api because I wanted to ensure compiling the CQL to engine.Library only once (instead of on every iteration of the loop of the patient which required a new DataProviderFactory). But I was able to use cql_engine more directly to get it working with the same underlying classes.

In the test I'm using 1,000 Synthea FHIR bundles. First, they are all loaded from disk into memory and parsed as HAPI FHIR bundles. Then all the CQL is translated into ELM and loaded into engine.Library classes.

The test uses a small to mid sized library with 4 total included libraries (including FHIRHelpers). It also includes two ValueSets with 116 codes and 15 codes respectively.

The current performance is quite good with 2,745 FHIR Bundles per second over 15 runs. But just to see if there was any room for improvement, I created a FlameGraph to check out the execution and some low hanging fruit immediately popped out:

81% of the execution is taking place in the anyCodeInValueSet function.

This algorithm is effectively an O(mn) calculation, where:

• m is the number of resources of a certain type (such as Encounter resources)
• n is the number of codes in the ValueSet

Instead, if the Coding was stored in a hashable lookup of some sort you should be able to use set operations and get this down to O(min(m,n)). See python set operations time complexity for reference.

Instead of:

for resource in resources:
     for valueSetCode in valueSet.codes:
          if (resource.code === valueSetCode)
               return true

You could do:

for resource in resources:
      if (resource.code in valueSet.hashedCodeLookup)
           return true

There are a few things to work out, such as there being multiple codes in a resource, or the case that a ValueSet has less codes than the number of resources where you'd want to flip the loop.

I'll spend a little time prototyping when I get a chance, but interested in your thoughts on this and if it makes sense.

I cloned the cql-evaluator repository about 3 days ago and built the project. I ran into this IllegalArgumentException, which I do not believe is a valid error. In order to make sure that this was not specific to version 1.4.6, I re-cloned and rebuilt the cql-evaluator this morning. I now have version 2.1.0 of the cql-evaluator and this issue still remains.

A test file named Test2.cql.txt (.txt because github's file attacher will not take a .cql file) is attached Test2.cql.txt. When I run this file in the evaluator (using the command line java -jar evaluator-2.1.0.jar cql -lu . -ln Test2 --terminology-url=vocabulary\valueset -fv R4 -c "Unfiltered=Add Medical API Rule 1" -mu \Users\fadrian\Projects\cql-test\input\tests), execution succeeds as I expect, giving the output:

test=org.opencds.cqf.cql.engine.runtime.ValueSet@4cbd03e7
testA=[Procedure(id=2B70A37918AAA2E54240A967896F7B95-46319170-PRCDR), Procedure(id=4C00C4C21FED574151AE7B63D6A42070-46319170-PRCDR), Procedure(id=981F2D079317C4D7CE85003DE2EFE81B-46319170-PRCDR), Procedure(id=DF9172F3F6B0B6CC7F633AB28C04F894-46319170-PRCDR), Procedure(id=2B70A37918AAA2E54240A967896F7B95-46319170-PRCDR), Procedure(id=4C00C4C21FED574151AE7B63D6A42070-46319170-PRCDR), Procedure(id=981F2D079317C4D7CE85003DE2EFE81B-46319170-PRCDR), Procedure(id=DF9172F3F6B0B6CC7F633AB28C04F894-46319170-PRCDR)]

This demonstrates that things seem to be set up properly for execution and that, under normal circumstances, the system can locate ValueSet 20785a and the Procedures it will be asking for in testC.

I then uncomment testC and its dependency, test3. Instead of returning results for test and testA and an empty list (which is what I expect) for testC, I get the following traceback:

org.opencds.cqf.cql.engine.exception.CqlException: java.lang.IllegalArgumentException: Unable to locate ValueSet 20785
at org.opencds.cqf.cql.engine.elm.execution.Executable.evaluate(Executable.java:33)
at org.opencds.cqf.cql.engine.elm.execution.FunctionRefEvaluator.internalEvaluate(FunctionRefEvaluator.java:33)
at org.opencds.cqf.cql.engine.elm.execution.Executable.evaluate(Executable.java:14)
at org.opencds.cqf.cql.engine.elm.execution.ExpressionDefEvaluator.internalEvaluate(ExpressionDefEvaluator.java:19)
at org.opencds.cqf.cql.engine.elm.execution.Executable.evaluate(Executable.java:14)
at org.opencds.cqf.cql.engine.execution.CqlEngine.evaluateExpressions(CqlEngine.java:191)
at org.opencds.cqf.cql.engine.execution.CqlEngine.evaluate(CqlEngine.java:169)
at org.opencds.cqf.cql.engine.execution.CqlEngine.evaluate(CqlEngine.java:148)
at org.opencds.cqf.cql.evaluator.CqlEvaluator.evaluate(CqlEvaluator.java:89)
at org.opencds.cqf.cql.evaluator.CqlEvaluator.evaluate(CqlEvaluator.java:76)
at org.opencds.cqf.cql.evaluator.cli.command.CqlCommand.call(CqlCommand.java:159)
at org.opencds.cqf.cql.evaluator.cli.command.CqlCommand.call(CqlCommand.java:33)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine.execute(CommandLine.java:2078)
at org.opencds.cqf.cql.evaluator.cli.Main.run(Main.java:19)
at org.opencds.cqf.cql.evaluator.cli.Main.main(Main.java:12)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: java.lang.IllegalArgumentException: Unable to locate ValueSet 20785
at org.opencds.cqf.cql.evaluator.engine.terminology.BundleTerminologyProvider.expand(BundleTerminologyProvider.java:87)
at org.opencds.cqf.cql.evaluator.engine.terminology.PrivateCachingTerminologyProviderDecorator.expand(PrivateCachingTerminologyProviderDecorator.java:47)
at org.opencds.cqf.cql.evaluator.engine.terminology.PrivateCachingTerminologyProviderDecorator.in(PrivateCachingTerminologyProviderDecorator.java:25)
at org.opencds.cqf.cql.evaluator.engine.retrieve.BundleRetrieveProvider.anyCodeInValueSet(BundleRetrieveProvider.java:89)
at org.opencds.cqf.cql.evaluator.engine.retrieve.BundleRetrieveProvider.filterByTerminology(BundleRetrieveProvider.java:151)
at org.opencds.cqf.cql.evaluator.engine.retrieve.BundleRetrieveProvider.retrieve(BundleRetrieveProvider.java:56)
at org.opencds.cqf.cql.evaluator.engine.retrieve.PriorityRetrieveProvider.retrieve(PriorityRetrieveProvider.java:28)
at org.opencds.cqf.cql.engine.data.CompositeDataProvider.retrieve(CompositeDataProvider.java:96)
at org.opencds.cqf.cql.engine.elm.execution.RetrieveEvaluator.internalEvaluate(RetrieveEvaluator.java:57)
at org.opencds.cqf.cql.engine.elm.execution.Executable.evaluate(Executable.java:14)
... 28 more

which states that the engine is now unable to locate ValueSet 20785a. This is the same traceback that I got in the previous version of the evaluator (generally speaking) in that the BundleTerminologyProvider seems to throw an IllegalArgumentException in this situation.

There are two potential issues here - either there is something that the evaluator code is expecting and not finding which is not required in a value set, or the bundle file is malformed in some way. As such, I have also attached the valueset bundle file 20785a.json.txt so you can assure yourselves that this is a valid JSON file containing a valid FHIR 4.x Bundle which contains a valid FHIR 4.x ValueSet.

Thanks for looking at this.

For our main project, we've been using a forked version of cql-execution to handle our processing, using CQL-to-ELM translations from clinical_quality_language with .json as our output. From a partner we're working with, we confirmed the results of this arrangement are correct.

We've been looking at cql-evaluator to see if it is faster than the other option. Unfortunately, we trying to use the same CQL, we're getting very different results against the same data. Are there any tips on how to handle the translation? Or is there a way to use XML or JSON directly, skipping the CQL translation?

Acceptance Criteria:
The Clinical Reasoning Repo is migrated to the cqframework organization, all developers are made aware of this, and the following tasks are completed

Task list:

• Add architecture overview
• Add description of modules/packages, determine packages names
• Add coding conventions / style guide
• Import FhirRepository API code from translator

Version 2.0.0 was the last time the generated classes were included in the binary file of Dagger. Something happened between 2.0.0 and 2.1.0.

Interesting enough, the generated classes are present in the -source jars, but not in the binaries, for the last 2 versions, 2.1.0 and 2.1.1.

If you download and unzip the jars, you will find class org.opencds.cqf.cql.evaluator.fhir.DirectoryBundler_Factory only in the source jars and in the 2.0.0 version.

The binary jar for the Dagger module is supposed to be 48KB and not 26KB as it is today.

I cannot replicate this behavior in my local builds. The generated files are always there. It must be something happening in the release publishing machine.

Hi Everyone.
We are trying to evaluate the "BCSE HEDIS MY2022" measure using Cql Evaluator and facing this issue. @brynrhodes @JPercival @jreyno77 can you please look into this?

In the evaluatedResources the link-back extension to the specific criteria reference uses code instead of id.

The current MeaureReportBuilder code is HAPI specific. As there is a significant amount of logic there it'd be nice to have a base class with as much shared logic as is feasible for other FHIR models. Specifically, the IBM FHIR Models.

Need to roll out the last couple of weeks of bugfixes out for a client project. Build and publish a 2.1.1 version of the cql-evaluator. IT should depend on the latest published releases on cql-engine and translator.

The current PlanDefinition implementation uses APIs that not compatible with the JPA server. Need to update the APIs that used to look like those described here:

https://github.com/DBCG/cql-evaluator/discussions/90

This backlog item is blocked, pending the resolution of the design for #114, which may render this implementation obsolete

Needs additional information:

Measures can be defined to include SupplementalDataElements that me either Resources or primitive values. In some cases the primitive values should be aggregated (for example, the number of males across the entire population) for demographic purposes. In other cases we do not want to aggregate the values because they don't have any meaning when aggregated (for example, days since last appointment)

Need to support the ability to load libraries from NPM packages in the evaluator

These two works good:
2019-01-01T00:00:00.0-08:00
2020-01-01T00:00:00.0-08:00
The following two is having exception
2022-01-01T00:00:00.0-08:00
2020-01-01T00:00:00.0-08:00

Git some findings: actually it throws exception when the periodEnd is lesser than the periodStart.
The still outstanding problem is: it is still not working on time level precision when the millisecond part is not set.

The logic currently looks for the word increase/decrease, rather than using the coded improvementNotation concepts (holdover from STU3 when it was only defined as text). Update this code to use the improvementNotation concepts

https://github.com/DBCG/cqf-ruler/blob/master/r4/src/main/java/org/opencds/cqf/r4/providers/MeasureOperationsProvider.java#L302

The cql-evaluator needs documentation around the APIs a platform needs to implement in order to support CQL execution and the FHIR operations. These are roughly:

1. FHIR Resource Crud
2. FHIR Searching/Batching/Transactions
3. Terminology Services
4. CQL Services (Caching / Configuration)

Additionally, specific operations could have their own requirements, like job scheduling for PlanDefinition/ActivityDefinition $apply.

Repro:

• clone this repo: https://github.com/HL7/davinci-ra.git
• checkout this commit: 181842bc
• In VS Code, open the ThunderClient extension
  • update the Env to point to your local CQF Ruler instance (http://localhost:8080/fhir/)
  • in Collections, open $risk-adjustment
  • in Config, run Knowledge Artifacts - HCC189 - Historic to load knowledge artifacts
  • in Config, run Test Data - HCC189 - Historic to load data
  • run Evaluate HCC189 - Historic to generate a MR
  • note that the MR does not include any SDEs

I am trying to run some test and not able to find - org.opencds.cqf.cql.evaluator.dagger.DaggerCqlEvaluatorComponent.

"Inject prefetch resources some other way, like parameters in CQL", but that breaks a whole bunch of ecosystem stuff.
null = unknown empty
empty = known empty
otherwise = known non-empty.

That obviously has potentially far-reaching implications for engine behavior. But I think a first pass is not too bad.
Null dataProvider result =
new CqlException("DataProvider was unable to determine if retrieve XXX was null or empty. Please ensure the evaluation environment has all data necessary to make that determination")

Then in the context of prefetch, we can fall back. In the context of using a REST provider directly, something went horribly wrong.

Given an Observation.subject reference like this in a Bundle:

"subject": {
  "reference": "urn:uuid:0ae954fa-eff8-0d38-c840-3af401abc9fd"
},

The evaluator/engine doesn't retrieve the Observation. The FileDataProvider should probably ignore the urn:uuid: scheme just like the bundle retrieve loader does for the patient id.

mvn package fails for the latest evaluator code due to using some APIs that aren't supported on Android. We need to fix those API usages and ensure all the checks pass.

Given the following library:

library QuestionnaireExtraction

using FHIR version '4.0.1'

include FHIRHelpers version '4.0.1'

context Patient

define QuestionnaireResponses:
  [QuestionnaireResponse]

Even with an appropriate test case, the result of the QuestionnaireResponses expression is an empty list.

When I tried to run help on evaluator.cli, got the following message:

no main manifest attribute, in ./evaluator.cli/target/evaluator.cli-1.3.1-SNAPSHOT.jar

When accessing a bundle directly, if the fullUrl is a urn, the loaded resource has an "id" that includes the prefix, which means the provider resolution doesn't see it.

This is related to HAPI issues here:
https://github.com/hapifhir/hapi-fhir/blob/cbb16ce3affd3fc53dcbfe98dd3181644fe68604/hapi-fhir-base/src/main/java/ca/uhn/fhir/parser/JsonParser.java#L639

hapifhir/hapi-fhir#1868
hapifhir/hapi-fhir#2057

Unexpanded valuesets should be alerted with a "WARN" status
Naively expanded valuesets should be alerted with "INFO" status

Consider the following expression:

define TestQuestionnaireResponse:
  QuestionnaireResponse {
    "id": id('phq-9-questionnaireresponse'),
    "questionnaire": canonical('http://somewhere.org/fhir/uv/mycontentig/Questionnaire/phq-9-questionnaire'),
    "status": QuestionnaireResponseStatus('completed'),
    "subject": Reference {
      "reference": string('Patient/example')
    },
    "authored": dateTime(@2021-09-13T16:29:00-07:00),
    "item": {
      FHIR.QuestionnaireResponse.Item {
        "linkId": string('LittleInterest'),
        "text": string('Little interest or pleasure in doing things'),
        "answer": {
          FHIR.QuestionnaireResponse.Item.Answer {
            "value": Coding {
              "system": uri('http://loinc.org'),
              "code": code('LA6568-5'),
              "display": string('Not at all')
            }
          }
        }
      },
      FHIR.QuestionnaireResponse.Item {
        "linkId": string('TotalScore'),
        "text": string('Total score'),
        "answer": {
          FHIR.QuestionnaireResponse.Item.Answer {
            "value": integer(3)
          }
        }
      }
    }
  }

Although this translates without error, it cannot be invoked because the component types cannot be resolved by the FHIR package

Consider this CQL:

define "DateTime in GMT":
    DateTime(2022, 8, 8, 1, 30, 0, 0, 0)

define "FHIR DateTime in GMT":
    FHIR.dateTime { value: "DateTime in GMT" }

define "To Date":
    Date(year from "DateTime in GMT", month from "DateTime in GMT", day from "DateTime in GMT")

define "To Date from FHIR":
    Date(year from "FHIR DateTime in GMT", month from "FHIR DateTime in GMT", day from "FHIR DateTime in GMT")

The output of this code is:

DateTime in GMT=2022-08-08T01:30:00.000
To Date=2022-08-08
FHIR DateTime in GMT=dateTime
To Date from FHIR=2022-08-07

I ran this from a computer set to EST.

It appears that when constructing the FHIR.dateTime in CQL, the evaluator is converting the GMT dateTime to EST. This is inconsistent with constructed system dateTime types.

Add the following test cases:

5 '{inr}' = 5 '{INR}' // true
5 '{foo}' = 5 '{INR}' // true on the grounds that annotations are meaningless

The MeasureEvaluation class in the evaluator.measure module does an excellent job of staying model agnostic all the way up to the point where it links some of the parameterized types to HAPI's IBase interface. In order to implement this for a different model (e.g. the IBM FHIR model objects), I would need for that base class to either be parameterized or just left out entirely. It would also be helpful to have the implementations in a separate module from the common classes so that consumers don't pick up a stray HAPI dependency.

I tried to have a read through of the pom.xml file and it seems that there are some server-side dependencies that would make this not run on Android. Would it be possible to get a version of this lib running on Android?

https://github.com/DBCG/cql-evaluator/blob/master/pom.xml

This will be highly appreciated

Thanks

Add an operation to the CQL Evaluator that for a given library will return test case definitions.

CQL Unit Testing · Issue #40 · DBCG/cql-language-server

Acceptance Criteria:
The cql-evaluator has an operation that can return a set of test-case definitions from a CQL Test library as described in the linked ticket. Each test case should have a primary library, context set, input parameters, and the path to the source for data, terminology, and content as applicable per test case. I should be able to invoke this operation from the command-line and get JSON structures being output.

Steps:

1. The cql-evaluator currently only supports one operation. Need to expand that framework to support multiple operations.
2. The TestCase structure should have the same input parameters as the current CQL operation. Each test case is a “run” of that operation that may result in a failure.
3. the cql-evaluator needs to translate the CQL and read the “test” tags described above. That creates a “TagSet”. There’s work in the cql-translator to start that. Needs to visit the ELM graph and collect those.
4. Once we have a TagSet, we need to interpret those into TestCases. The semantics of the tags are such that tags declared at the Library level apply to all test cases, while tags at the definition level apply to specific test cases and override the library-level tags
5. Write the test cases out as JSON.

The CQL/Measure runtime can query resources from a data source (pre-existing or "persistent" resources) OR create resources on the fly as evaluation happens (instance or "created" resources). The former should show up as evaluatedResources in a MeasureReport while the latter should not. An example of a "created" Resource is an Observation created from an SDE coding.

When evaluating an expression that references a Quantity parameter, the code that constructs the parameter declaration does not qualify the type of the parameter, so it is incorrectly resolved to a FHIR.Quantity when the evaluator is passing a System.Quantity:

library expression version '1.0.0'
using FHIR version '4.0.1'
include FHIRHelpers version '4.0.1' called FHIRHelpers
parameter "%normalReportingDuration" Quantity
parameter "%encounter" Encounter
define "return":
       %encounter.where((status = 'in-progress' and period.start + %normalReportingDuration >= Now()) or status = 'finished' and period.end <= Now() - 72 hours)

The issue is that the CqlParameterDefinition uses the unqualified type name.

Currently, the population criteria must return a type consistent with the population basis, which is only allowed to be a FHIR type.
We can consider an option to return a non-FHIR type such as Tuple.

Also there needs to be another issue on the evaluator to correct the issue with non-patient-based measures not computing intersection of results

Measure evaluation doesn't currently have a way to bubble up validations to the report building components. Need a way to do that, such as a list of "ValidationMessages" on the returned MeasureDef. These can then be mapped in a data model specific way by the Report builders.

The first validation to add is that the denominator is a subset of the initial population.

Given some cql like the following:

using FHIR version '4.0.1'

define "New Patient":
   FHIR.Patient { id : "123" }

The new "123" Patient should show up as a contained Resource in a MeasureReport

As a processor of SDE extensions I need to be able to tie the SDE extension back to the SDE definition in the Measure.

The contained SDE entries include an extension that addresses this, but by tying it back to the CQL Expression. Ideally both contained SDEs and Resource SDEs would link back to the Measure.

This effort requires design. Implementation is blocked for said design.

Measure evaluation in the CQL Evaluator generally requires measure to conform to the CQF measures profile. Add validation for measure resources and report to use any validation errors.

More context: The CQF Measures IG specifies certain requirements that are more stringent for Measure resources than the base FHIR spec. In particular, certain elements like Groups, Stratifiers, and SDEs require unique "ids" for each element. For example, an SDE that collects Race demographic information may have the id "sde-race", the initial population for the first group element may have the id "group-1-initial-pop", etc. It's not necessary that the ids be human readable, but it is useful. The ids are required to support certain extensions in the MeasureReport, which tie Resources and other results back to the specific elements of the Measure resource that created or evaluated those. For example, if the InitialPopulation referenced several Encounters, each Encounter in the evaluatedResources section of the MeasureReport would have an extension that was like { URL : "sourceCriteria", value: "group-1-initial-pop" } and so on.

Acceptance Criteria:
Should be optional.
Should use the actual cqf-measures profile definition to stay in sync as a default. Should be configurable to use other profiles as needed.
Should validate as first step of evaluation.
Should return a reasonable error if profile is not matched

If evaluation fails, the entire operation should return an operationoutcome
If evaluation succeeds, then the MeasureReport should contain a contained OperationOutcome with any warnings/messages generated by the evaluation, consistent with the proposed specification change here: https://jira.hl7.org/browse/FHIR-37774

Correct Behavior:

MeasureReport A {
   Group 1 = 8
   Group 2 = 3
}

+

MeasureReport B {
   Group 2 = 8
   Group 3 = 5
}

=

MeasureReport C {
   Group 1 = 8   <- Included from A
   Group 2 = 11  <- Included from A and B
   Group 3 = 5   <- Includes from B
}

Current Behavior:

Current Behavior:
MeasureReport A {
   Group 1 = 8
   Group 2 = 3
}

+

MeasureReport B {
   Group 2 = 8
   Group 3 = 5
}

=

MeasureReport C {
   Group 1 = 8   <- Included from A
   Group 2 = 11  <- Included from A and B
   Group 3 = MISSING! Wasn't present in A so it wasn't added! 
}

When running current "population" reportType $evaluate-measure operation SDE is not clearing unaggregated items.

When evaluating PlanDefinition/$apply, ActivityDefinition/$apply, Library/$evaluate, and Measure/$evaluate-measure, log the URL and version of the artifact that is being evaluated.

Update evaluator to expose multiple subjects overload of evaluate-measure

This is needed for use with both Spark DQMP and CDR DQMP to increase performance of batch jobs. This will allow us to not have to iterate through measures one by one in parallel

Acceptance Criteria:
The subjectids list overload of the evaluator is available to use.

The PlanDefinition and ActivityDefinition processors are passing 'null' as the context when evaluating FhirPath:

https://github.com/DBCG/cql-evaluator/blob/3c35ddab0cee8d26f86f0d161c4eee73b44de6b8/evaluator.plandefinition/src/main/java/org/opencds/cqf/cql/evaluator/plandefinition/r4/PlanDefinitionProcessor.java#L695

However, PlanDefinition, ActivityDefinition, and Measure resources are knowledge artifacts that are specifically set up to evaluate in the context specified by the subjectType element (typically Patient).

The cql-evaluator needs to be able to load subjects via the new FhirRepository APIs