linuxforhealth / fhir Goto Github PK

Is your feature request related to a problem? Please describe.
Document the setup and configuration of DB2 on Cloud as a persistence provider for FHIR R4.

Describe the solution you'd like
A markdown document

Describe alternatives you've considered
Is there anything better than markdown?

Additional context
N/A

Create an @binding annotation on model classes that facilitates conformance validation at runtime.

The fhir-benchmark project brings in Saxon as a dependency which closes the XMLStreamReader when it's finished writing Narrative XHTML content. We need to implement a non-closing XMLStreamWriter delegate to avoid this scenario.

This epic covers user stories related to profiling of the FHIR specification.

Enable Travis for the FHIR Repository

Create .travis.yml

###############################################################################
# (C) Copyright IBM Corp. 2019
#
# SPDX-License-Identifier: Apache-2.0
###############################################################################

# Notes: 
# SKIPPING BUILD via Travis: 
# To skip the build use: 
# [skip <KEYWORD>]
# ci, travis, travis ci, travis-ci, or travisci
# Example: [skip travis] Update README

dist: xenial
sudo: false

language: java

# use Open JDK 8
jdk:
  - openjdk8

install:
  - java -version

matrix: 
  fast_finish: true

# Using a depth of false, we don't really modify git. 
git:
  depth: false
  quiet: true
  submodules: false

before_install: 
  - echo "MAVEN_OPTS='-Xmx2G -Xms1G -XX:MaxPermSize=512m -Djava.awt.headless=true -B'" > ~/.mavenrc

# Generate the VERSION - TRAVIS_BUILD_ID
# fhir-parent-it = Integration test
script: 
  - env
  - df
  - ulimit -a
  # fhir-tools - installs the generator
  - "mvn -f fhir-tools/pom.xml -B -P jenkins clean install -Dmaven.javadoc.skip=true"
  # fhir-examples = Exampes Data
  - "mvn -f fhir-examples/pom.xml -B -P jenkins clean install -Dmaven.javadoc.skip=true"
  # fhir-parent-ut = Unit Test
  - "mvn  -f fhir-parent/pom.xml -B -P jenkins clean install -Dcom.ibm.watsonhealth.fhir.model.spec.test.R4ExamplesDriver.testType=MINIMAL"

# cache the build files
cache:
  directories:
  - $HOME/.m2

notifications:
  email:
    recipients:
      - [email protected], [email protected]
    on_success: never # default: change
    on_failure: always # default: always  

Update ci/license/issuemanagement

Uses tag-release on push tag

Changes the mvn version (does the modification on the tag)

Travis.txt

ci for push build

ci for pr build

ci for tag build and release

Protect Branches

Update travis / bintray

we could do something like this...
git log <PRIOR_TAG>...<CURRENT_TAG> --name-only - if files changed since prior, then build the project, and release only the files which match the name changes

Describe the bug
Under certain circumstances, the server will return a 500 error and there will be no exception details in the server logs.

To Reproduce

1. Implement a fake AuditLogger and configure the FHIR Server to use it.
2. Have it throw an exception when you call the logX methods
3. Issue a request and note the 500 error response
4. Check the logs and note there is no stacktrace

Expected behavior
All 500 errors should result in an exception in the server logs.

Additional context
We always had this issue, but since we weren't doing much of anything in the finally block, we never hit it. We recently moved audit logging to the finally block at the end of the interactions and this has widened

I suggest we move the log statement to the REST interaction methods (create, update, read, etc. instead of the doCreate, doUpdate, doRead variants).

Update build for OpenJDK 11

Extra EAR projects are created needlessly on Maven Project import

Warning in fhir-persistence-jdbc is reported in the build.

Aug 30, 2019 2:31:12 PM com.ibm.watson.health.fhir.search.parameters.ParametersUtil populateSearchParameterMapFromStreamByFormat
WARNING: Error condition reading the FHIR Bundle of Search Parameters -> from_stream

Update the server.xml to reflect the OpenSource pacakage name in the trace specification.

<logging traceSpecification="=info:com.ibm.watsonhealth.fhir.=finer" traceFormat="BASIC"/>

<logging traceSpecification="=info:com.ibm.watson.health.fhir.=finer" traceFormat="BASIC"/>

Refine $import, $export and $export-status operations

• confirm the multi-tenancy
• resolve TODO
• refactor to support DELETE operations
• push batch code down to the library for bulkdata and pull into the Operation

Reference - https://javaee.github.io/tutorial/batch-processing006.html#BCGCAHCB

 <dependency>
            <groupId>javax.batch</groupId>
            <artifactId>javax.batch-api</artifactId>
            <version>1.0.1</version>
        </dependency>


 JobOperator jobOperator = BatchRuntime.getJobOperator();
        jobOperator.start("FhirBulkExportChunkJob", toProperties(properties));


 private Properties toProperties(Map<String,String> props){
        Properties properties = new Properties();
        for(Entry<String, String> entry : props.entrySet()) {
            properties.put(entry.getKey(), entry.getValue());
        }
        
        return properties;
    }

Includes enhancements needed to be compliant with the Spec.

https://build.fhir.org/ig/HL7/bulk-data/export/index.html

As parting of our effort to reduce the total number of projects.

The current export javabatch can only support the system level export of the spec.

We need to support patient export - export all patients' related resources.
https://build.fhir.org/ig/HL7/bulk-data/OperationDefinition-patient-export.html

Steps

1. Update the API $export Operation to accept the submission pattern from client for patient export with parameters:
   (1) -since (optional, if without, then export for all patient)
   (2) -format (optional, only support ndjson)
   (3) -resourceTypes (optional, if no resource Type, then the job will export all resourcetypes for patient)

2. Update the API $export Operation to call the patient export java batch job and pass in the parameters.

3. Create new java batch job for patient export
   (1) Take the parameters from patient export operation.
   (2) Construct the to-be-exported resource types - resource types parsed from the passed-in parameter or all compartment resource types of Patient.
   (3) For each resource type
   (a) Search and get a page of patients, for each found patient:
   -- compartment search using patient id to get the resources.
   -- write the found resources into buffer stream.
   (b) If checkpoint satisfied, then write the buffered resources to COS bucket and append the exported file info to batch job exit status.
   checkpoint algorithm: page of patients per exported file or by file size.
   (4) After all resource types are processed, exit the batch job.

4. Update the API $export Operation to parse the job exit status and construct links of exported resource type files and response to client.

Describe the bug
When _outputFormat is submitted as a query parameter, the UriInfo injected in FHIRResource replaces the "+" char with a space.

This is the same issue we hit with the FHIR "_format" query parameter, but in a different part of the code.

To Reproduce
{{http}}://{{host}}:{{port}}/{{basePath}}/$export?_outputFormat=application/fhir+ndjson&_since=2019-01-01T08:21:26.94-04:00&_type=Patient

Expected behavior
Not sure if its liberty or JAX-RS or CXF at fault here, but we need to work around the issue for this to work.

We need to figure out a way to support multiple resource types inside the javabatch job or we will have to ask the operation codes to send multiple job requests - one for each resource type and then consolidate the results.

we need to support group-export per the spec.

https://build.fhir.org/ig/HL7/bulk-data/OperationDefinition-group-export.html

Steps

1. Update the API $export Operation to accept the submission pattern from client for patient export with parameters:
   (1) -since (optional, if without, then export for all patient)
   (2) -format (optional, only support ndjson)
   (3) -resourceTypes (optional, if no resource Type, then the job will export all resourcetypes for patient)
   (4) -group (mandatory)

2. Update the API $export Operation to call the group export java batch job and pass in the parameters.

3. Create new java batch job for group export
   (1) Take the parameters from patient export operation.
   (2) Construct the to-be-exported resource types - resource types parsed from the passed-in parameter or all compartment resource types of Patient.
   (3) For each resource type
   (a) Search and get the group via /Group/grpID.
   (b) get one page of patients from the group member, and for each found patient:
   -- compartment search using patient id to get the resources.
   -- write the found resources into buffer stream.
   (c) If checkpoint satisfied, then write the buffered resources to COS bucket and append the exported file info to batch job exit status.
   checkpoint algorithm: page of patients per exported file or by file size.
   (4) After all resource types are processed, exit the batch job.

4. Update the API $export Operation to parse the job exit status and construct links of exported resource type files and response to client.

During a review of the codebase on August 30, 2019, we identified an out-of-date and unused file with key iv used for ondisk encryption of logs. The file is fhir-audit/src/main/resources/config.properties

There are no running instances which use the library that use this configuration file specific to audit.
There is no compilable code which uses the config.properties.

The team has opted to remove the unused file.

"output": [
    {
        "type": "Practitioner",
        "url": "http://serverpath2/practitioner_file_1.ndjson",
        "count": 10000
    },

....

During Connectathon, we found the job submission was very slow, from the logs, looks like the job submission response was only returned after the job was actually finished. We hope separating job dispatch and executor can make the job submission faster.

Bulk Export/Import $export-status operation support for Delete
The operation should stop the JavaBatch Job.

Input

• After a JavaBatch job is submitted, there is a response which includes the ID.
• Validate the Tenant Id for the Job
• Call the $export-status operation and validate the tenant

Output
400 - Bad Request
404 - Job Execution not found
403 - forbidden access to job (beyond normal 403)
500 - Internal error

Steps

• Use the Java BatchRuntime to access and stop the Job.

Alternative, URL to stop the above job is:
https://localhost:9443/ibm/api/batch/jobexecutions/1?action=stop

Related to Issue #55

Create Examples of Using Builders

Implement $import and $export operations so the JavaBatch framework can be added to the operation.

Query Parameters

• _outputFormat
• _since
• _type

Response
In-Progress
Error
Complete

Reference

• https://build.fhir.org/ig/HL7/bulk-data/export/index.html
• https://www.hl7.org/documentcenter/public_temp_45E1557F-1C23-BA17-0CF72AF5739BC769/calendarofevents/himss/2018/HL7%20FHIR%20Bulk%20Data%20API.pdf

FHIRServerUserGuide is where we document how to download, install, and run the Watson Health FHIR Server. This needs a serious overhaul for the R4 server.

Create a registry that holds conformance resources (e.g. structure definitions, value sets, etc.) that other components can use for various purposes.

Describe the bug
Currently, the generator creates parameters like this:

        "/RiskAssessment": {
            "get": {
                "tags": [
                    "RiskAssessment"
                ],
                "summary": "Search for RiskAssessment resources",
                "operationId": "searchRiskAssessment",
                "produces": [
                    "application/fhir+json"
                ],
                "parameters": [
                    {
                        "$ref": "#/parameters/_idParam"
                    },
                    ...
                    {
                        "$ref": "#/parameters/_idParam"
                    },

To Reproduce
Run FHIRSwaggerGenerator for search operations

Additional context
We recently changed the behavior of FHIRUtil.getSearchParameters so that it will return all the search parameters for a given resource type, including the ones inhereted from the generic Resource definition.
We still need to document that change (or revert it), but for now we just need to fix the generator.

$apply is a FHIR Operation which captures the current subject, and generates a CarePlan relative to the current subject using the PlanDefinition. $apply is idempotent, and generates the same results over-and-over again based on the subjects data.

URLs
URL: [base]/PlanDefinition/$apply
URL: [base]/PlanDefinition/[id]/$apply

The Operation definition includes many specifics on the $apply function and must be implemented with corresponding tests.

Reference
Plan Definition - http://hl7.org/fhir/plandefinition.html
Operation - http://hl7.org/fhir/plandefinition-operation-apply.html
Connectathon Track - https://confluence.hl7.org/display/FHIR/2019-09+Care+Planning+and+Management+Track

This epic covers user stories related to documentation of the FHIR server and its components.

The FHIRSwaggerGenerator for R4 still has some issues:

• looks for XMLElement annotation even though we don't use that any more
• only adds inhereted properties when the parent is a resource
• misses all enum due to using wrong delimiter for nested classes
• missing logic for handling Boolean
• choice elements get serialized without their typeSuffices
• object definitions are no longer scoped by their container classes

Describe the bug
DB2 stored procedure resource files have no content after copyright update.

To Reproduce
Try to deploy the schema.

Expected behavior
Schema should deploy successfully.

Additional context
N/A

We should either revive the tests under com.ibm.watson.health.fhir.persistence.jdbc.search.test or create something comparable.
The idea is to systematically define search parameter definitions for each combination of SearchParameter type and target element type.

Perhaps some of the other tests are worth reviving as well. If not, then they should be deleted.

Acceptance criteria 1: Unit tests (under either fhir-search or fhir-persistence or both) cover search parameters of each search parameter type and each fhir data type.
Acceptance criteria 2: Unit tests that are not executed are deleted (or have a clear comment about why they are still needed but aren't included in the build)

Refactor $apply for PlanDefinition/CarePlan based on Feedback

• Create sample using Obesity Assessment Protocol where there is a measure of BMI http://hl7.org/fhir/plandefinition-protocol-example.json.html
• Address comments from Lee - #42

This will verify if the javabatch bulkexportimport job is compatible with Amazon S3 or not. if not, then make necessary code changes as needed.

The FHIR definition for each fields consists of:

• Definition
• Binding
• Type
• Comments
• Invariants

Today only the first 3 are apparent from the API. Invariants are captured as annotations on the model classes, but that is hidden during normal usage.

Often times the comments are redundant with the Definition and/or Invariants, but for completeness I think we should be adding all 3 to the corresponding setter methods on the Builders.

Connectathon Issues with Bulkdata

Conformance.md is where we document our conformance to the FHIR DSTU2 specification. We need to update that to cover where we're at on the R4 implementation.

fhir-install missing bulkimportexport-webapp yet has specified in assembly.

• add dependency

        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>fhir-bulkimportexport-webapp</artifactId>
            <version>${project.version}</version>
            <type>war</type>
        </dependency>

• check to see other dependencies are specified.

fhir-coverage-reports missing code coverage dependencies

• add dependencies

        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>fhir-task-api</artifactId>
            <version>${project.version}</version>
        </dependency>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>fhir-task-core</artifactId>
            <version>${project.version}</version>
        </dependency>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>fhir-database-utils</artifactId>
            <version>${project.version}</version>
        </dependency>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>fhir-operation-bulkdata</artifactId>
            <version>${project.version}</version>
        </dependency>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>fhir-operation-apply</artifactId>
            <version>${project.version}</version>
        </dependency>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>fhir-bulkimportexport</artifactId>
            <version>${project.version}</version>
        </dependency>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>fhir-conformance</artifactId>
            <version>${project.version}</version>
        </dependency>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>fhir-client</artifactId>
            <version>${project.version}</version>
        </dependency>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>fhir-cli</artifactId>
            <version>${project.version}</version>
        </dependency>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>fhir-persistence-schema</artifactId>
            <version>${project.version}</version>
        </dependency>

fhir-task-core reference invalid version

• update 4.0.0 to 4.0.0-SNAPSHOT

    <parent>
        <groupId>com.ibm.watson.health</groupId>
        <artifactId>fhir-parent</artifactId>
        <version>4.0.0</version>
        <relativePath>../fhir-parent</relativePath>
    </parent>

fhir-bulkimportexport update bulkimportexport

• add exclusion for derbytools

        <dependency>
            <groupId>com.ibm.watson.health</groupId>
            <artifactId>fhir-persistence-jdbc</artifactId>
            <version>4.0.0-SNAPSHOT</version>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.derby</groupId>
                    <artifactId>derbytools</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

• put in dependency management

        <dependency>
            <groupId>com.ibm.cos</groupId>
            <artifactId>ibm-cos-java-sdk</artifactId>
            <version>2.1.0</version>
        </dependency>
        <dependency>
            <groupId>javax.batch</groupId>
            <artifactId>javax.batch-api</artifactId>
            <version>1.0</version>
        </dependency>
        <dependency>
            <groupId>javax.inject</groupId>
            <artifactId>javax.inject</artifactId>
            <version>1</version>
        </dependency>

• change inject to

<dependency>
    <groupId>jakarta.inject</groupId>
    <artifactId>jakarta.inject-api</artifactId>
    <version>1.0</version>
</dependency>

• change batch api

<dependency>
    <groupId>jakarta.batch</groupId>
    <artifactId>jakarta.batch-api</artifactId>
    <version>1.0.2</version>
</dependency>

Rough Dependency Hierarchy

This epic covers user stories related to performance of the FHIR server and its components.

Work should be timeboxed and guided by needs of upcoming announcement.
Numbers should be repeatable.

For persistence of the javabatch jobs, DB2 will be used per instruction here https://www-01.ibm.com/support/docview.wss?uid=tss1wp102544&aid=11 page 14 and page 17

Is your feature request related to a problem? Please describe.
We're running the FHIR Server behind a proxy and that proxy modifies the request URLs.

Request come in to an endpoint like https://[host]/wh-fhir-dev/api/v4 but the server is returning URIs like https://[host]/fhir-server/api/v4/$export-status?job=9.

Is there a way to use the origin request URL in the server URLs returned by the server?

I think it affects at least:

• search result self uri
• Bundle entry fullUrl values
• bulkimportexport polling URLs

Describe the solution you'd like
https://[host]/wh-fhir-dev/api/v4
Either:

• a setting the server will use as the "base" in all self uris; or
• a way to tell the server which HTTP header has the original request URI (so it can use that to construct self uris)

Describe the bug
While trying to invoke $export operation via GET, it complains that we can't process an input parameter of type instant.

To Reproduce
Invoke export with a request like this: {{http}}://{{host}}:{{port}}/{{basePath}}/$export?_outputFormat=application/fhir+ndjson&_since=2019-01-01T08:21:26.94-04:00&_type=Patient

Additional context
Spec says we should support passing of "primitive" types via URL.

On startup, the userlib is required for derbySproc. This hard requirement is no longer valid as the derby-sproc is no longer used in R4.

The derby bootstrap (in FHIR Servlet Context Listener) checks for userlib in the server and stops startup if the bootstrap fails.

We used to have multiple relational schemas:

1. a "basic" schema that puts all resource types into the same tables
2. a "normalized" schema that uses a set of tables per resource type

At the implementation level, the normalized schema extends the basic schema, which makes things more complicated than they need to be. We never updated the "basic" schema for R4 and so this is needless complexity.

Acceptance criteria 1: Class hierarchy in fhir-persistence-jdbc should be simplified and without reference to the basic schema
Acceptance criteria 2: Configuration and documentation related to the "basic" schema should be removed