kids-first / kf-api-fhir-service Goto Github PK

Add the Github workflow from https://github.com/d3b-center/d3b-release-maker to this repository so that the release process is automated.

Since others may start to use the Smile CDR server and the utility scripts in this repo, we should add some tests to make sure things work.

This adds CircleCI pipeline to test new smile CDR deploys and server settings updates. Test will be very simple to start - load KF FHIR model, example data, and curl a FHIR endpoint. Later on the simple FHIR tests will be replaced with integration tests that are developed as a part of the kf-model-fhir repo.

Later on a Jenkins pipeline will be added so that the FHIR service is deployed to the standard AWS KF environments with the standard ecs-service-type-1 infrastructure. It is still important to keep the CircleCI pipeline since other developers outside of KF (without access to Jenkins) can use this repo with CI in place.

Per https://smilecdr.freshdesk.com/support/tickets/287, it is best to update server settings using the admin endpoint. The properties file by iteself is only used to initialize the server when it is first started.

This script should read in the properties file and send the appropriate requests to update the server settings and restart the necessary server modules.

Filing here for awareness. This ticket is still open. https://smilecdr.freshdesk.com/support/tickets/294

We’re having trouble searching for StructureDefinitions by the valueset search parameter.

First query http://10.10.1.191:8000/StructureDefinition/Patient to verify that it’s snapshot has an element with a binding to the languages ValueSet
Next query http://10.10.1.191:8000/ValueSet/languages to verify that the value set exists
Last try the query we really want: http://10.10.1.191:8000/StructureDefinition/Patient?valueset=languages

That last query always returns 0 results. My understanding of the valueset search parameter is that it searches the list of elements in the resource snapshot, and for any element that has a binding, it will check to see if that binding’s valueset matches the one in the query.

Add a line to the modeling loading script to force re-indexing after search parameters are loaded into the server.

POST /$mark-all-resources-for-reindexing

Production settings that are not secrets should be written into the prod docker image so that they can be controlled from this repository. Right now if you make changes to prod.env, they don't take effect in the deployment, since the deployment pipeline sets environment variables using .env files stored in S3.

• Write non-secrets in server/settings/prod.env as ENV in Dockerfile
• Remove these settings from the .env files on S3
• Remove server/settings/prod.env

• Implement #38
• Implement #40
• Implement #48
• Implement #49
• Implement #71

Optional

• Setup reverse proxies for FHIR API, Admin API, Admin Dashboard and test it with nginx
• Test out Smile CDR's APM service for application perf monitoring (?)

Smile CDR is working on a fix for the /$snapshot endpoint which does not currently work. When the new image arrives, we should upgrade to it.

Small ontologies can be imported via CodeSystem FHIR resources but for larger ontologies such as HPO, this is not feasible. We should figure out how to use an external terminology server to host large CodeSystems such as HPO.

An external terminology server can be registered in Smile CDR: https://smilecdr.com/docs/fhir_repository/terminology.html. We should try registering and using the Ontoserver: https://r4.ontoserver.csiro.au/fhir

Part of d3b-center/clinical-data-flow#37

In order to deploy the smile CDR server to production we will need to secure it in several ways. Right now all smile endpoint ports have inbound rules in the firewall and none of the requests are encrypted.

We should setup a reverse proxy from ports 80, 443 to all smile CDR endpoints so that we can remove inbound rules for all ports except 80 and 443 in firewall. The reverse proxy will also make it easier to remember all of the endpoints (<host>/fhir_api is easier to remember than <host>:8000) Then we can setup TLS on the reverse proxy so all http traffic coming into the internal network is encrypted.

(We could also encrypt all traffic from the reverse proxy to smile CDR since smile CDR implements TLS but this may be overkill for an internal data service)

In preparation for this, we can add NGINX to the docker stack and test smile CDR to make sure everything works as expected with the reverse proxy and TLS.

1. Add NGINX to docker stack

• Remove inbound firewall rules for all http ports except 80 and 443
• Setup reverse proxy (443 -> 9000, 9100, 8000, 8001) for all smile CDR endpoints
• Secure nginx endpoint with TLS (1.2 or 1.3 not sure yet)

2. Update appropriate configuration (respect forward headers) on smile CDR

3. Test FHIR data dashboard and FHIR ingest

We need to do this as a work around to #35. Update the FHIR mappers to point to the KF base profile (e.g. kfdrc-patient not kfdrc-patient-no-phi) and then re-load the data.

This is likely a bug in Smile CDR. Issue filed here: https://smilecdr.freshdesk.com/support/tickets/364

Details

• A StructureDefinition (e.g. kfdrc-patient) which extends a FHIR base StructureDefinition (e.g. Patient) seems to have a valid snapshot that was generated (verified using /StructureDefinition/kfdrc-patient/$snaphost)
• A StructureDefinition (e.g. kfdrc-patient-no-phi) which extends kfdrc-patient does not have a snapshot (verified using /StructureDefinition/kfdrc-patient-no-phi/$snaphost)

Replace Phenopackets model with Kids First model

• Replace temporary secrets with real ones and re-deploy
• Turn on validation after deploy via admin endpoint
• Grant anonymous user CRUD access to all non-conformance FHIR resources
• Create users for team members

We should add https://github.com/kids-first/kf-ui-fhir-data-dashboard to the docker-compose stack so that it can be deployed along with the server. This will make it much easier for users to browse through data in their FHIR server.

When a SearchParameter is submitted to the server, the server must re-index all of the resources for the parameter before a query using that search parameter can work.

The re-indexing can take several minutes and there is no reliable way to tell when the server is finished with re-indexing. Right now the logs must be inspected to determine when indexing finishes. When there are no more log statements saying 0 concepts to index, then the server has complete indexing.

Smile CDR tech rep says that Smile CDR supports Postgres version 11.4 but the server doesn't recognize the setting POSTGRES_11_4 in the property file.

Current server is using the Postgres default: version 9.4, but we would like to use a version 11.

Ticket submitted here: https://smilecdr.freshdesk.com/support/tickets/367

https://smilecdr.com/docs/configuration_categories/fhir_performance.html

cc: @abgeorge7 Turns out smile cdr does support this!

Our production deployment will require the Smile CDR image to be in its own repo on ECR. We need to update the Dockerfile to use this image rather than the one hosted on Dockerhub.

The compose stack is for local development and doesn't need to use the production docker image so we will also need to update the docker-compose file to use the image on Dockerhub rather than the image built from the Dockerfile.

The ecs-service-type-1 deployment creates the service's database as part of the service deployment. It also writes the DB secrets to an database.env file in the appropriate S3 bucket.

The Smile CDR properties file, docker-compose.yml, and dev.env files should same env var names for DB secrets so that these vars can be sourced from the ecs-service-type-1 generated database.env file.

This setting allows HTTP requests to be logged in the main Smile CDR log.
https://smilecdr.com/docs/product_configuration/http_server_setup.html#access-logs

This should be done for both dev and production since it is useful for both debugging and auditing.

It is possible to substitute server property values with environment variables: https://smilecdr.com/docs/installation/installing_smile_cdr.html#variable-substitution.

We should update the properties file to use the variables defined in smilecdr/.env so that they are defined in one place rather than two (.env and properties file).

The smile CDR server is currently setup with basic authentication using the internal smile CDR users database.

We eventually want to augment this setup (or move away from it entirely) with an OIDC setup where smile CDR doesn't have to authenticate users itself. Users should be able to use Google to sign in.

Integrating with Auth0 will allow us to do this (among other useful things) and make user management a lot easier. This would also setup smile CDR so that authorization (RBAC) could also be managed with Auth0 rather than smile CDR.

Create a version of the Smile CDR server (docker image) suitable for FHIR model integration tests. This server will be spun up for use during integration tests of kf-model-fhir.

The integration test server will use the embedded H2 database instead of an external Postgres database and should have all of the base FHIR model loaded in it so that server startup is minimal.

The integration test server image will be stored in ECR and Dockerhub and will be updated whenever a new release of Smile CDR comes out or the test target in the Dockerfile changes.

Supposedly Smile CDR supports Postgres 11. If this is the case, we should upgrade to it. We are currently on the Postgres 9.4 which is the default Postgres version that Smile CDR supports

Initially it was thought that the properties file could only be used to configure the server on the first deployment and thereafter the admin API had to be used to update and persist server settings.

This was a misconception. The properties file can always be used as the source of truth IF the
node.propertysource setting is set to PROPERTIES. This eliminates the need for the load_settings.py script which was reading the properties file and updating server configuration thru the admin API.

We're unable to rollback from Smile CDR version 2020.05.PRE-14 to 2020.02.R01. We want to do this because of #35. There seems to be a db migration issue.

Issue filed here: https://smilecdr.freshdesk.com/support/tickets/365

Server throws exception because it doesn't recognize the env var substitution syntax. Either the Smile CDR docs need to be updated with the correct syntax or Smile CDR has a bug in it.

Ticket submitted here: https://smilecdr.freshdesk.com/support/tickets/368

Update the line in the README about getting access to the smilecdr docker image. Users need a docker hub account and should be added as a collaborator to the docker hub repo.

Consider using an internal docker repo like AWS ECR so that any Kids First developer with access to the KF AWS account can pull the image after logging in. This way they don't need to create a Dockerhub account if they don't have one.

Not all deployments of the Smile CDR server will need to seed the server with the base FHIR conformance resources.

This setting should be configurable from the environment so that in prod it can be turned off for all deployments after the first one.

Referring to this fresh desk ticket, Smile released a new bug fix for a remote terminology service. It may resolve the following issues:

• d3b-center/clinical-data-flow#74
• kids-first/kf-model-fhir#142
• #32

The new Smile CDR release has a number of improvements, one being snapshot generation. We were told that upgrading to this release may resolve issues #35 and #30.

• Run merge_properties.py script to check for settings updates
• Update master.properties with any new settings
• Run update_smilecdr.sh script to update container registries
• Update Dockerfile with new tag
• Update docker-compose.yml with new tag

FHIR data models consist of conformance resources from 0 or more other data models. More specifically, a FHIR data model is distributed as an npm package, where the package can have dependencies which are other FHIR packages.

Since the FHIR package spec is still in development there is no tool that is able to build a complete FHIR package - aka collect all conformance resources in the fhir package and all of its dependent packages.

Once this functionality is implemented, the loader script should be updated to deploy (POST to the server) the conformance resources from a complete FHIR package.

The FHIR data dashboard needs to make a lot of queries to aggregate data and populate the dashboard. We should turn on server side caching to improve performance of common queries.

Prior to 2020 docker images, the conformance resources were always created automatically. Starting with 2020.02, this functionality was turned off by default but can be turned ON again by enabling the parameter "Seed Base Validation Resources" in Smile CDR.

We should turn this feature on so that server deploys are always populated with the FHIR R4 base model.

If not already done, add a deployment config GH repo which will contain the infra configuration and the Jenkins CI pipeline inputs. Will likely only need to define the test stage.

The docs for Smile CDR say that the server is finished deploying when the statement Smile, we're up and running :) is logged. This is misleading because the server will fail resource validation if a resource is submitted that makes use of a code in a ValueSet that has not yet been "pre-expanded" by the server. Smile CDR takes a minute or two to "pre-expand" or index codes in the HL7 ValueSets after it is deployed.

There doesn't seem to be an adequate way to determine when the server is finished pre-expanding ValueSets.

This could potentially be solved in the new release. We should try the new release out to see if it fixes this.

Create a Jenkinsfile so that the FHIR service is picked up by Jenkins to be deployed to the standard AWS dev, qa, prd environments.

Create two settings files with dev and production settings. This may also require separate .env files that will feed secrets into the properties settings file.

The production settings should follow:
Follow the production checklist https://smilecdr.com/docs/appendix/production_checklist.html

1. Create PCGC ingest package
2. Use ingest library to push cleaned PCGC data into D3b Warehouse

An exception is thrown by the logger filter whenever the logger logs an HTTP request to the health check endpoint /endpoint-health

21:04:06.166 [fhir_endpoint.fhirEndpointServer-127] INFO  kidsfirstdrc.fhir_endpoint.access - 2020-06-23T21:04:06.157Z GET /endpoint-health HTTP/1.1 200
21:04:06.165 [fhir_endpoint.fhirEndpointServer-127] ERROR ca.cdr.api.web.ExceptionLoggerFilter - Failure during processing

This doesn't cause the server to go down, however it clutters the log a lot and makes it difficult to visually inspect.

Every team member should have a practical understanding of basic FHIR concepts:

• How are entities and their attributes represented in FHIR
• How to model things - add/remove entity attributes, change entity relationships, and change constraints on relationships and attributes
• How to make entity attributes searchable
• How to represent entity attributes in various ontologies

The FHIR data dashboard] needs to make use of full text search in order to implement this feature: kids-first/kf-ui-fhir-data-dashboard#5

In order to use the ECS Type 1 deployment, we need to provide a Dockerfile to the deployment process. The Dockerfile will simply extend the Smile CDR image and add the necessary default server settings. Production settings will be passed in through the environment and override the default settings.