kind-lab / mimic-fhir Goto Github PK
View Code? Open in Web Editor NEWA version of MIMIC-IV in FHIR
License: MIT License
A version of MIMIC-IV in FHIR
License: MIT License
Using thejava validator to validate resources in the python testing folder. Currently using HAPI but could be nice to have the java validator as another option since its quicker to use.
To Do
The structure of the microbiology data in FHIR is not the most straightforward. Go through and confirm the data is coming across fine. To test:
Misc questions
An update to the medication resources in FHIR needs to be done. The following updates to each resource:
Medication- MedicationMix
MedicationAdministration
MedicationRequest
pharmacy
. Pull from prescriptions
tablestatus
element for mapping of MIMIC-IV status value to FHIR standard value. status
element is bound in MedicationRequest as a code, so no codesystem can be supplied like in a CodeableConcept. Need this updated so that profiles will validate in anneal.dosageInstructions
. Look into prescriptions
and pharmacy
columns for anything on dose, rate, duration etc. Document in MIMIC-IV to FHIR mappings fileMedicationDispense
Misc To Do
emar
, emar_detail
, and inputevents
Create FHIR profiles to bind custom MIMIC valuesets
Currently passing in ethnicity from MIMIC for both race and ethnicity. Will definitely need to update race to map to proper race/ethnicity.
Thoughts:
Add example files for the validator to test. This will decouple the loading process from the db.
But then that is really only a validation of the validator... so not really worth it? I guess it says the validator is working BUT it does not test any current data.
The micobiologyevents table has a column spec_type_desc
that could be translated to the Specimen type.
Thoughts:
I noted two potential bugs in running python py_mimic_fhir
:
Bundle resources to automatically detect references and more efficiently send resources to HAPI. To Do:
For the tutorial demo of mimic-fhir, my hope was to use Pathling to demonstrate different fhir functions. But I am hitting a roadblock right off the bat, the $import is failing. The steps I have taken:
But I am getting an error saying that there is an error reading the file. I've given all permissions to the /usr/share/staging folder so that should be fine. @jpwiedekopf you had mentioned using Pathling, any ideas?
Need to download install mimic cxr for this, then add this to fhir_etl.subjects:
WITH cxr AS (
SELECT subject_id,
min(study_id) as study_id_min,
max(study_id) AS study_id_max,
count(*) as n
FROM mimic_cxr.record_list
GROUP BY 1
)
select pt.subject_id,
gender,
anchor_age,
anchor_year,
anchor_year_group,
dod,
cxr.n as n_cxr,
ie.stay_id,
ie.intime,
ie.outtime,
ROW_NUMBER() OVER (
PARTITION BY pt.subject_id
ORDER BY ie.intime
) AS rn,
dense_rank() OVER (
ORDER BY pt.subject_id
) AS rank
from mimic_core.patients pt
INNER JOIN mimic_icu.icustays ie ON pt.subject_id = ie.subject_id
LEFT JOIN cxr ON pt.subject_id = cxr.subject_id
where anchor_age > 0
and anchor_year_group in ('2011 - 2013', '2014 - 2016')
ORDER BY subject_id,
intime
limit 134;
Limit to 100 potentially, since that is the demo size we are going for.
With the latest release of MIMIC-IV, the medication tables have been updated to facilitate better medication mapping in FHIR.
New columns:
What the new values provide is matching between emar and prescriptions, but ICU will still just be medication text fields....
How medication will be mapped:
Potential issues
pharmacy_id
is present in pharmacy, prescriptions and emar_detail for the majority, but this would create a TON of duplicate medications if that was the source.Potential solutions:
Add comments to Observation.note for both ObservationMicroSusc and ObservationMicroTest.
For ObservationMicroSusc, store all comments in the note.
For ObservationMicroTest, currently storing the comments in the valueString
. Might be worth having in both spots? Debate this. Comments is a result when there is no result...
Add functionality to export patient bundles in place of just full export of resources. Currently outputting each research type and all the resources associated in one file. Exporting in a patient bundle seems to be the community push (but some bundles will be very big)
Patient bundles may need to be organized by patient and encounter? Cause an individual patient could have ~50,000 resources on their own (one patient has 42,000 chartevents).
Hey,
I was looking at your SQL scripts and wondering why the labs script is limited to 1000 at the end?
Check that bundles toss errors if invalid codes are present
Terminology is currently generated in a jupyter notebook, move this into the py_mimic_fhir package.
Steps to do this:
The $HOST
variable used in the .env is actually a standard environment variable. We should use something more specific like SQLHOST or something similar.
This is a list of ConceptMap resources that will likely be needed in the future
Encounter
type
from MIMIC-IV admissions ontology to US Core Encounter Typeclass
from MIMIC-IV admissions ontology to US Core Act Encounter Codehospitalization.admitSource
from MIMIC-IV admissions ontology to AdmitSourceEncounter (ICU)
type
from MIMIC-IV icustays ontology to US Core Encounter Typetype
from MIMIC-IV icustays ontology to US Core Act Encounter CodeCondition
code
- in MIMIC-IV it is a mix of ICD-9 / ICD-10 from multiple years.Medication
code
- in MIMIC-IV it is NDC/GSN/text, mapped to Medication Clinical DrugMedicationAdministration
method
from event_text
in emar and ordercategorydescription
in inputevents, map to SNOMED codes in Administration Method Codesroute
from emar_detail and pharmacy could be mapped to SNOMED route codesMedicatinAdministrationICU
category
from inputevents ordercategoryname
, could map to Medication Admin Category . MIMIC provides much more granularity here so we could keep the new Codesystem versus the category would become just inpatient/outpatient effectively.MedicationRequest
status
is converted within the MIMIC-IV to FHIR code, but should have a conceptmap as wellroute
to FHIR route codesMedicationRequest
method
is from emar event_txt
, should go to SNOMED administration code valuesetroute
is here tooObservationLabs
interpretation
is from labevents flag
, need to map into Observation InterpretationProcedureICU
bodysite
is from procedureevents location
, need to map into SNOMED bodysitecategory
is from procedureevents ordercategoryname
, need to map into Procedure CategoryThere a quite a few results that appear to be reported in the comments. For example there are ~4 million labevents entries that have no value
or valuenum
but have a comments
='NEG.'. This seems like a decent chunk of data that would not translate over properly to FHIR without some parsing.
Thoughts
valueString
if labevents value
and valuenum
are missing
comments
are currently being captured in FHIR notes
, is that enough?Test referencing the ValueSet in the CodeSystem, then expanding when it arrives in the FHIR Server.
Thoughts:
Currently, any identifiers in mimic-fhir reference a custom URLs such as http://mimic.fhir.mit.edu/identifier/patient, but nothing is defined for that. It was brought up that a NamingSystem could make sense here. Will look into this further!
Add fhir_etl tables that are used to generate the main tables. These are primarily tables for required bindings ie patient gender or procedure status.
We need a method to cleanse the server between runs to ensure there are no hanging references.
There are a couple methods to clean the server when it is up and running:
What I've tested:
Stopgap solutions:
The mimc4fhir team pulled in mimic_core.transfers. They set the transfer information in the Encounter.
What we can do then:
Demo mimic-fhir store should be based on the patients found in the MIMIC-IV to OMOP mapping.
Alistair sent over list of patients that should be pulled in, so write those subjects to fhir_etl.subjects. This will affect all tables
The terminology in mimic-fhir should reference the version of MIMIC that they are based off of.
There are a couple spots to put this in CodeSystem/ValueSet:
version
: this is the business version of the codesystem, but we could map this to the mimic version?useContext.value[x]
: could store the mimic version here to be the context of creating the mimic-fhir terminologypurpose
: why the code system is definied, so could put the version here. But this is a markdown datatypeWhen sending bundles to HAPI FHIR catch any failed bundles and write out to file. These can then be rerun later.
To Do
Hi, when running fhir_patient.sql it doesn't find mimiciv_ed.edstays, I believe this table is not part of MIMIC-IV v2.2, or am I missing something?
The script runs fine when the related code is removed (just using adm_RACE).
Test the 100 demo patients through the current pipeline to see timing and issues.
Steps to test
The ICU observations (chartevents, datetimeevents, outputevents) all have a warning
column. There is no real mapping into the Observation resources for it. Is it worth creating an extension for warning
? Also what does the warning even tell us?
I want to get a better idea of the warning
column before converting into an extension
Tests to make sure HAPI server setup is working for basic resource actions. Tests:
Add CLI functionality to run through py_mimic_fhir. I see it with two modes:
validator
argument with options Java or HAPIoutput
argument with options: resources or bundlesenv
argument to point to env file? Or have default assumption probablyICU tables were converted to FHIR resources with references to only the ICU encounter, not hospital encounter. The limitation is that FHIR only allows one Encounter reference from the resources.
Current solution:
To Do
Find the largest bundle in our demo dataset. This can be a benchmark to see what might break the bundling and validation.
The microbiologyevents table has codes and labels for all specimen. Only taking in the spec_type_desc right now. Should have pulled in spec_itemId. So the update would be:
Currently just using spec_type_desc as the code
Currently passing bundles in for loop. But HAPI seems to have a bit of a limit when passing, so may not be as big an issue.
Attempted to pass bundles using list comprehension but was 7-8 times slower.
Other ideas
Need to make sure terminology generation can be run repeatably when updating between versions of mimic.
Current steps to generate terminology:
To Do
Hey,
I ran the script and not all tables went through without issues. The problem is the mappings in fhir_etl. Only subjects and uuid_namespace were created. The others were not found.
psql:fhir_patient.sql:117: FEHLER: Relation »fhir_etl.map_gender« existiert nicht ZEILE 52: LEFT JOIN fhir_etl.map_gender mg
The resources have bindings to custom valuesets for MIMIC. We need to verify the codes are being validated properly.
Current issue:
To Do:
Make sure the terminology that references mimic tables users there exact table names.
ie datetime-d-items should be datetimeevents-d-items
To update:
Go through valueset bindings of the resources and find which ones can be mapped to custom valuesets and which ones need to be mapped in Postgres with fhir_etl tables.
Review labevents and confirm that all columns we want are making it into FHIR. The initial pass I did pulled in anything that I could map to the ObservationLab profile. There may need to be extensions to fit others (ie lab urgency).
Some notes:
status
are set to 'final' in fhir currently
status
can be set to registered/preliminary/final/amendedThe Bundler runs over each of the resources to bundle and validate. The class has grown and could potentially be better organized.
General thoughts:
None of these changes are necessary for Bundler to run, just potential refactor ideas
Estimate the time it will take to generate all the fhir resources in Postgres. Test 100,000 generation for each resource/profile.
MIMIC ICD Codes are not strictly ICD-9 or ICD-10. There are codes in MIMIC that are obsolete ICD codes across ICD versions.
Current solution
To Do
Set the canonical ValueSet in the CodeSystem if it captures the whole codesystem.
To test this:
To ensure the integrity of the data some tests need to be created:
Referential integrity
Check for null json strings
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.