miriamkw / glupredkit Goto Github PK
View Code? Open in Web Editor NEWGluPredKit aims to make blood glucose model training and prediction more accessible.
License: MIT License
GluPredKit aims to make blood glucose model training and prediction more accessible.
License: MIT License
To enhance usability even further, do the following:
The forward method in lstm pytorch should not be squeesed. this leads to the error: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
Suggested from a JOSS track editor: I suggest that you make this something that anyone can do - post the dataset somewhere public, and update your README and/or docs to tell people where it is and how to test the software with it. This is just a suggestion, but I think it would make the software easier for people to access and try.
Originally posted by @danielskatz in openjournals/joss-reviews#6904 (comment)
Address this issue by adding a synthetic dataset with simulated, realistic data
when I run calculate_metrics on the config 1 LSTM models, I get a shape error (Input 0 of layer "model" is incompatible with the layer: expected shape=(None, 24, 27), found shape=(None, 12, 1)).
Enhance the data parsers:
Note: This should be a helper for the cli instead of downstream in the parsers/preprocessors to avoid redundancy.
Add nighscout parser:
Tidepool and nighscout:
Other
Handle units:
y_pred, y_true consistency In models, metrics etc:
rename repository:
Add a command line interface used to
Add a possibility in the scatter plot to allow/disallow certain models so you can interactively choose which models to compare.
Tasks remaining to finish the tool:
We will build heavily on the benchmarkpaper for this tool, to test it on real life scenarios.
Preparations
Parsers
Summary: All the parsers should do the same: return a "raw dataframe" with all of the datatypes in the same time grid, but without any fuzz.
Preprocessors
Models
Evaluation
Real-time plots:
Settings
Command line interface
Documentation
Code improvements (feedback from chatgpt):
Add a nightscout parser so that we can check predictions up against loop in real time.
Find a solution to more elegantly handle glucose units mmol/L vs mg/dL
Adjusting GluPredKit to pave the way for making "what-if"-events the common choice in model training:
Update the repo to use whole trajectories in model training.
How it works now:
Data goes through a preprocessor that adds one target column (given by the prediction horizon). Then, for each prediction horizon, one model is trained. Models assume that there is a target column named "target", and will use this name convention to "find" the target.
My thoughts about the refactor:
Enhance the generate_config command in the CLI:
The solution can be to give explanatory error messages immediately after an input error, and reprompting the input field to let users try again from where they left off.
The pip install of GluPredKit does not create an executable entry point.
I have:
#!/usr/bin/env python3
to cli.pychmod +x glupredkit/cli.py
setup.py
Resources:
The loop model currently runs without errors and follows the pyloopkit detailed instructions for implementation. However, we should add tests to ensure that the output is the same as in the Loop Swift implementation. The work is slightly started in tests/test_pyloopkit
.
Here are the answers from Pete:
There isn’t currently a command line interface, but some of the tests run a full top to bottom exercise of the algorithm using a json input file and compare to the contents of a json output file, so a command line program would essentially wrap that.
The loadScenario ones load an input and the output (recommendation). https://github.com/tidepool-org/LoopAlgorithm/blob/main/Tests/LoopAlgorithmTests/LoopAlgorithmTests.swift#L28
LoopAlgorithmTests.swift
let (input, recommendation) = loadScenario("suspend")
tidepool-org/LoopAlgorithm
Test live capture also tests the predicted glucose curve: https://github.com/tidepool-org/LoopAlgorithm/blob/main/Tests/LoopAlgorithmTests/LoopAlgorithmTests.swift#L148
LoopAlgorithmTests.swift
func testLiveCapture() {
We can use these instructions to check the swift model output using different mock data files and compare the loop predictions.
Lets do the plotting stuff in ipython notebooks, no? Then the plots will be visible in GitHub for all to see, among other benefits.
The repository should have a new name - more descriptive of the purpose which is to evaluate any kind of blood glucose prediction based on any kind of penalty function. Suggestions:
Following the usage guide at the generate-model-training-configuration step, I ran into a minor issue with the input paths requirements for the glupredkit generate_config
CLI function.
The preferred usage of the CLI is to navigate to the working directory containing the data
folder structure, and all file path should be auto resolved. However, some users may provide the full path, leading to an error message that's not very clear:
Example:
glupredkit generate_config --file-name data/configurations/synthetic_configs --data data/raw/synthetic_data_with_datetime.csv --prediction-horizon 5 --num-lagged-features 12 --num-features CGM,insulin,carbs --cat-features is_test
Error:
ValueError: Data file 'data/raw/synthetic_data_with_datetime.csv' not found in 'data/raw/' folder.
The error message might be confusing to certain users as it implies the tool is unable to find synthetic_data_with_datetime.csv
in data/raw
, even when it is.
A suggestion is to validate --file-name
and --data
args to only contain file names and not include any directory paths, as that would directly inform the user on the correct usage. I'd also suggest to include default or example values for each argument when the user runs glupredkit generate_config --help
.
JOSS review: openjournals/joss-reviews#6904
The first step to make sure that we are using pyloopkit correctly and fetching correct predictions, is to run the example_nightscout.py
and verify with out Loop app in real time. Currently the predictions are slightly off.
Possible solutions:
Will make adoption WAY easier if in the readme we can just say
pip install glupredkit
or pip install git+git://github.com/miriamkw/GluPredKit
This involves a few steps
Basic tutorial: https://packaging.python.org/tutorials/packaging-projects/ that includes these steps.
We should make sure it works via pip install glupredkit
as well as pip install git+git://github.com/miriamkw/GluPredKit
, probably w/ initial emphasis in the second one, since that doesnt require doing any uploads to pypi. People will use this both to gain access to the framework within python and from the command line. So python -m glupredkit <command>
will become glupredkit <command>
.
Helper methods/classes to process data:
Implement a github action for automatic version bumbing and upload to pypi when merging to Main branch.
Finish current script:
Create a script that:
Before merge:
My understanding is that the apple_health parser should instead take a filename as an argument, but it complains that no USERNAME is supplied:
Line 72 in d8ee553
In models, metrics etc:
The install steps are outlined in clear enough terms in the readme - no reason the user should have to do each thing independently. Let's make a manual install.sh script (idk if this is the best name, but it will be a bash script), that runs everything at once and/or in chunks as needed by setup.py. This will also be useful for organizational purposes when we make the whole thing installable via pip, which it looks like you've started with (b/c you have setup.py).
Jupyter notebook (?):
Further work:
After feedback on the figures, add the updated versions.
The scripts run very slowly, and some optimisation should be performed in:
Before we start on this we should measure which processes that are most time consuming.
Create a helper class for preprocessing:
GluPredKit refactor for enhanced and strict model evaluation:
Refactor:
Configuration:
Generate pdf reports for model evaluation:
Models:
Parsers:
Metrics:
Testing:
General improvements:
Checklist before merge
Ohio T1DM fixes:
Currently, the trajectories plot is cutting off all predictions that dont have corresponding measurements to all of the trajectory. Should this be changed?
Evaluator:
Prediction models:
tidepool_parser.py
:
tidepool_parser.py
and other code has no unused code, and that the code is written efficientlyExample scripts:
Tests:
Cleanup:
Plots (low priority):
As discussed, create the data directory as needed in the directory where the cli is being installed/used
The python dependency mpi4py is either in requirements.txt or is needed by one of the other packages used, but the system wide Message Passing Interface requirements may not be there.
on Mac this is solved by doing brew install mpich
before the install
on debian/linux this is solved by doing sudo apt-get install libopenmpi-dev
before the install
It would be nice to automate this process for people installing the package via pypi - (or maybe you already have).
Currently there is configuration for that data processing. This configuration may be used with any model.
However, models might also have many different configurations, like hyperparameters or therapy settings. These can be cumbersome to define in a CLI, and should probably be added as configuration files as well.
This means a refactor that:
Currently in the generate_config command, it is only possible to use the promts to generate configurations. It should also be an option to generate configs programmatically.
After generating the configurations to use a categorical feature and running glupredkit train_model random_forest synthetic_configs
give an error:
Starting pipeline to train model random_forest with configurations in synthetic_configs...
Preprocessing data using standardscaler from file data/raw/synthetic_data_with_datetime.csv, with a prediction horizon of 5 minutes...
Traceback (most recent call last):
File "/micromamba/envs/glupredkit/bin/glupredkit", line 8, in <module>
sys.exit(cli())
File "/micromamba/envs/glupredkit/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/micromamba/envs/glupredkit/lib/python3.9/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/micromamba/envs/glupredkit/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/micromamba/envs/glupredkit/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/micromamba/envs/glupredkit/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/micromamba/envs/glupredkit/lib/python3.9/site-packages/glupredkit/cli.py", line 207, in train_model
train_data, _ = helpers.get_preprocessed_data(prediction_horizon, model_config_manager)
File "/micromamba/envs/glupredkit/lib/python3.9/site-packages/glupredkit/helpers/cli.py", line 106, in get_preprocessed_data
train_data, test_data = chosen_preprocessor(data)
File "/micromamba/envs/glupredkit/lib/python3.9/site-packages/glupredkit/preprocessors/standardscaler.py", line 14, in __call__
train_df, test_df = self.preprocess(df)
File "/micromamba/envs/glupredkit/lib/python3.9/site-packages/glupredkit/preprocessors/standardscaler.py", line 82, in preprocess
processed_test_df = self.transform_with_encoder(processed_test_df, encoder)
File "/micromamba/envs/glupredkit/lib/python3.9/site-packages/glupredkit/preprocessors/standardscaler.py", line 87, in transform_with_encoder
encoded_cols = encoder.transform(df.loc[:, self.categorical_features])
File "/micromamba/envs/glupredkit/lib/python3.9/site-packages/sklearn/utils/_set_output.py", line 313, in wrapped
data_to_wrap = f(self, X, *args, **kwargs)
File "/micromamba/envs/glupredkit/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py", line 1024, in transform
X_int, X_mask = self._transform(
File "/micromamba/envs/glupredkit/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py", line 214, in _transform
raise ValueError(msg)
ValueError: Found unknown categories [np.True_] in column 0 during transform
The configuration file was generated with:
glupredkit generate_config --file-name synthetic_configs --data synthetic_data_with_datetime.csv --prediction-horizon 5 --num-lagged-features 12 --num-features CGM,insulin,carbs --cat-features is_test --preprocessor standardscaler
{
"data": "synthetic_data_with_datetime.csv",
"subject_ids": [],
"preprocessor": "standardscaler",
"prediction_horizon": 5,
"num_lagged_features": 12,
"num_features": [
"CGM",
"insulin",
"carbs"
],
"cat_features": [
"is_test"
],
"what_if_features": []
}
This happens for either the basic or the standardscaler preprocessor.
JOSS review: openjournals/joss-reviews#6904
Refactoring:
cli
in helpers, and add methods like "getModule" for models etc. to reduce cli.pyAdd real-time predictions (after config, so you dont need to refactor):
Update CLI:
Documentation:
Finish:
Add metrics:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.