• Setting Github repo
• Data preparation in the local PC or AWS (S3 bucket)
• Setup feather instead of CSV
• Make test-set for gene expression data with random samples
• Establishing engineering strategy (IDE, Jupyter notebook)

publications, GTeX, GEO for validation or other analysis

• Think about MLOps strategy with Hydra + AWS + Github Action

• AWS Aurora, DynamoDB or etc

@kicheolkim It would be good if we have a new person for AI/ML support. We have knowledge about that obviously, but we need more advanced optimization techniques for ML model. What do you think? Do we have to find another person for this project?

#Error occurred in Mac conda-
dyld: Library not loaded: @rpath/libreadline.6.2.dylib

mamba update -c rdonnellyr -c main --all

mamba create --name new-env r-essentials=3.6.1 #it works, not working with DEseq

# First, create conda env with r-essentials
mamba create --name new-env r-essentials=3.6.1 

# Install packages with Rscript
Rscript notebook/installers/installer_Rpackage.R 

Reference:
https://towardsdatascience.com/jupyter-data-science-stack-docker-in-under-15-minutes-19d8f822bd45

Think about archiving

tar cvfz - ./ | aws s3 cp - s3://openkbc-test-lambda-bucket/test1.tar.gz

To have pathway activation score, we need to localize MSigDB datset

Currently, we are using docker instance to deploy notebook and it could be replaced with AWS sageMaker, possibly, pipeline container can be deployed in ECS.

dataset summary (what we have, what we can do), data analysis strategy

• Need progress page for monitoring snakemake log in controller

• Boto3 supports AWS control by python, current bash script needs to be replaced to boto

DEG pipeline has memory issue in reading all files

• Docker container has default memory limit(=2g).
• Attempted to change the limit to 4g, but failed (reference: docker/compose#4513)
• Needs code optimization

*Newly setup reviewNB for the repo, and need to be tested with jupyter code

Pipelines are working in a container through snakamake workflow. It needs a controller for user configuration and data input dynamically without up and down running a container.

• Need to upload normalized expression data for each dataset(CD4, CD8, CD14, etc) to S3
• Need to upload Rcode (DEseq2) which was used to get DEG