suinleelab / monet Goto Github PK

Transparent medical image AI via an image–text foundation model grounded in medical literature

License: Other

Python 97.41% Shell 2.59%

artificial-intelligence dermatology medical-imaging

monet's Introduction

MONET (Medical cONcept rETriever)

MONET is an image-text foundation model trained on 105,550 dermatological images paired with natural language descriptions from a large collection of medical literature. MONET can accurately annotate concepts across dermatology images as verified by board-certified dermatologists, competitively with supervised models built on previously concept-annotated dermatology datasets of clinical images. MONET enables AI transparency across the entire AI system development pipeline from building inherently interpretable models to dataset and model auditing.

Getting started

Install

To install the required packages, run the following bash commands:

# clone project
git clone https://github.com/suinleelab/MONET
cd MONET

# [OPTIONAL] create conda environment
conda create -n MONET python=3.9.15
conda activate MONET

# install PyTorch according to instructions at https://pytorch.org/get-started/ v.1.13.0 was used during development.
# example: conda install pytorch==1.13.0 torchvision==0.14.0 pytorch-cuda=11.7 -c pytorch -c nvidia

# install other required python packages
pip install -r requirements.txt
pip install git+https://github.com/openai/CLIP.git

Initialize model

Using original openai CLIP implementation

import clip

def get_transform(n_px):
    def convert_image_to_rgb(image):
        return image.convert("RGB")
    return T.Compose(
        [
            T.Resize(n_px, interpolation=T.InterpolationMode.BICUBIC),
            T.CenterCrop(n_px),
            convert_image_to_rgb,
            T.ToTensor(),
            T.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),        
        ]
    )

model, preprocess = clip.load("ViT-L/14", device="cuda:0", jit=False), get_transform(n_px=224)
model.load_state_dict(torch.hub.load_state_dict_from_url("https://aimslab.cs.washington.edu/MONET/weight_clip.pt"))
model.eval()

Using huggingface CLIP implementation

from transformers import AutoProcessor, AutoModelForZeroShotImageClassification

processor_hf = AutoProcessor.from_pretrained("chanwkim/monet")
model_hf = AutoModelForZeroShotImageClassification.from_pretrained("chanwkim/monet")
model_hf.to("cuda:0")
model_hf.eval()

Usage

We provide jupyter notebooks to demonstrate how to use MONET for automatic concept annotation and various transparency tasks such as data auditing, model auditing, and inherently interpretable model building.

Automatic concept annotation: tutorial/automatic_concept_annotation.ipynb
Data auditing: tutorial/data_auditing.ipynb
Model auditing: tutorial/model_auditing.ipynb
Inherently interpretable model building: tutorial/inherently_interpretable_model_building.ipynb

MONET Training data

For code to download and preprocess the training data, please refer to the following scripts:

scripts/preprocess/preprocess_pubmed.sh
scripts/preprocess/preprocess_pdf.sh

Training / Evaluation

Code for preprocessing data and training MONET is available in src folder. Code used for evaluation in our paper is available in experiments folder.

Citation

@article{kim2024transparent,
    title={Transparent medical image AI via an image–text foundation model grounded in
medical literature},
    author={Chanwoo Kim and Soham U. Gadgil and Alex J. DeGrave and Jesutofunmi A. Omiye and Zhuo Ran Cai and Roxana Daneshjou and Su-In Lee},
    journal={Nature Medicine},
    year={2024},
    doi={10.1038/s41591-024-02887-x},
    url={https://doi.org/10.1038/s41591-024-02887-x}    
}

monet's People

Contributors

Stargazers

Watchers

Forkers

shenyu10 shangchengzhao

monet's Issues

Failure to replicate the CLIP concept generation experiment.

Thanks for your great work! I am trying to follow your steps and replicate the CLIP concept generation process on the Fitzpatrick17k split of the SkinCon dataset, but only get an AUROC of the 0.55. Could you please kindly explain at a high level if I did something wrong here?

Exclude any with less than 30 positive examples, use a prompt of 'This is {symptom}' for every symptom example.
For every image, re-sized and center-cropped to be 224x224 dimensions. It is then normalized using the mean and standard deviation used in CLIP
Use a pre-trained CLIP model from huggingface, here I tried (a). openai/clip-vit-large-patch14 (b). openai/clip-vit-large-patch14-336 (c). laion/CLIP-ViT-g-14-laion2B-s34B-b88K

Thank you in advance for any instructions!

Request for ’data‘ folder

Hello,

I wanted to express my gratitude for the outstanding work you’ve done! I must apologize if I’ve missed it, but I was unable to locate the ‘data’ folder in this repository, which is supposed to contain the ‘/pubmed/search_query.csv’ file and ‘/textbook/pdf_files’. Could you kindly let me know if there are any plans to release these pertinent resources?

Best Regards,
Chenlin

License for code?