This is an official Pytorch Implementation of "CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training" [arxiv]
We have experimented the implementation on the following enviornment.
- Pytorch 1.12
- CUDA 11
pip install -r requirements.txtDatasets we used are as follows:
| Dataset | Download | Comment |
|---|---|---|
| MIMIC-CXR | Link | official split |
| CheXpert | Link | official split for train and val, and chexpert_5x200 from GLoRIA for test |
| ChestX-ray14 | Link | not used for test |
| VinDr-CXR | Link | official split for test, and random split for train and val |
| RSNA-Pneumonia | Link | same split as GLoRIA |
| SIIM-Pneumothorax | Link | same split as GLoRIA |
| OpenI | Link | all frontal images are used for evaluation |
For more details, please refer to data preparation.
We trained Resnet50 and SwinTiny models with three dataset compositions.
MIMIC-CXR (M), MIMIC-CXR + CheXpert (M,C), MIMIC-CXR + CheXpert + ChestX-ray14 (M,C,C14)
| model / dataset | M | M,C | M,C,C14 |
|---|---|---|---|
| ResNet50 | Link | Link | Link |
| SwinTiny | Link | Link | Link |
- single gpu
python train.py {--config-name default_config} - multi gpu
torchrun --nproc_per_node=4 --nnodes=1 --node_rank=0 --master_addr=127.0.0.1 --master_port=45678 train.py {--config-name default_config}
- Zero-shot classification
- perform zero-shot and image-text retrieval evaluation on ( vindr_cxr, rsna_pneumonia, siim_pneumothorax, chexpert5x200, mimic_cxr, openi )
python evaluate_clip.py test.checkpoint=${CKPT_PATH/model-best.tar}
- on rsna_pneumonia
# train
python finetune.py --config-name finetune_10 hydra.run.dir=${SAVE_DIR} data_train=rsna_pneumonia data_valid=rsna_pneumonia model.load_backbone_weights=${CKPT_PATH/model-best.tar} # 10%
python finetune.py hydra.run.dir=${SAVE_DIR} data_train=rsna_pneumonia data_valid=rsna_pneumonia model.load_backbone_weights=${CKPT_PATH/model-best.tar} # 100%
# evaluate
python evaluate_finetune.py data_test=rsna_pneumonia test.checkpoint=${FINETUNED_CKPT_PATH/model-best.tar}- on siim_pneumothorax
# train
python finetune.py --config-name finetune_10 hydra.run.dir=${SAVE_DIR} data_train=siim_pneumothorax data_valid=siim_pneumothorax model.load_backbone_weights=${CKPT_PATH/model-best.tar} # 10%
python finetune.py hydra.run.dir=${SAVE_DIR} data_train=siim_pneumothorax data_valid=siim_pneumothorax model.load_backbone_weights=${CKPT_PATH/model-best.tar} # 100%
# evaluate
python evaluate_finetune.py data_test=siim_pneumothorax test.checkpoint=${FINETUNED_CKPT_PATH/model-best.tar}- on vindr_cxr
# train
python finetune.py --config-name finetune_10 hydra.run.dir=${SAVE_DIR} data_train=vindr_cxr data_valid=vindr_cxr model.load_backbone_weights=${CKPT_PATH/model-best.tar} # 10%
python finetune.py hydra.run.dir=${SAVE_DIR} data_train=vindr_cxr data_valid=vindr_cxr model.load_backbone_weights=${CKPT_PATH/model-best.tar} # 100%
# evaluate
python evaluate_finetune.py data_test=vindr_cxr test.checkpoint=${FINETUNED_CKPT_PATH/model-best.tar}@incollection{You_2023,
doi = {10.1007/978-3-031-43895-0_10},
url = {https://doi.org/10.1007%2F978-3-031-43895-0_10},
year = 2023,
publisher = {Springer Nature Switzerland},
pages = {101--111},
author = {Kihyun You and Jawook Gu and Jiyeon Ham and Beomhee Park and Jiho Kim and Eun K. Hong and Woonhyuk Baek and Byungseok Roh},
title="CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training",
booktitle="Medical Image Computing and Computer Assisted Intervention -- MICCAI 2023",
}
CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training © 2023 is licensed under CC BY-NC 4.0
Kihyun You, [email protected]
Jawook Gu, [email protected]
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent