xmed-lab / css-semivideo Goto Github PK

IEEE TMI 2022: Cyclical Self-Supervision for Semi-Supervised Ejection Fraction Prediction from Echocardiogram Videos

Python 100.00%

css-semivideo's Introduction

Cyclical Self-Supervision for Semi-Supervised Ejection Fraction Prediction from Echocardiogram Videos

This is the implementation of CSS for Semi-Supervised Ejection Fraction Prediction for the paper "Cyclical Self-Supervision for Semi-Supervised Ejection Fraction Prediction from Echocardiogram Videos".

Data

Researchers can request the EchoNet-Dynamic dataset at https://echonet.github.io/dynamic/ and set the directory path in the configuration file, echonet.cfg.

Environment

It is recommended to use PyTorch conda environments for running the program. A requirements file has been included.

Training and testing

The code must first be installed by running

pip3 install --user .

under the repository directory CSS-SemiVideo. Training consists of three components:

1) To train the CSS semi-supervised segmentation model, run:

echonet seg_cycle --batch_size=20 --output=output/css_seg --loss_cyc_w=0.01 --num_epochs=25 --rd_label=920 --rd_unlabel=6440 --run_test --reduced_set

The LV segmentation prediction masks of all frames must be inferred for the second stage. To do so, run:

echonet seg_cycle --batch_size=20 --output=output/css_seg --loss_cyc_w=0.01 --num_epochs=25 --rd_label=920 --rd_unlabel=6440 --skip_test --reduced_set --run_inference=train

echonet seg_cycle --batch_size=20 --output=output/css_seg --loss_cyc_w=0.01 --num_epochs=25 --rd_label=920 --rd_unlabel=6440 --skip_test --reduced_set --run_inference=val

echonet seg_cycle --batch_size=20 --output=output/css_seg --loss_cyc_w=0.01 --num_epochs=25 --rd_label=920 --rd_unlabel=6440 --skip_test --reduced_set --run_inference=test

The segmentation prediction outputs will be located under the output folder output/css_seg. To reduce installation time for EchoNet-Dynamic, these are moved to a separatate directory parallel to CSS-SemiVideo, i.e. CSS-SemiVideo/../infer_buffers/css_seg. Segmentation masks are also sourced from this location for Step 2 of the framework.

To do this, run:

mkdir ../infer_buffers/css_seg
mv output/css_seg/*_infer_cmpct ../infer_buffers/css_seg/

2) To train the multi-modal LVEF prediction model, run:

echonet video_segin --frames=32 --model_name=r2plus1d_18 --period=2 --batch_size=20 --output=output/teacher_model --num_epochs=25 --rd_label=920 --rd_unlabel=6440 --run_test --segsource=css_seg

3) To train teacher-student distillation, run:

echonet vidsegin_teachstd_kd --frames=32 --model_name=r2plus1d_18 --period=2 --batch_size=20 --output=output/end2end_model --num_epochs=25 --rd_label=920 --rd_unlabel=6440 --run_test --reduced_set --max_block=20 --segsource=css_seg --w_unlb=5 --batch_size_unlb=10 --weights_0=output/teacher_model/best.pt

Pretrained models

Trained checkpoints and models can be downloaded from:

CSS for semi-supervised segmentation: https://hkustconnect-my.sharepoint.com/:f:/g/personal/wdaiaj_connect_ust_hk/EqiP-N0MDRZGlwqr5PeZUrYBtLki8QWtBlMqRK1FNkjbcw?e=DIpkIm
Multi-modal LVEF regression: https://hkustconnect-my.sharepoint.com/:f:/g/personal/wdaiaj_connect_ust_hk/ErxaHepi4ndAnMcvSOwTH5wBDI6rHypqdcBiXF8B0XYvmg?e=Rud7Pf
Teacher-student distillation: https://hkustconnect-my.sharepoint.com/:f:/g/personal/wdaiaj_connect_ust_hk/Ev7mQ1ReI05LtiDIqQu1IpYBC6xN4R47PsYnhDUQr4n3fw?e=US4caq

To run with the pretrained model weights, replace the .pts files in the target output directory with the downloaded files.

Experiments	MAE	RMSE	R²
Multi-Modal	5.13 ± 0.05	6.90 ± 0.07	67.6% ± 0.5
Teacher-student Distillation	4.90 ± 0.04	6.57 ± 0.06	71.1% ± 0.4

Notes

Contact: DAI Weihang ([email protected])

Citation

If this code is useful for your research, please consider citing:

(to be released)

css-semivideo's People

Stargazers

Watchers

Forkers

pinghe-stan

css-semivideo's Issues

I can't find "echonet. Models. Rnet2dp1. VideoResNet_ncor" this function

Hello, I'm sorry to bother you. I'd like to ask you some questions

I had this problem when I was running to train the multi-modal LVEF prediction model, I can't find "echonet. Models. Rnet2dp1. VideoResNet_ncor" this function.

The following two errors occur when I use VideoResNet or VideoResNet_kd.

I don't know how to solve this problem. I would like to ask if you have encountered similar situations before and how to solve them.
I hope you can provide some help, which will be greatly appreciated. Thank you

The calculation method of 5 times run

dear authors,

i am quite interested about your work, and i want to know the way calculate std.
i noticed that in file 'vidsegin_teachstd_kd.py/video_segin.py', the eval is calucated for 5 times, is the reported std calculated between the 5 tests; or more like 5times run*5 tests?
Does the 5 times run use the same split file?

thanks

Questions about external validation on camus

Dear authors,

I noticed that you did external test on CAMUS, I wonder if it is possible to get more detail about it.
I tried to use the official weight of echonet LVEF prediction, I directly crop the size to (112,112). I also did temporal mirroring mentioned in your paper, but the output MSE is really large, instead of 6.82 reported in your paper.
I want to know what else did you do on the dataset? like finetune or split dataset or something like calc (who's) mean&std.

thanks.

Reproducibility of results

Hi, I download the checkpoints and put them in the output directory, then I use the commands to train the three models, but I can't get the same results as in the table. Is there anything else that needs to be changed?
Your results

Experiments	MAE	RMSE	R^2
Multi-Modal	5.13±0.05	6.90±0.07	67.6%±0.5
Teacher-student Distillation	4.90±0.04	6.57±0.06	71.1%±0.4

my results

Experiments	MAE	RMSE	R^2
Multi-Modal	5.07	6.90	68.1%
Teacher-student Distillation	4.99	6.75	69.6%

My steps:

download the weights from the link and create a folder named "output" and subfolders "css_seg" "end2end_model" "teacher_model", put the weights in the subfolders,respectively.
run commands in terminal one by one.

echonet seg_cycle --batch_size=20 --output=output/css_seg --loss_cyc_w=0.01 --num_epochs=25 --rd_label=920 --rd_unlabel=6440 --run_test --reduced_set 

echonet seg_cycle --batch_size=20 --output=output/css_seg --loss_cyc_w=0.01 --num_epochs=25 --rd_label=920 --rd_unlabel=6440 --skip_test --reduced_set --run_inference=train

echonet seg_cycle --batch_size=20 --output=output/css_seg --loss_cyc_w=0.01 --num_epochs=25 --rd_label=920 --rd_unlabel=6440 --skip_test --reduced_set --run_inference=val

echonet seg_cycle --batch_size=20 --output=output/css_seg --loss_cyc_w=0.01 --num_epochs=25 --rd_label=920 --rd_unlabel=6440 --skip_test --reduced_set --run_inference=test

mkdir ../infer_buffers/css_seg
mv output/css_seg/*_infer_cmpct ../infer_buffers/css_seg/

echonet video_segin --frames=32 --model_name=r2plus1d_18 --period=2 --batch_size=20 --output=output/teacher_model --num_epochs=25 --rd_label=920 --rd_unlabel=6440 --run_test --segsource=css_seg

echonet vidsegin_teachstd_kd --frames=32 --model_name=r2plus1d_18 --period=2 --batch_size=20 --output=output/end2end_model --num_epochs=25 --rd_label=920 --rd_unlabel=6440 --run_test --reduced_set --max_block=20 --segsource=css_seg --w_unlb=5 --batch_size_unlb=10 --weights_0=output/teacher_model/best.pt

Furthermore, when I train the model from scratch using the commands given, I get the results below. Is there any other way to get a better result than loading the weights that you've given?
my results from scratch

Experiments	MAE	RMSE	R^2
Multi-Modal	5.31	7.26	64.8%
Teacher-student Distillation	5.59	7.55	61.9%

since I only have a 24G 3090 GPU, when run the vidsegin_teachstd_kd from scratch I set --batch_size=8 --batch_size_unlb=4

Besides, How to calculate the std ? Is it in the log.csv?