Mluti-modal multi-lingual Pre-trained model based on CLIP
This pre-trained model supports
English (en)
German (de)
French (fr)
Czech (cs)
Chinese (zh)
Russian (ru)
Vietnamese (vi)
Swahili (sw)
Spanish (es)
conda create -n mclip4vla python=3.7
conda activate mclip4vla
conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch
pip install tqdm transformers soundfile opencv-python boto3 ftfy pandas
pip install h5py librosa dominate
Download MSR-VTT from Baiduyun (passward:qhq7)
unzip it withtar -zxvf msrvtt.tar.gz
and place it in ./data
.
process the dataset with the following command:
python data_processor.py --extract_audios --load_video_into_frames
cd data/msrvtt
mv softlink.sh audios_16k/
cd audios_16k
bash softlink.sh
Download the pre-trained model from Baiduyun (passward:tp1k)
Place it in ./weights
Download the multilingual tokenizer setting from Baiduyun (passward:vth7)
Unzip it with tar -zxvf M-BERT-Based-69-ViT-B.tar.gz
and place it in ./weights
.
To evaluate our model, run
bash msrvtt_finetune.sh