With transfer learning, only 3000 training images is required to reach an average clip final pairwise correlation of 0.736 (compared to 0.687 without transfer learning).
- Follow instructions from brain-diffusor to create the python environment
Note: please make sure tokenizers==0.12.1 and transformers==4.19.2. For the diffusion environment, you may userequirement.txt
- For mac and linux:
virtualenv pyenv --python=3.10.12
source pyenv/bin/activate
pip install -r requirements.txt
- For Windows:
virtualenv pyenv --python=3.10.12
pyenv\Scripts\activate
pip install -r requirements.txt
- Download preprocessed eeg data, unzip "sub01", "sub02", etc under data/thingseeg2_preproc.
cd data/
wget https://files.de-1.osf.io/v1/resources/anp5v/providers/osfstorage/?zip=
mv index.html?zip= thingseeg2_preproc.zip
unzip thingseeg2_preproc.zip -d thingseeg2_preproc
cd thingseeg2_preproc/
unzip sub-01.zip
unzip sub-02.zip
unzip sub-03.zip
unzip sub-04.zip
unzip sub-05.zip
unzip sub-06.zip
unzip sub-07.zip
unzip sub-08.zip
unzip sub-09.zip
unzip sub-10.zip
cd ../../
python thingseeg2_data_preparation_scripts/prepare_thingseeg2_data.py
- Download ground truth images, unzip "training_images", "test_images" under data/thingseeg2_metadata
cd data/
wget https://files.de-1.osf.io/v1/resources/y63gw/providers/osfstorage/?zip=
mv index.html?zip= thingseeg2_metadata.zip
unzip thingseeg2_metadata.zip -d thingseeg2_metadata
cd thingseeg2_metadata/
unzip training_images.zip
unzip test_images.zip
cd ../../
python thingseeg2_data_preparation_scripts/save_thingseeg2_images.py
python thingseeg2_data_preparation_scripts/save_thingseeg2_concepts.py
- Download VDVAE and Versatile Diffusion weights
cd vdvae/model/
wget https://openaipublic.blob.core.windows.net/very-deep-vaes-assets/vdvae-assets-2/imagenet64-iter-1600000-log.jsonl
wget https://openaipublic.blob.core.windows.net/very-deep-vaes-assets/vdvae-assets-2/imagenet64-iter-1600000-model.th
wget https://openaipublic.blob.core.windows.net/very-deep-vaes-assets/vdvae-assets-2/imagenet64-iter-1600000-model-ema.th
wget https://openaipublic.blob.core.windows.net/very-deep-vaes-assets/vdvae-assets-2/imagenet64-iter-1600000-opt.th
cd ../../versatile_diffusion/pretrained/
wget https://huggingface.co/shi-labs/versatile-diffusion/resolve/main/pretrained_pth/vd-four-flow-v1-0-fp16-deprecated.pth
wget https://huggingface.co/shi-labs/versatile-diffusion/resolve/main/pretrained_pth/kl-f8.pth
wget https://huggingface.co/shi-labs/versatile-diffusion/resolve/main/pretrained_pth/optimus-vae.pth
cd ../../
- Extract train and test latent embeddings from images and text labels
python thingseeg2_data_preparation_scripts/vdvae_extract_features.py
python thingseeg2_data_preparation_scripts/clipvision_extract_features.py
python thingseeg2_data_preparation_scripts/cliptext_extract_features.py
python thingseeg2_data_preparation_scripts/evaluation_extract_features_from_test_images.py
python thingseeg2_scripts/train_regression.py
python thingseeg2_scripts/reconstruct_from_embeddings.py
python thingseeg2_scripts/evaluation_extract_features.py
python thingseeg2_scripts/evaluate_reconstruction.py
python thingseeg2_scripts/plot_reconstructions.py -ordered True
python thingseeg2_transfer_learning_scripts/transfer_learning.py
python thingseeg2_transfer_learning_scripts/train_regression.py
python thingseeg2_transfer_learning_scripts/reconstruct_from_embeddings.py
python thingseeg2_transfer_learning_scripts/evaluation_extract_features.py
python thingseeg2_transfer_learning_scripts/evaluate_reconstruction.py
python thingseeg2_transfer_learning_scripts/plot_reconstructions.py -ordered True
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent