zjc062 / mind-vis Goto Github PK

Hello authers

I was going through the codebase and noticed that in the main function, the test set is used for fine-tuning the model. I think this could introduce data leakage and lead to overly optimistic performance evaluations, right? Below is the section of code I'm referring to:

# create dataset and dataloader
if config.dataset == "GOD":
    _, test_set = create_Kamitani_dataset(
        ...
    )
elif config.dataset == "BOLD5000":
    _, test_set = create_BOLD5000_dataset(
        ...
    )
else:
    raise NotImplementedError

# ... later in the code
print("Finetuning MAE on test fMRI ... ...")
for ep in range(config.num_epoch):
    ...
    cor = train_one_epoch(
        model, dataloader_hcp, optimizer, device, ep, loss_scaler, logger, config, start_time, model_without_ddp
    )

https://github.com/zjc062/mind-vis/blob/main/code/stageA2_mbm_finetune.py#L125

Congratulations! Thank you for your nice work!
Can you tell me how to visualize the stimuli like Figure 2 in the main paper of MinD-Vis?
Thank you for your nice work so much

A fantastic work, very interested in it.
Is the dataset of Human Connectome Projects (HCP) downloaded from this link?

Greetings,

First of all I am really fascinated by the work Your team has done. Thank you!
I am excited to run and test your code, but unfortunately I do not have the resources to train the SC-MBM part in a reasonable amount of time.
Is it possible for You to share the weights and biases that You've acquired during Your training? I would appreciate it a lot.

Regards, Alexander.

The file you shared on FigShare has turned into a private link, making the data and model inaccessible, can you please check this?

I noticed that there are two files with the code for sampling. (DDIM.py and PLMS.py). Can you explain the code a bit more or give me a reference so that I can understand the code better?

Hi, thanks for the cool paper.

I try to reproduce your result through finetuning ldm of StageB by myself. I directly use your code and fmri_encoder checkpoint without any modifications.
When I compare the performance between the model you provide (finetune.pth) and the model finetuned by myself, there is a gap in both the generated image results and the numerical metrics. For example, the "summary/top-1-class(max)" of your checkpoint is 0.226 while my checkpoint is only 0.1916.
I wonder if you could kindly provide any insights on what may have caused this discrepancy.

Thanks for your time!

Hi, thanks for the very cool paper!

I'm trying to reproduce your results, and I noticed that your FigShare link stores the Kamitani dataset as /data/.../.npz files, which differs from the format that the original GOD repository uses (.h5 files in their FigShare). How did you perform the conversion from their .h5 files to your .npz files? Is the code available?

Thanks!

Is it reasonable to pretrain a vision decoder from rest-state data when subjects viewing only a cross? What if we pretrain on task data with more vision activities?

Hello, I would like to know if the training parameters used when fine-tuning the pre-trained model using GOD or BOLD5000 are the default values in config.py. Is 15 epochs sufficient? What is the reasonable correlation obtained in the end? Looking forward to your reply.

Hey there!! I loved your work, amazing.

I'm trying to use this model to reconstruct images from a different fMRI data that I have but I have a time restriction so to do all the pre-training from scratch will result in a lot of time loss for me. I was wondering if you can share the corresponding files with me so that I can continue with the pre-trained model and skip the stage A. That would be crazy helpful.

Thank you

After reading the paper, my understanding is that the pre-training task in this paper is designed to predict the signal at the same time point of the region that is masked out from the BOLD signal of the neighboring region. My question is why not design the task to recover the signal at the masked time point from the signal in the same region before and after the time point？Because of the high spatial correlation of the BOLD signal, recovering the signal at the same time point based on neighboring regions seems to be a relatively easy task.

Hi,

We are currently replicating your work, but encountered some issues.
How to get files indicating visual cortex area information in the HCP dataset, like the "HCP_visual_voxel.npz" file in your code dataset.py. Did you regenerate the parcellation using software like Freesurfer, or something else?

Thanks for your reply !

HI! I really appreciate your work and plan to follow it in our area. By the way, may I ask a question about the training correlations calculated by torch.corrcoef?

I have applied the MBM model to another dataset and made several changes. However, I have recently found that the model may not perform well under the correlation metric. Could you please tell me the training correlations in your settings, and what is your opinion on this phenomenon?

The attached image shows the models that we have trained recently, with the red and grey lines representing the models.

Hello, I downloaded your code and the training dataset you provided, and when I pre-trained SC-MBM, I found that there are only 6360 fMRI samples in the training set, not 136000 samples as mentioned in the original paper.
How can I solve this problem?

Hii there, I am trying to run the model on Google colab but after 200 epoch in A1 stage, GPU gets disconnected and the entire training of 200 epoch get lost.
Can any one tell me how to train 200 epochs and save model and next day, start with 201 epoch to finish my training for 500 epoch.

Hi! Thank you for your EXCELLENT job! It really helps a lot!
I have a quesiont about SC_MBM. I see you've used a very large dataset and got very good results. However training on such a large dataset is not realistic for me. So I am wondering if I only use 6360 fMRI samples provided in your project, can I get acceptable results, like 80% of yours?
I'll really appreciate it if you answer my questions. Thanks a log!
Have a nice day!

Because my English writing is not so good, I will translate into Chinese to avoid misunderstanding:
作者您好，非常感谢您如此出色的工作！我有一个关于SC_MBM的问题想请教。
我看到你们在一个相当庞大的数据集上进行了SC_MBM的pretrain，但是如此庞大的数据量对于我来说是不可能完成训练的。所以我想请问如果我只使用工程内提供的6360个fMRI数据，可以得到可接受的结果吗，例如您目前结果的80%？
期待您的回复，十分感谢！
祝好！

Hi! Thank you for your EXCELLENT job! It really helps a lot!
I have a quesiont about SC_MBM. I see you've used a very large dataset and got very good results. However training on such a large dataset is not realistic for me. So I am wondering if I only use 6360 fMRI samples provided in your project, can I get acceptable results, like 80% of yours?
I'll really appreciate it if you answer my questions. Thanks a log!
Have a nice day!

Because my English writing is not so good, I will translate into Chinese to avoid misunderstanding:
作者您好，非常感谢您如此出色的工作！我有一个关于SC_MBM的问题想请教。
我看到你们在一个相当庞大的数据集上进行了SC_MBM的pretrain，但是如此庞大的数据量对于我来说是不可能完成训练的。所以我想请问如果我只使用工程内提供的6360个fMRI数据，可以得到可接受的结果吗，例如您目前结果的80%？
期待您的回复，十分感谢！
祝好！

Hello!

I am trying to visualize the architecture of mind-vis down to the layer for better understanding but I have trouble finding a description of the architecture.
In the paper it is written that the encoder depth is 24. Does it mean that it has 24 layers? If so where can I look up the input and output size for each layer?
Do you have any general recommendations on comprehending the exact architecture of mind-vis?

With regards, Alexander.

Thank you for your excellent work. Could you please help me with questions below?
It is weird that when I first ran this file it worked well. However, when I repeated my operation, it threws out error info like:
Traceback (most recent call last):
File "code/stageB_ldm_finetune.py", line 245, in
main(config)
File "code/stageB_ldm_finetune.py", line 163, in main
generative_model.finetune(trainer, fmri_latents_dataset_train, fmri_latents_dataset_test,
File "/public1/home/ungradu/home/gra02/lyh_test/mind-vis/mind-vis-main/code/dc_ldm/ldm_for_fmri.py", line 103, in finetune
trainers.fit(self.model, dataloader, val_dataloaders=test_loader)
File "/public1/home/ungradu/home/gra02/anaconda3/envs/mind-vis/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in fit
self._call_and_handle_interrupt(
File "/public1/home/ungradu/home/gra02/anaconda3/envs/mind-vis/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
File "/public1/home/ungradu/home/gra02/anaconda3/envs/mind-vis/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/spawn.py", line 78, in launch
mp.spawn(
File "/public1/home/ungradu/home/gra02/anaconda3/envs/mind-vis/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/public1/home/ungradu/home/gra02/anaconda3/envs/mind-vis/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 189, in start_processes
process.start()
File "/public1/home/ungradu/home/gra02/anaconda3/envs/mind-vis/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/public1/home/ungradu/home/gra02/anaconda3/envs/mind-vis/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/public1/home/ungradu/home/gra02/anaconda3/envs/mind-vis/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/public1/home/ungradu/home/gra02/anaconda3/envs/mind-vis/lib/python3.8/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/public1/home/ungradu/home/gra02/anaconda3/envs/mind-vis/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/public1/home/ungradu/home/gra02/anaconda3/envs/mind-vis/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'TorchHistory.add_log_parameters_hook..'

p.s.
before the error info there is also a userwarning about wandb, but I guess it is not the cause of this problem.

Thanks for all your efforts!

When running "stageA2_mbm_finetune" with the GOD dataset, the obtained correlations are very small (approximately 0.02). When I execute "stageA2_mbm_finetune" using the following command, the correlations obtained on the GOD test dataset are extremely low, as shown in the figure below, reaching a maximum of only around 0.02. Is this a normal result?

python code/stageA2_mbm_finetune.py --dataset GOD --pretrain_mbm_path results/pretrains/GOD/fmri_encoder.pth

This is only questions.
I'm not professional on deep learning. I didnt read the paper.
I have 2 questions.
1, is this technologe "one neural nets for all" or one neural nets for capita?
If this is a one for all tech, it means, if I buy some gear to read my brain no matter it's an inplant(like Neuralink) or helmet(a lot prototypes we can find on youtube), all I need to do is download this neural nets. Then everything is good to go.
If this is a per capita tech, it means, I have to train a neural nets for my self, and I have to fine tune it periodically. If I change to another gear, I probably need to do all the jobs again.

2, have you test this with some artists who work on photo-realistic digital painting?
I've trained on this area for like 2,3 years. I believe this hard training change the way my brain handle vision stimuli a lot. General people can't handle the rotation in vision, this is because our brain prefer to handle tokens to shapes. Tokens don't have directions, or even precise length or ratio. You can find some training tool in my github which helps with such trainings, I had a hard time training on these, really hard.

Let me provide a suggestion. You can test with some basic geometry shapes. This could probably provide the idea how brain deal with simple images, and basic elements. Shape, count, color, texture, ratio, relative pos.

Thank you for your contribution to the project. We encountered the following issue while running the code: in code/eval_metrics.py, line 119, in n_way_top_k_acc, acc = accuracy(pred_picked.unsqueeze(0), torch.tensor([0], device=pred.device), top_k=top_k) TypeError: accuracy() missing 1 required positional argument: 'task'. Could you please advise on how to fix this?

Hi

I'm reproducing this very interesting project. However, I don't know how to get the HCP dataset with the "HCP_visual_voxel.npz" to implement stageA1_mbm_pretrain.py. In fact, I was able to download the fMRI data in “nii.gz” format of the HCP 1200 dataset containing 1113 subjects. But I can't extract the ROI of the visual cortex from the nii file in 4-D and save it as HCP_visual_voxel.npz for each subject.

Can you provide more detailed implementation steps about HCP_visual_voxel.npz？

Thanks

Congratulations that your impressive work has been published on CVPR2023.

Your paper supplementary file wrote: "Data locality is included in the representations by dividing the fMRI time series into patches and transforming patches into embeddings." But both Bold5000 and GOD datasets don't have the time series. Your work seems to be more likely dividing voxels into different patches.

This question really troubles me, could you explain it to me which can help me have a better understanding of this?

In stageA2_mbm_finetune.py, I found finetuning seemed to happen on test set. Am I mistaken?