Hi Luo, thanks for your remarkable work! I am here wondering why use batch 128 in

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

About batch_size in training configuration? about clip4clip HOT 3 CLOSED

arrowluo commented on May 20, 2024

About batch_size in training configuration?

from clip4clip.

Comments (3)

ArrowLuo commented on May 20, 2024

Hi @xiao2mo, there is no special purpose. We set batch size 32 per GPU because some of our machines are 16G per card, and we need to test some other parameters like frame number. It is an appropriate batch size to finish the hyper-parameters study. If your card has more than 16G, a suggestion is to test a large frame number, then a large batch size.

from clip4clip.

xiao2mo commented on May 20, 2024

I see it. The main problem in my exps is that batch size in DDP configuration may result in different results. Thank u.

from clip4clip.

dengfenglai321 commented on May 20, 2024

hi
我这边是两张卡，每张卡是16G。
可是我能设置的batchsize是 16，请问这是为什么呢？

训练配置如下

01/10/2022 16:18:00 - INFO -   ***** Running test *****
01/10/2022 16:18:00 - INFO -     Num examples = 497
01/10/2022 16:18:00 - INFO -     Batch size = 16
01/10/2022 16:18:00 - INFO -     Num steps = 32
01/10/2022 16:18:00 - INFO -   ***** Running val *****
01/10/2022 16:18:00 - INFO -     Num examples = 497
222
333
01/10/2022 16:18:12 - INFO -   ***** Running training *****
01/10/2022 16:18:12 - INFO -     Num examples = 130260
01/10/2022 16:18:12 - INFO -     Batch size = 16
01/10/2022 16:18:12 - INFO -     Num steps = 40705
01/10/2022 16:21:34 - INFO -   Epoch: 1/5, Step: 50/8141, Lr: 0.000000001, Loss: 0.455173, Time/step: 4.041784

用的资源如图：

运行命令如下

python -m torch.distributed.launch --nproc_per_node=2 \
main_task_retrieval.py --do_train --num_thread_reader=0 \
--epochs=5 --batch_size=16 --n_display=50 \
--output_dir ckpts/ckpt_msrvtt_retrieval_looseType \
--lr 1e-4 --max_words 32 --max_frames 12 --batch_size_val 16 \
--datatype msrvtt --expand_msrvtt_sentences  \
--feature_framerate 1 --coef_lr 1e-3 \
--freeze_layer_num 0  --slice_framepos 2 \
--loose_type --linear_patch 2d --sim_header meanP \
--pretrained_clip_name ViT-B/16

from clip4clip.

Recommend Projects