Is there a distributed version of the code? I'd like to reproduce the effect of t5 about resdsql HOT 4 CLOSED

If I want to use 4 consumer-grade graphics cards, can I do it this way?

just change --device "0,1,2,3"

python -u text2sql.py \
    --batch_size 6 \
    --gradient_descent_step 16 \
    --device "0,1,2,3" \
    --learning_rate 5e-5 \
    --epochs 128 \
    --seed 42 \
    --save_path "./models/text2sql-t5-3b" \
    --tensorboard_save_path "./tensorboard_log/text2sql-t5-3b" \
    --model_name_or_path "t5-3b" \
    --use_adafactor \
    --mode train \
    --train_filepath "./data/preprocessed_data/resdsql_train_spider.json"

However, it seems that this flag I saw in the code is the GPU's ID.

I would like to ask if there is a way to perform multi-card training and inference. thank you.

from resdsql.

There is currently no distributed version.

I've been busy lately, so maybe you can follow Huggingface Accelerate (https://huggingface.co/docs/accelerate/index) and implement the distributed code yourself. It's simple and easy to use.

from resdsql.

ok. thank you for your answer. i will try the t5-small to represent it.

from resdsql.

I have made a PR about dataParallel, which can accelerate the training speed. However, for running the larger model, you may have to use model parallel, which i will create a PR lately

from resdsql.