Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training
- Install the custom version of ESPnet:
git clone --branch v.202304 --depth 1 [email protected]:espnet/espnet.git /path/to/espnet-bloomzmms
- Copy the modifications:
rsync -av espnet-bloomzmms-files/ /path/to/espnet-bloomzmms/- Follow the ESPnet installation instructions.
- Run the data preparation stages:
cd /path/to/espnet-bloomzmms/egs2/fleurs/asr1
./run.sh --stop-stage 5- Download the synthetic multi-instractional training targets and create additional training data directories:
wget 'https://zenodo.org/records/10900287/files/tmi_training_targets.tar.gz?download=1' -O downloads/tmi_training_targets.tar.gz
mkdir downloads/tmi_training_targets
tar xf downloads/tmi_training_targets.tar.gz -C downloads/tmi_training_targets
mkdir dump/raw/train_tmi_sp dump/raw/train_mi_sp
cp -v downloads/tmi_training_targets/* dump/raw/train_tmi_sp/
cp -v dump/raw/train_t_sp/feats_type dump/raw/train_tmi_sp/
for f in wav.scp utt2spk; do
for i in {1..3}; do
perl -p -e 's/^(\S+)/$1-'$i'/' dump/raw/train_t_sp/$f >> dump/raw/train_tmi_sp/$f
done
done
utils/fix_data_dir.sh dump/raw/train_tmi_sp
cp -v downloads/tmi_training_targets/* dump/raw/train_mi_sp/
cp -v dump/raw/train_t_sp/feats_type dump/raw/train_mi_sp/
for f in wav.scp utt2spk; do
for i in {1..2}; do
perl -p -e 's/^(\S+)/$1-'$i'/' dump/raw/train_t_sp/$f >> dump/raw/train_mi_sp/$f
done
done
utils/fix_data_dir.sh dump/raw/train_mi_spNote: "pretrained" models in the following commands are outputs of the previous steps, it's not necessary to download the pretrained models from the next section to perform training. If you wish to skip training, you can download the pretrained models from the next section.
- CTC pretraining:
./run.sh \
--stage 6 \
--stop-stage 11 \
--pretrained_model downloads/bloomz_token_embeddings.pth- AED training:
./run.sh \
--stage 11 \
--stop-stage 11 \
--ngpu 4 \
--asr_config conf/tuning/train_asr_e_branchformer_mms1b-asr_bloomz7b_aed.yaml \
--train_set train_tmi \
--auxiliary_data_tags "decoder_prefix decoder_postfix" \
--train_text_type text_int \
--pretrained_model exp/asr_train_asr_e_branchformer_mms1b-asr_bloomz7b_ctc_raw_hugging_face_bigscience-bloomz-7b1_sp/valid.cer_ctc.best.pth:::ctc- CTC: akreal/bloomzmms-ctc
- CE:
./run.sh \
--stage 12 \
--asr_config conf/tuning/train_asr_e_branchformer_mms1b-asr_bloomz7b_aed.yaml- Run the data preparation steps.
For CoVoST 2, run:
./run.sh \
--stage 3 \
--stop-stage 3 \
--test_sets test_covost2_en-deFor FLEURS, run:
local/prepare_fleurs_translate.py en-us de-de- Generate a config with language-specific instructions:
perl -p \
-e 's/prefix: "Repeat the sentence: "/prefix: "Translate the following text from English to German\\n"/;' \
-e 's/postfix: ". "/postfix: "\\n"/;' -e 's/keep_nbest_models: 3/keep_nbest_models: 1/;' \
-e 's/max_epoch: 70/max_epoch: 0/;' \
conf/tuning/train_asr_e_branchformer_mms1b-asr_bloomz7b_aed.yaml \
> conf/tuning/train_asr_e_branchformer_mms1b-asr_bloomz7b_aed_translate_en-de.yaml- Run the model construction and inference stages:
./run.sh \
--stage 11 \
--test_sets test_covost2_en-de \
--pretrained_model exp/asr_train_asr_e_branchformer_mms1b-asr_bloomz7b_aed_raw_hugging_face_bigscience-bloomz-7b1_sp/valid.acc.ave.pth
--asr_config conf/tuning/train_asr_e_branchformer_mms1b-asr_bloomz7b_aed_translate_en-de.yaml-
Prepare the SpeechGLUE data: https://github.com/ashi-ta/speechGLUE?tab=readme-ov-file#data-preparation.
-
Run the inference script:
mkdir speechglue_output
. path.sh
for task in cola mnli_matched mnli_mismatched mrpc qnli qqp rte sst2 stsb wnli; do
local/speechglue_inference.py \
/path/to/speechGLUE/dump \
${task} \
exp/asr_train_asr_e_branchformer_mms1b-asr_bloomz7b_aed_raw_hugging_face_bigscience-bloomz-7b1_sp \
speechglue_output
done- Download the SpeechXNLI data:
wget https://zenodo.org/records/10900287/files/speech_xnli.tar.gz?download=1 -O speech_xnli.tar.gz
tar xvf speech_xnli.tar.gz- Run the inference script:
mkdir speechxnli_output
. path.sh
for lang in ar bg de el en es fr hi ru sw th tr ur vi zh; do
local/speechxnli_inference.py \
/path/to/speechXNLI \
${lang} \
exp/asr_train_asr_e_branchformer_mms1b-asr_bloomz7b_aed_raw_hugging_face_bigscience-bloomz-7b1_sp \
speechxnli_output
done@inproceedings{denisov-vu-2024-teaching,
title = "Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training",
author = "Denisov, Pavel and
Vu, Thang",
editor = "Duh, Kevin and
Gomez, Helena and
Bethard, Steven",
booktitle = "Findings of the Association for Computational Linguistics: NAACL 2024",
month = jun,
year = "2024",
address = "Mexico City, Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-naacl.52",
pages = "814--834",
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent