Check out my Speech-to-speech translation (in French) model: STST

• Introduction to audio data (time domain, frequency domain, (log-mel) spectrogram)
• Load and explore an audio dataset (🤗 Datasets library)
• Preprocessing audio data (resampling with 🤗 Datasets’ cast_column function & Audio module, filtering the dataset, pre-processing with 🤗 Transformers AutoFeatureExtractor/AutoProcessor)
• Streaming audio data

• Audio classification with a pipeline (🤗 Transformers pipeline)
• Automatic speech recognition with a pipeline
• Audio generation with a pipeline
• Hands-on exercise (ungraded)

• Refresher on transformer models (Waveform (Wav2Vec2, HuBERT/None) or Spectrogram (Whisper/SpeechT5) input/output)
• CTC (Connectionist Temporal Classification) architectures (CTC algorithm, Wav2Vec2, HuBERT, M-CTC-T)
• Seq2Seq architectures (Whisper, SpeechT5)
• Audio classification architectures (Audio Spectrogram Transformer)

• Pre-trained models for audio classification (Keyword Spotting, Language Identification, 0-shot Audio Classification w/ CLAP)
• Fine-tuning a model for music classification (DistilHuBERT on GTZAN dataset)
• Build a demo with Gradio
• Hands-on exercise (Graded)

• Pre-trained models for automatic speech recognition (Limitations of CTC, Graduation to Seq2Seq, especially Whisper)
• Choosing a dataset (Summary of popular datasets, link)
• Evaluation and metrics for speech recognition (Word Error rate (WER))
• How to fine-tune an ASR system with the Trainer API (Whisper on Common Voice 13 Dhivehi data, note for the Data Collator and evaluation metrics)
• Build a demo
• Hands-on exercise (Graded)

• Text-to-speech datasets (LJSpeech, Multilingual LibriSpeech, VCTK, LibriTTS)
• Pre-trained models for text-to-speech (SpeechT5 w/ HifiGAN, Bark, Massive Multilingual Speech (MMS))
• Fine-tuning SpeechT5
• Evaluating text-to-speech models
• Hands-on exercise (Graded)

• Speech-to-speech translation (Speech translation -> text-to-speech)
• Creating a voice assistant (Wake word detection -> Speech transcription -> Language model query -> Synthesise speech)
• Transcribe a meeting (Speaker diarization)
• Hands-on exercise (Graded)