speechbrain · Adel-Moumen · Nov 28, 2024 · Feb 8, 2024 · Feb 8, 2024 · Feb 8, 2024
diff --git a/recipes/PeoplesSpeech/ASR/transformer/README.md b/recipes/PeoplesSpeech/ASR/transformer/README.md
@@ -0,0 +1,65 @@
+# How to run an ASR experiment with People's Speech
+
+This recipe provides the necessary recipe to train a speech recognizer on the People's Speech dataset.
+
+## Downloading the dataset
+
+The full dataset will occupy around 3 TB of storage and must be obtained following the standard HuggingFace
+dataset process in the [corresponding HuggingFace people's speech repository](https://huggingface.co/datasets/MLCommons/peoples_speech).
+
+By default, our recipe set the HuggingFace environmental variable *HF_DATASETS_OFFLINE* to disable the download
+of the dataset. This is because we ask the user to download it before, like for any other recipe. Indeed, we do
+not want a recipe script, potentially run on GPUs nodes, to take hours downloading a dataset first.
+
+**One must download the wanted train subset as well as the validation and test splits.**
+
+## Important note on the data
+
+[People's speech](https://arxiv.org/pdf/2111.09344) is a very challenging dataset containing two main sets 'clean' and 'dirty' totalising 28,000 hours of error-prone transcribed speech. From our experience, it is most likely that this dataset should not be utilised alone, as the training material is very hard. It mostly contains spontaneous speech, with a few errors in transcription alignments and a lot of transcription inconsistency e.g. sometimes transcribing filler words, sometimes not - sometimes transcribing repetition, sometimes not. This makes the models trained on this data fairly hard to evaluate... The provided validation and test sets seem to be also a bit out of domain.
+
+However, this dataset remain very valuable due to the high quantity of spontaneous speech provided.
+
+## People's Speech Clean vs Dirty
+
+The clean subset contains 5.8k hours of speech while the dirty one around 22k hours. According to the authors, samples in the clean subset exhibits a CER lower or equal to 20% on their transcriptions compared to pseudo-labels obtained from an external ASR model. The dirty set is for samples with between 20% and 38% of CER. This confirm that even the clean subset is very 'noisy' from the perspective of the quality of the labels.
+
+## Results
+
+It is tricky to evaluate this dataset. Here we provide the results on the official validation
+and test split as well as Voxpopuli test set and the test-clean from LibriSpeech. It is important
+to remember that the results for the two last sets are out-of-domain and vocabulary for the
+tokenizer.
+
+| hyperparams file | validation WER | test WER | VoxPopuli | LibriSpeech test-clean WER | GPUs |
+|:-------------:|:-------------:|:-------------:|:---------------------------:| :-----:| :-----:|
+| conformer_large.yaml | 28.44 | 31.04 | 20.0 | 9.45 | 8xA100 80GB |
+
+# **About SpeechBrain**
+- Website: https://speechbrain.github.io/
+- Code: https://github.com/speechbrain/speechbrain/
+- HuggingFace: https://huggingface.co/speechbrain/
+
+# **Citing SpeechBrain**
+Please, cite SpeechBrain if you use it for your research or business.
+
+```bibtex
+@misc{speechbrainV1,
+  title={Open-Source Conversational AI with SpeechBrain 1.0},
+  author={Mirco Ravanelli and Titouan Parcollet and Adel Moumen and Sylvain de Langen and Cem Subakan and Peter Plantinga and Yingzhi Wang and Pooneh Mousavi and Luca Della Libera and Artem Ploujnikov and Francesco Paissan and Davide Borra and Salah Zaiem and Zeyu Zhao and Shucong Zhang and Georgios Karakasidis and Sung-Lin Yeh and Pierre Champion and Aku Rouhe and Rudolf Braun and Florian Mai and Juan Zuluaga-Gomez and Seyed Mahed Mousavi and Andreas Nautsch and Xuechen Liu and Sangeet Sagar and Jarod Duret and Salima Mdhaffar and Gaelle Laperriere and Mickael Rouvier and Renato De Mori and Yannick Esteve},
+  year={2024},
+  eprint={2407.00463},
+  archivePrefix={arXiv},
+  primaryClass={cs.LG},
+  url={https://arxiv.org/abs/2407.00463},
+}
+@misc{speechbrain,
+  title={{SpeechBrain}: A General-Purpose Speech Toolkit},
+  author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},
+  year={2021},
+  eprint={2106.04624},
+  archivePrefix={arXiv},
+  primaryClass={eess.AS},
+  note={arXiv:2106.04624}
+}
+```
+
diff --git a/recipes/PeoplesSpeech/ASR/transformer/extra_requirements.txt b/recipes/PeoplesSpeech/ASR/transformer/extra_requirements.txt
@@ -0,0 +1,3 @@
+datasets==3.1.0
+librosa
+soundfile
diff --git a/recipes/PeoplesSpeech/ASR/transformer/hparams/conformer_large.yaml b/recipes/PeoplesSpeech/ASR/transformer/hparams/conformer_large.yaml
@@ -0,0 +1,284 @@
+# ############################################################################
+# Model: E2E ASR with Transformer
+# Encoder: Conformer Encoder
+# Decoder: Transformer Decoder + (CTC/ATT joint) beamsearch + TransformerLM
+# Tokens: BPE
+# losses: CTC + KLdiv (Label Smoothing loss)
+# Training: People's Speech 28,000 hours
+# Authors:  Titouan Parcollet
+# ############################################################################
+
+# Seed needs to be set at top of yaml, before objects with parameters are made
+seed: 3407
+__set_seed: !apply:speechbrain.utils.seed_everything [!ref <seed>]
+output_folder: !ref results/conformer_transducer_large/<seed>
+output_wer_folder: !ref <output_folder>/
+save_folder: !ref <output_folder>/save
+train_csv: !ref <save_folder>/train.csv
+train_log: !ref <output_folder>/train_log.txt
+
+# Data files
+hf_download_folder: !PLACEHOLDER
+subsets: ["clean", "dirty"] # _sa sets are removed.
+ckpt_interval_minutes: 5 # save checkpoint every N min
+
+
+####################### Training Parameters ####################################
+
+number_of_epochs: 30
+optimizer_step_limit: 400000
+ctc_weight: 0.3
+grad_accumulation_factor: 1
+max_grad_norm: 5.0
+loss_reduction: 'batchmean'
+sorting: random
+num_workers: 4
+precision: fp16 # bf16, fp16 or fp32
+skip_prep: False
+
+# stages related parameters
+lr_adam: 0.0008
+
+# Feature parameters
+sample_rate: 16000
+n_fft: 512
+n_mels: 80
+win_length: 32
+
+# This setup works well for A100 80GB GPU, adapts it to your needs.
+# Or turn it off (but training speed will decrease)
+max_batch_length_train: 500
+max_batch_length_val: 100 # we reduce it as the beam is much wider (VRAM)
+num_bucket: 200
+shuffle: True # if true re-creates batches at each epoch shuffling examples.
+batch_ordering: random
+max_batch_ex: 256
+
+# BPE parameters
+token_type: bpe  # ["unigram", "bpe", "char"]
+character_coverage: 1.0
+
+dynamic_batch_sampler:
+    max_batch_length: !ref <max_batch_length_train>
+    num_buckets: !ref <num_bucket>
+    shuffle: !ref <shuffle>
+    batch_ordering: !ref <batch_ordering>
+    max_batch_ex: !ref <max_batch_ex>
+
+dynamic_batch_sampler_valid:
+    max_batch_length: !ref <max_batch_length_val>
+    num_buckets: !ref <num_bucket>
+    shuffle: !ref <shuffle>
+    batch_ordering: !ref <batch_ordering>
+    max_batch_ex: !ref <max_batch_ex>
+
+# Dataloader options
+train_dataloader_opts:
+    shuffle: True
+    num_workers: !ref <num_workers>
+
+valid_dataloader_opts:
+    num_workers: !ref <num_workers>
+
+test_dataloader_opts:
+    batch_size: 4
+
+####################### Model Parameters #######################################
+
+# Transformer
+d_model: 640
+nhead: 8
+num_encoder_layers: 14
+num_decoder_layers: 6
+d_ffn: 2048
+transformer_dropout: 0.1
+activation: !name:speechbrain.nnet.activations.Swish
+output_neurons: 5120
+
+# Outputs
+label_smoothing: 0.1
+bos_index: 1
+eos_index: 2
+
+# Decoding parameters
+min_decode_ratio: 0.0
+max_decode_ratio: 1.0
+valid_search_interval: 10
+valid_beam_size: 5
+test_beam_size: 10
+ctc_weight_decode: 0.3
+
+############################## Models ##########################################
+
+CNN: !new:speechbrain.lobes.models.convolution.ConvolutionFrontEnd
+    input_shape: (8, 10, 80)
+    num_blocks: 2
+    num_layers_per_block: 1
+    out_channels: (64, 32)
+    kernel_sizes: (3, 3)
+    strides: (2, 2)
+    residuals: (False, False)
+
+Transformer: !new:speechbrain.lobes.models.transformer.TransformerASR.TransformerASR # yamllint disable-line rule:line-length
+    input_size: 640
+    tgt_vocab: !ref <output_neurons>
+    d_model: !ref <d_model>
+    nhead: !ref <nhead>
+    num_encoder_layers: !ref <num_encoder_layers>
+    num_decoder_layers: !ref <num_decoder_layers>
+    d_ffn: !ref <d_ffn>
+    dropout: !ref <transformer_dropout>
+    activation: !ref <activation>
+    conformer_activation: !ref <activation>
+    encoder_module: conformer
+    attention_type: RelPosMHAXL
+    normalize_before: True
+    causal: False
+
+ctc_lin: !new:speechbrain.nnet.linear.Linear
+    input_size: !ref <d_model>
+    n_neurons: !ref <output_neurons>
+
+seq_lin: !new:speechbrain.nnet.linear.Linear
+    input_size: !ref <d_model>
+    n_neurons: !ref <output_neurons>
+
+normalize: !new:speechbrain.processing.features.InputNormalization
+    norm_type: global
+    update_until_epoch: 2
+
+modules:
+    CNN: !ref <CNN>
+    Transformer: !ref <Transformer>
+    seq_lin: !ref <seq_lin>
+    ctc_lin: !ref <ctc_lin>
+    normalize: !ref <normalize>
+
+# define two optimizers here for two-stage training
+Adam: !name:torch.optim.AdamW
+    lr: !ref <lr_adam>
+    betas: (0.9, 0.98)
+    eps: 0.000000001
+    weight_decay: 0.001
+
+model: !new:torch.nn.ModuleList
+    - [!ref <CNN>, !ref <Transformer>, !ref <seq_lin>, !ref <ctc_lin>]
+
+####################### Decoding & optimiser ###########################
+
+ctc_scorer: !new:speechbrain.decoders.scorer.CTCScorer
+    eos_index: !ref <eos_index>
+    blank_index: 0
+    ctc_fc: !ref <ctc_lin>
+
+
+scorer_test_search: !new:speechbrain.decoders.scorer.ScorerBuilder
+    full_scorers: [!ref <ctc_scorer>]
+    weights:
+        ctc: !ref <ctc_weight_decode>
+
+scorer_valid_search: !new:speechbrain.decoders.scorer.ScorerBuilder
+    full_scorers: [!ref <ctc_scorer>]
+    weights:
+        ctc: !ref <ctc_weight_decode>
+
+valid_search: !new:speechbrain.decoders.S2STransformerBeamSearcher
+    modules: [!ref <Transformer>, !ref <seq_lin>]
+    bos_index: !ref <bos_index>
+    eos_index: !ref <eos_index>
+    min_decode_ratio: !ref <min_decode_ratio>
+    max_decode_ratio: !ref <max_decode_ratio>
+    beam_size: !ref <valid_beam_size>
+    using_eos_threshold: False
+    length_normalization: True
+    scorer: !ref <scorer_valid_search>
+
+test_search: !new:speechbrain.decoders.S2STransformerBeamSearcher
+    modules: [!ref <Transformer>, !ref <seq_lin>]
+    bos_index: !ref <bos_index>
+    eos_index: !ref <eos_index>
+    min_decode_ratio: !ref <min_decode_ratio>
+    max_decode_ratio: !ref <max_decode_ratio>
+    beam_size: !ref <test_beam_size>
+    temperature: 1.15
+    using_eos_threshold: False
+    length_normalization: True
+    scorer: !ref <scorer_test_search>
+
+log_softmax: !new:torch.nn.LogSoftmax
+    dim: -1
+
+ctc_cost: !name:speechbrain.nnet.losses.ctc_loss
+    blank_index: 0
+    reduction: !ref <loss_reduction>
+
+seq_cost: !name:speechbrain.nnet.losses.kldiv_loss
+    label_smoothing: !ref <label_smoothing>
+    reduction: !ref <loss_reduction>
+
+noam_annealing: !new:speechbrain.nnet.schedulers.NoamScheduler
+    lr_initial: !ref <lr_adam>
+    n_warmup_steps: 40000
+
+checkpointer: !new:speechbrain.utils.checkpoints.Checkpointer
+    checkpoints_dir: !ref <save_folder>
+    recoverables:
+        model: !ref <model>
+        noam_scheduler: !ref <noam_annealing>
+        normalizer: !ref <normalize>
+        counter: !ref <epoch_counter>
+
+epoch_counter: !new:speechbrain.utils.epoch_loop.EpochCounter
+    limit: !ref <number_of_epochs>
+
+############################## Augmentations ###################################
+
+# Speed perturbation
+speed_perturb: !new:speechbrain.augment.time_domain.SpeedPerturb
+    orig_freq: !ref <sample_rate>
+    speeds: [95, 100, 105]
+
+# Time Drop
+time_drop: !new:speechbrain.augment.freq_domain.SpectrogramDrop
+    drop_length_low: 15
+    drop_length_high: 25
+    drop_count_low: 1
+    drop_count_high: 2
+    replace: "mean"
+
+# Freq Drop
+freq_drop: !new:speechbrain.augment.freq_domain.SpectrogramDrop
+    drop_length_low: 10
+    drop_length_high: 20
+    drop_count_low: 1
+    drop_count_high: 2
+    replace: "mean"
+    dim: 2
+
+# Time warp
+time_warp: !new:speechbrain.augment.freq_domain.Warping
+
+# We remove augmentations as we are using the full 28k hour set.
+# Uncomment to bring augmentations back!
+# fea_augment: !new:speechbrain.augment.augmenter.Augmenter
+#    min_augmentations: 2
+#    max_augmentations: 2
+#    augment_prob: 1.0
+#    augmentations: [
+#        !ref <time_drop>,
+#        !ref <freq_drop>,
+#        !ref <time_warp>]
+
+compute_features: !new:speechbrain.lobes.features.Fbank
+    sample_rate: !ref <sample_rate>
+    n_fft: !ref <n_fft>
+    n_mels: !ref <n_mels>
+    win_length: !ref <win_length>
+
+############################## Logging and Pretrainer ##########################
+
+train_logger: !new:speechbrain.utils.train_logger.FileTrainLogger
+    save_file: !ref <train_log>
+
+error_rate_computer: !name:speechbrain.utils.metric_stats.ErrorRateStats
+acc_computer: !name:speechbrain.utils.Accuracy.AccuracyStats
diff --git a/recipes/PeoplesSpeech/ASR/transformer/peoples_speech_prepare.py b/recipes/PeoplesSpeech/ASR/transformer/peoples_speech_prepare.py
@@ -0,0 +1 @@
+../../peoples_speech_prepare.py