Skip to content

add FastSpeech2 with internal alignment - #2119

Merged
mravanelli merged 18 commits into
speechbrain:developfrom
BenoitWang:fs2_internal_alignment
Sep 25, 2023
Merged

mravanelli merged 18 commits into
speechbrain:developfrom
BenoitWang:fs2_internal_alignment

Conversation

@BenoitWang

@BenoitWang BenoitWang commented Aug 11, 2023

Copy link
Copy Markdown
Collaborator

Contribution in a nutshell

This recipe allows training FastSpeech2 without forced aligner referring to One TTS Alignment To Rule Them All. The alignment can be learned by an internal alignment network that is added to FastSpeech2.

This PR aims to:

  1. Provide a better alignment for punctuation
  2. Facilitates FastSpeech2 training when using custom data (everything can be done inside speechbrain).

As for the phonemes, they are automatically obtained with the speechbrain g2p model during preparation.

Scope

  • Benchmark the training period with binary alignment loss
  • Benchmark if a higher dropout is needed
  • Docstrings and examples
  • Dropbox + HF

@mravanelli
mravanelli self-requested a review August 13, 2023 15:33
@mravanelli mravanelli added the enhancement New feature or request label Aug 13, 2023
@BenoitWang
BenoitWang marked this pull request as ready for review September 21, 2023 15:46
@mravanelli

Copy link
Copy Markdown
Collaborator

Thank you, @BenoitWang , for your PR contribution! t's great to see this valuable feature being added to the FastSpeech model.
I'm running the recipe, but I still have some comments to share with you:

  1. Recipe Tests:
    The recipe tests are currently failing. Please run the following command to check:
python -c 'from tests.utils.recipe_tests import run_recipe_tests; print("TEST FAILED!") if not(run_recipe_tests(filters_fields=["Dataset"], filters=[["LJSpeech"]], do_checks=False, run_opts="--device=cuda")) else print("TEST PASSED")'

I encountered this error:

RuntimeError: These keys are still unaccounted for in the data pipeline: phonemes
  1. FastSpeech2.py:
    In the speechbrain/lobes/models/FastSpeech2.py file, please consider the following improvements:
  • Add a docstring example for the LossWithAlignment class.
  • Describe the arguments and add a docstring example for the ForwardSumLoss class.
  • Describe the arguments and add a docstring example for the BinaryAlignmentLoss class.
  • Add a docstring example for the maximum_path_numpy function.
  • Consider whether it makes sense to write a docstring example for the TextMelCollateWithAlignment class.
  1. interfaces.py:
    In the speechbrain/pretrained/interfaces.py file, I noticed the addition of FastSpeech2InternalAlignment. Is there are reason for that? Ideally, we should strive to reuse the existing FastSpeech2 interface if possible and convenient.

  2. text_to_sequence.py:
    For the g2p_keep_punctuations function in speechbrain/utils/text_to_sequence.py, please describe the arguments and add a docstring for clarity.

@BenoitWang

Copy link
Copy Markdown
Collaborator Author

Hi @mravanelli , thanks for the comments! I've fixed the recipe test and added the necessary docstrings and examples.
However, for LossWithAlignment, since it's a summary of the other losses, an example takes too many inputs and seems repeated because an example can be found in each component loss.
As for the new interface, I created a new one mainly because the outputs of the models are different, the internal alignment model outputs the alignments in addition. The inference processes are very different as well, for example the original one needs the spn prediction while the new one focuses on keeping the punctuations during g2p.

@mravanelli

Copy link
Copy Markdown
Collaborator

Thank you @BenoitWang. I did the last tests and everything seems to work properly. Thank you for this new model that improves SpeechBrain's TTS systems currently available.

@mravanelli
mravanelli merged commit 233e72c into speechbrain:develop Sep 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants