Skip to content

[Bug]: RuntimeError: The size of tensor a (140590) must match the size of tensor b (2500) at non-singleton dimension 1 #2071

Description

@fx-walsh

Describe the bug

I see a similar issue was raised before but didn't have enough detail to answer it. #827

When running the following code:

from speechbrain.pretrained import EncoderDecoderASR

asr_model = EncoderDecoderASR.from_hparams(source="speechbrain/asr-transformer-transformerlm-librispeech", savedir="pretrained_models/asr-transformer-transformerlm-librispeech")
trans = asr_model.transcribe_file("my/local/path/file.wav")

I get this error:

RuntimeError                              Traceback (most recent call last)
Cell In[10], line 3
      1 with torch.no_grad():
      2     wav_lens = rel_length.to(asr_model.device)
----> 3     encoder_out = asr_model.encode_batch(batch, rel_length)

File [c:\Users\francisw\Documents\repos\speech-to-text\.venv\Lib\site-packages\speechbrain\pretrained\interfaces.py:604](file:///C:/Users/francisw/Documents/repos/speech-to-text/.venv/Lib/site-packages/speechbrain/pretrained/interfaces.py:604), in EncoderDecoderASR.encode_batch(self, wavs, wav_lens)
    602 wavs = wavs.float()
    603 wavs, wav_lens = wavs.to(self.device), wav_lens.to(self.device)
--> 604 encoder_out = self.mods.encoder(wavs, wav_lens)
    605 return encoder_out

File [c:\Users\francisw\Documents\repos\speech-to-text\.venv\Lib\site-packages\torch\nn\modules\module.py:1501](file:///C:/Users/francisw/Documents/repos/speech-to-text/.venv/Lib/site-packages/torch/nn/modules/module.py:1501), in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File [c:\Users\francisw\Documents\repos\speech-to-text\.venv\Lib\site-packages\speechbrain\nnet\containers.py:191](file:///C:/Users/francisw/Documents/repos/speech-to-text/.venv/Lib/site-packages/speechbrain/nnet/containers.py:191), in LengthsCapableSequential.forward(self, x, lengths)
    189     x = layer(x, lengths=lengths)
    190 else:
--> 191     x = layer(x)
    192 if isinstance(x, tuple):
    193     x = x[0]

File [c:\Users\francisw\Documents\repos\speech-to-text\.venv\Lib\site-packages\torch\nn\modules\module.py:1501](file:///C:/Users/francisw/Documents/repos/speech-to-text/.venv/Lib/site-packages/torch/nn/modules/module.py:1501), in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File [c:\Users\francisw\Documents\repos\speech-to-text\.venv\Lib\site-packages\speechbrain\lobes\models\transformer\TransformerASR.py:348](file:///C:/Users/francisw/Documents/repos/speech-to-text/.venv/Lib/site-packages/speechbrain/lobes/models/transformer/TransformerASR.py:348), in EncoderWrapper.forward(self, x, wav_lens)
    346 def forward(self, x, wav_lens=None):
    347     """ Processes the input tensor x and returns an output tensor."""
--> 348     x = self.transformer.encode(x, wav_lens)
    349     return x

File [c:\Users\francisw\Documents\repos\speech-to-text\.venv\Lib\site-packages\speechbrain\lobes\models\transformer\TransformerASR.py:301](file:///C:/Users/francisw/Documents/repos/speech-to-text/.venv/Lib/site-packages/speechbrain/lobes/models/transformer/TransformerASR.py:301), in TransformerASR.encode(self, src, wav_len)
    298     pos_embs_source = self.positional_encoding(src)
    300 elif self.positional_encoding_type == "fixed_abs_sine":
--> 301     src = src + self.positional_encoding(src)
    302     pos_embs_source = None
    304 encoder_out, _ = self.encoder(
    305     src=src,
    306     src_key_padding_mask=src_key_padding_mask,
    307     pos_embs=pos_embs_source,
    308 )

RuntimeError: The size of tensor a (140590) must match the size of tensor b (2500) at non-singleton dimension 1

I'm not sure this is necessarily a bug or just that I need to do some additional pre-processing of the wav file that I'm using.

When I run the lines leading up to the encoder_out = asr_model.encode_batch(batch, rel_length) line, the dimension of the batch that I'm passing to encode_batch is [1, 89977000].

waveform = asr_model.load_audio("my/local/path/file.wav")
batch = waveform.unsqueeze(0)
rel_length = torch.tensor([1.0])
print(batch.shape)
torch.Size([1, 89977000])

When inspecting the properties of my wave file, I see it has a bit rate of 705kbps. As I understand it, load_file should normalize this to meet the sampling used in the pre-trained model, but I think this part seems to be having issues? At least that's my best guess.

Any help would be greatly appreciated, thanks!

Expected behaviour

The audio file should be transcribed to text.

To Reproduce

No response

Versions

No response

Relevant log output

No response

Additional context

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    confirmedBug officially confirmed or reproducible; discuss resolution or start writing PRenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions