Describe the bug
The pos here as well as the one bellow is unexpectedly zero-ed out by srcmask who has True values on pad positions instead of non-pad positions.
It is supposed to use srcmask_inverted when calling sinusoidal_positional_embed_encoder.
To prove this, just deliberately assign pos to zero with pos = 0 after sinusoidal_positional_embed_encoder , and you will get exactly the same output audio.
I suggest using length_to_mask() to get the mask instead of get_key_padding_mask, because a "<pad>" token in the middle of a sequence does need attention (this may happen for mel feature):
def length_to_mask(lens, max_len=None):
if max_len is None:
max_len = lens.max()
return torch.arange(max_len)[None, :] < lens[:, None]One more thing, I think a mask should have True values for non-pad positions as convention, otherwise we will name it "pad_mask".
Expected behaviour
pos should not be all 0.
To Reproduce
No response
Environment Details
No response
Relevant Log Output
No response
Additional Context
No response
Describe the bug
The
poshere as well as the one bellow is unexpectedly zero-ed out bysrcmaskwho hasTruevalues on pad positions instead of non-pad positions.It is supposed to use
srcmask_invertedwhen callingsinusoidal_positional_embed_encoder.To prove this, just deliberately assign
posto zero withpos = 0aftersinusoidal_positional_embed_encoder, and you will get exactly the same output audio.I suggest using
length_to_mask()to get the mask instead ofget_key_padding_mask, because a "<pad>" token in the middle of a sequence does need attention (this may happen for mel feature):One more thing, I think a mask should have
Truevalues for non-pad positions as convention, otherwise we will name it "pad_mask".Expected behaviour
posshould not be all 0.To Reproduce
No response
Environment Details
No response
Relevant Log Output
No response
Additional Context
No response