Skip to content

Fix attentive statistics pooling dtype mismatch with FP16 and BF16 - #3081

Open
Con-Benksl wants to merge 1 commit into
speechbrain:developfrom
Con-Benksl:fix/attentive-pooling-half-precision
Open

Con-Benksl wants to merge 1 commit into
speechbrain:developfrom
Con-Benksl:fix/attentive-pooling-half-precision

Conversation

@Con-Benksl

Copy link
Copy Markdown

What does this PR do?

Fixes #2544.

With FP16/BF16 inputs and weights, AttentiveStatisticsPooling computes its global statistics in FP32. Concatenating these statistics with the input promotes the attention input to FP32, so the following half-precision convolution raises a dtype mismatch.

Cast the concatenated attention input back to x.dtype. The global statistics still accumulate at their original precision, and the path without global context is unchanged. No new dependencies or API changes.

Validation (CPU, PyTorch 2.6.0):

  • New parameterized forward/backward test covers FP16, BF16 and FP32, with/without global context and supplied lengths. It checks known masked statistics, output dtype and finite gradients. Before the fix: 4 failed, 8 passed.
  • After the fix, the new test, existing pooling/CNN/normalization tests and ECAPA-TDNN doctests: 31 passed.
  • Independent local checks with nonzero attention gradients and 70,000 frames passed for all three dtypes.
  • pre-commit run -a, changed-file pre-commit and git diff --check passed.

The full test suite, GPU execution and pretrained-model/training recipes were not run. This fixes the pooling layer's reported mismatch; it does not claim to validate full ECAPA training in half precision.

Implemented and tested with OpenAI Codex, with independent read-only review by another Codex agent.

Before submitting
  • Read the contributor guideline.
  • This PR does one thing.
  • Documentation checked; the existing API/docstrings remain applicable.
  • Added necessary regression tests.
  • Verified new and related existing tests locally (scope above).
  • Listed breaking changes: none.
  • Followed project code style and conventions.

PR review

Reviewer checklist
  • Is this pull request ready for review?
  • Check that all items from Before submitting are resolved.
  • Make sure the title is self-explanatory and the description concisely explains the PR.
  • Add labels and milestones (and optionally projects) to classify the PR.
  • Confirm compatibility requirements.
  • Review the self-review checklist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dtype mismatch in AttentiveStatisticsPooling with FP16 training mode

1 participant