Add Mel-Band RoFormer model support - #1
Open
devin-ai-integration[bot] wants to merge 3 commits into
Open
Conversation
Implements MelBandRoformerModel for audio demixing/source separation. The model uses a transformer-based architecture with mel-scale frequency bands to separate audio sources (e.g., vocals from instrumental). Architecture: - STFT preprocessing (JS) - converts audio to frequency domain - Mel-band indexing (JS) - maps frequencies to mel-scale bands - Core transformer model (ONNX) - band_split -> transformers -> mask_estimators - Mask application + iSTFT (JS) - reconstructs separated audio Features: - Full complex STFT/iSTFT implementation in JavaScript - Stereo audio support with proper channel interleaving - Chunked processing with overlap-add for long audio files - Compatible with voc_fv7 and other Mel-Band RoFormer checkpoints Co-Authored-By: Raymond <20248577+javaarchive@users.noreply.github.com>
…freq Rewrote _applyMasks to match the PyTorch forward method exactly: 1. Build full interleaved STFT representation (2050 entries for stereo) 2. Scatter-add mask values back to full 2050 positions 3. Divide by num_bands_per_freq to average overlapping bands 4. Complex multiply full STFT by averaged mask 5. Split back to per-channel and perform iSTFT Co-Authored-By: Raymond <20248577+javaarchive@users.noreply.github.com>
Co-Authored-By: Raymond <20248577+javaarchive@users.noreply.github.com>
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Mel-Band RoFormer demixing model (STFT + ONNX masks + iSTFT)
Summary
This PR adds first-class support for a Mel-Band RoFormer audio source-separation (demixing) model in transformers.js:
MelBandRoformerModelwith an end-to-endseparate()API:freq_indices)sessionRunmel-band-roformermodel type in the model registry.packages/transformers/src/models/models.js.Files changed:
packages/transformers/src/models/mel_band_roformer/modeling_mel_band_roformer.js(new)packages/transformers/src/models/models.jspackages/transformers/src/models/registry.jsReview & Testing Checklist for Human
sessionRun(..., { input: ... })and expectingoutputs.masksmatches the exported ONNX model’s actual input/output names.(1, num_stems, channels, time), and sounds like vocal isolation.fft.fft.inverseTransform(...)access is stable/acceptable (uses an internal property throughanycasting).Notes