Load audio with PyAV - #2354
Load audio with PyAV#2354pplantinga wants to merge 2 commits into
Conversation
|
I tried this PR to load To reproduce: and then: for _ in range(10):
load_audio_pyav("example_rw.mp3", normalize=False)[0]I will try to use pyav on common voice and see how it behaves compared to Torchaudio since creating a new conda env in my cluster may lead to issues with torchaudio and mp3. |
| elif frame_offset > 0: | ||
| audio = audio[:, frame_offset:] | ||
|
|
||
| return torch.tensor(audio), sample_rate |
There was a problem hiding this comment.
I think using torch.from_numpy method instead of torch.tensor would be better (in terms of readability/torch.from_numpy infers the dtype of your audio file while torch.tensor is an alias of torch.FloatTensor)
| if dtype.kind == "i" and normalize: | ||
| audio = audio.astype(np.float32) / np.iinfo(dtype).max | ||
|
|
||
| if num_frames >= 0: |
There was a problem hiding this comment.
Not sure how much of a proof-of-concept this currently is but I'll leave some comments anyway.
Would it be possible to implement frame_offset/num_frames without needing to decode the entire file? Certain datasets I have encountered have long files with utterances representing a very small segment of the file.
There was a problem hiding this comment.
I did think about this, and its probably important to have it, but I'm not sure how straightforward it is to do it for compressed audio files.
| logger = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| def load_audio_pyav(uri, num_frames=-1, frame_offset=0, normalize=True): |
There was a problem hiding this comment.
Maybe we could have a more generic load_audio function in SB that is not necessarily tied to PyAV and could ultimately be made to support different backends (e.g. still allowing to use torchaudio if desired)?
In theory PyAV should already sort of cover our usecases but it would still be more future-proof, I think.
There was a problem hiding this comment.
I do like this. In lhoste they have a function that switch the backend if one fails:
@lru_cache(maxsize=1)
def get_default_audio_backend() -> "AudioBackend":
"""
Return a backend that can be used to read all audio formats supported by Lhotse.
It first looks for special cases that need very specific handling
(such as: opus, sphere/shorten, in-memory buffers)
and tries to match them against relevant audio backends.
Then, it tries to use several audio loading libraries (torchaudio, soundfile, audioread).
In case the first fails, it tries the next one, and so on.
"""
return CompositeAudioBackend(
[
# First handle special cases: OPUS and SPHERE (SPHERE may be encoded with shorten,
# which can only be decoded by binaries "shorten" and "sph2pipe").
FfmpegSubprocessOpusBackend(),
Sph2pipeSubprocessBackend(),
# New FFMPEG backend available only in torchaudio 2.0.x+
TorchaudioFFMPEGBackend(),
# Prefer libsndfile for in-memory buffers only
LibsndfileBackend(),
# Torchaudio should be able to deal with most audio types...
TorchaudioDefaultBackend(),
# ... if not, try audioread...
AudioreadBackend(),
# ... oops.
]
)
see: https://github.com/lhotse-speech/lhotse/blob/master/lhotse/audio/backend.py
We could have something more generic like that.
There was a problem hiding this comment.
I am against this option. More backend means more maintenance. We already had to manage one until now, it's already causing a mess.
There was a problem hiding this comment.
I think I'm with Titouan on this. Perhaps its worth noting that users are free to use lhotse with speechbrain if they want the more flexible audio loading.
|
I will try it. Something important to do would be to benchmark the reading speed of this lib. Some backend of torchaudio were awfully slow for instance leading to slower training times. |
I have seen this warning before, but as far as I know it can be ignored. Perhaps we can find a way to suppress it. |
The problem is that the warning shows on every call so having the user ignore it is not really an option. IMO the warning underlines a valid concern so the user should still be made aware of it in some way. |
|
I think we can close this PR as we decided to not support for now PyAV as it is slower than torchaudio. Free free to reopen it in the future if you find the integration with PyAV valuable. |
Torchaudio depends on one of a few backends being installed, which has caused a number of problems. See #2225
This PR solves it by using PyAV to load the audio, because it bundles ffmpeg and makes for easy install.
https://pyav.org/docs/stable/
This is the approach taken by faster whisper (see https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/audio.py#L1)