Describe the bug
Randomly get this error
OSError: [Errno 39] Directory not empty: 'location/to/CKPT'
and again when I look at the CKPT directory the optimizer.ckpt is still in there
Expected behaviour
I expect the CKPT to be entirely removed.
To Reproduce
No response
Environment Details
- Using DDP
- Happen on the SpeechBrain development branch an 1 other branch
Relevant Log Output
No response
Additional Context
I think this is just an issue with DDP that rarely happens. I think sometimes when running on DDP the program tries to remove the CKPT directory before it is has removed all the files inside of it.
I'm not too familiar with speech brain and DDP but maybe there could be a simple fix to wait or check if the CKPT folder is empty.
Describe the bug
Randomly get this error
OSError: [Errno 39] Directory not empty: 'location/to/CKPT'and again when I look at the CKPT directory the optimizer.ckpt is still in there
Expected behaviour
I expect the CKPT to be entirely removed.
To Reproduce
No response
Environment Details
Relevant Log Output
No response
Additional Context
I think this is just an issue with DDP that rarely happens. I think sometimes when running on DDP the program tries to remove the CKPT directory before it is has removed all the files inside of it.
I'm not too familiar with speech brain and DDP but maybe there could be a simple fix to wait or check if the CKPT folder is empty.