Wondering if this PR should attempt to address #1939 as well. The issue I guess is that on_stage_end(stage==valid) is only called on the main process, so the loop stops on the main process but not the other processes. This could be handled by synchronizing the relevant metric across all processes, but the difficulty is that on_stage_end is only called on a single process so I'm not sure how we'd receive the result on the non-main processes...

I think it makes sense to add the fix in this PR too.

Depends on #2053 or something like it

#2059 just landed, and I think it fixes the DDP bug without having to change anything, because now the update_metrics() gets called on all processes. This PR should be ready for review now by @Adel-Moumen @Gastron @mravanelli

I don't have easy access to multi-gpu machine right now, so if anyone else can test that would be welcome.

@Adel-Moumen, any news?

I don't know if it's because of EpochCounterWithStopper but I have an issue with the G2P recipe with DDP (didn't test with single GPU yet). Basically, the EpochCounterWithStopper is working properly. I have set the limit to be equal to 2 and I got the following message:

speechbrain.utils.epoch_loop - 2 epochs without improvement.
Patience of 2 is exhausted, stopping.
main - Attempting to restore checkpoint for step lexicon
speechbrain.utils.checkpoints - Loading a checkpoint from results/transformer/1234/save/CKPT+2023-08-09+12-06-58+00

However, now, my training is stopped. I can't do anything. Nothing show up. Additionally, the recipe doesn't seem to save the stopping information, when I kill the program an re run it, I am going back to training while it should skip it as we reached the patience limit...

What do you think @pplantinga ?

main - seq2seq, and PER stats written to file: results/transformer/1234/reports/lexicon/35/wer_lexicon.txt
speechbrain.utils.epoch_loop - 2 epochs without improvement.
Patience of 2 is exhausted, stopping.
main - Attempting to restore checkpoint for step lexicon
speechbrain.utils.checkpoints - Loading a checkpoint from results/transformer/1234/save/CKPT+2023-08-09+12-42-08+00
100%|██████████████████████████████████████████████████████████████████████████████████| 17/17 [00:28<00:00, 1.69s/it]
speechbrain.utils.train_logger - Epoch loaded: 33 - test loss: 1.87e-01, test PER: 9.13
main - seq2seq, and PER stats written to file: results/transformer/1234/save/wer_lexicon.txt
[E ProcessGroupNCCL.cpp:828] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=35615, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1807776 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:455] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:460] To avoid data inconsistency, we are taking the entire process down.
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 72113 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 1 (pid: 72114) of binary: /users/amoumen/.conda/envs/2051/bin/python

However, now, my training is stopped. I can't do anything. Nothing show up. Additionally, the recipe doesn't seem to save the stopping information, when I kill the program an re run it, I am going back to training while it should skip it as we reached the patience limit...

Ah yes, the EpochCounterWithStopper currently doesn't override the saver/loader from EpochCounter but would need to in order to work correctly on restart (there's more info to be stored: the best score so far and the epoch that occurred).

I can add this. As for the other issue, I don't immediately know what's wrong.

It is working! Thanks.

I will open an issue related to the G2P recipe. There's a lot of problems when using it with DDP....

Yes, confirmed it worked for me. Sorry for the late reply