[Feature Request]: multi-GPU docs should be improved/updated #1860

🚀The feature

See:

• https://speechbrain.readthedocs.io/en/latest/multigpu.html
• https://github.com/speechbrain/speechbrain/blob/develop/docs/multigpu.md

There are several issues:

• SpeechBrain recommends against DataParallel, but none of that is mentioned in this document (it instead seems like a recommended option, if not more than DDP).
• It is not immediately clear what one needs to ensure to make their recipe DDP-safe.
• "For further information, please see our multi-gpu tutorial: amazing multi-gpu tutorial" -> either the tutorial does not exist/is the page itself (I don't recall seeing one?) or the link was forgotten.
• The page mentions that when issues are being encountered with nccl, the gloo backend can be tested instead. However, PyTorch docs seem to indicate that gloo simply cannot be used with CUDA builds of PyTorch. It is only recommended for CPU training. Hence, that recommendation seems unnecessary.
• ... Hence providing the backend in the commandline seems unnecessary (?)
• "The future version of pytorch will improve the stability of DDP." -> I am not sure what this refers to, but this is most likely outdated?
• The page could be laid out slightly better for navigability - in particular, "Slurm" could be put into its own section, the single-node multi-GPU DDP case could be put in a separate subsection from multi-node multi-GPU, etc.

Some points could benefit from more details.

Solution outline

The above issues should be addressed.

Additional context

I could work on making a PR to update the document so I assigned myself.