Adapters + LLama -- re-design. #2526

Describe the bug

It's not a bug, just a discussion. I think that people including @Adel-Moumen @poonehmousavi @mravanelli and maybe @pplantinga @asumagic may want to engage.

Adapters, or, more generally, altering an existing pre-trained model (you can see it as an object originating from the Pretrainer or checkpointer) is something becoming more and more common. Due to this, we, imho, must define a proper design in SpeechBrain to do so. Recently, I implemented LoRA and Houlsby for our Transformer, on my side. But I also realised that @poonehmousavi did some work for Llama here. I don't think we are doing this correctly. The code in Llama 2, for instance, might be hard to understand, and some functions (like the one replacing a module in an existing module) should be generalised and considered a general SpeechBrain util. My strategy would be to create an Adapters.py in lobes where we could put everything relating to them, instead of having them randomly appearing in lobes files.

What do you folks think?

Expected behaviour

Respect the Zen of SpeechBrain.

To Reproduce

No response

Environment Details

No response

Relevant Log Output

No response

Additional Context

No response