Generic Hooks Sketch - #309

Preface:

• This is a proof of concept. It should not be implemented as is. It's meant as a demonstration on what might be possible.
• I modified the test.yml to upload include a full trace log because that helps when the test host crashes.
• I added InternalsVisibleToAttribute to MonoMod.Core for MonoMod.RuntimeDetours because I wasn't sure where to put the classes and just wanted to keep everything in one place for now.
• I'm not sure if this should be part of MonoMod. I originally started this because I needed to detour generics on one specific runtime under specific circumstances. Expanding support to all runtimes/architectures included in the test matrix was only possible by using a lot of tricks of which I'm not sure how reliable they are.

Disclaimer:

• I had a basic idea for the initial approach. Which worked up to a point.
• .NET Core and anything outside WinX64 caused various issues. At that point I started trying different approaches until one of them worked (worked == gets test to pass), so the current implementation will certainly have issues
• I was pretty unsure on whether some of the things I do already have helpers
• This is a proof of concept, so I ignored potential performance issues

Basic API:

• Priming:
  • While a detour is applied to a method, the detour will only dispatch for primed types. That means that you can detour a type A with your hook H, but your hook will only be called if the concrete T during a call is primed
  • AutoPrimeMode (this only works for reference types, and on Mono it will not work if the generics contain any value type):
    • Since you often don't just want to detour a single concrete type, autoPrime exists.
    • AutoPrimeMode.None: Only the instantiations explicitly primed are hooked
    • AutoPrimeMode.Compatible: will auto-prime any type t it encounters if that t is compatible with any of your primed types. E.g. if you primed object, then string will automatically be primed if it's called.
    • AutoPrimeMode.All: will auto-prime any type t it encounters.
• API:

var myHook = new GenericHook(MethodOf(Src), MethodOf(Hook), [[typeof(int)]], AutoPrimeMode.Compatible);
// Src<int> will detour to Hook<int>
// Any other call will not be detoured at this point
myHook.Prime(typeof(object));
// Src<object> will detour to Hook<object>
// Any other call with a reference type will also be detoured, as they are compatible with object
// Any value types other than the already primed int will not be detoured

Design:

[call] =>

• -> shared (detoured) native body
  • -> (Mono) GenericContextCaptureStub: move RGCTX reg into an arg reg
  • -> UniversalDispatcher:
    • -> Read Generic Context
    • -> Resolve Instantiation
    • -> Hit -> SharedDispatchTrampoline -> User Replacement (+orig chain)
    • -> Miss -> orig

Caveats:

• You can't prime interfaces (yet?)
• I'm not sure if exceptions are properly handled
• I ignore the detour config
• Comments (especially docstrings for the public API) are missing
• Despite being the platform I originally developed this for; I didn't run the test matrix on Unity Mono (as x86 Mono is not supported and running it in a game seems to cause issues with test discovery)
• Value types all need explicit priming. While this works for what I need it for, I'm not sure that suffices for a general purpose API? I guess it would be possible to use the compile JIT hook on CoreCLR to discover new value type instantiations, but e.g. Mono only exposes a Profiler API for that (and I think that would have performance issues?); maybe there's a better way to implement all of this that would take care of that.
• Mono does not support auto priming if the generics contain at least one value type.
• Unity Mono seems to not work with this sketch. Adding support is pretty simple, however.