fix: add checks to ensure evaluate model args are valid - #1795

@leseb I noticed you introduced a function is_model_gguf for the backend work - do you think that's worth using here or this check is sufficient? only noticed the func after I'd already written this

@leseb I noticed you introduced a function is_model_gguf for the backend work - do you think that's worth using here or this check is sufficient? only noticed the func after I'd already written this

Yes, that's the proper way to validate the model is GGUF so please use it.

The root of the issue is that MMLU and MMLU-Branch only work for .safetensors models right now. What we actually want to check is that the model is .safetensors and reject anything else.

Seb and JD worked on adding a check for this here: https://github.com/instructlab/instructlab/blob/main/src/instructlab/model/backends/backends.py#L115

@nathan-weinberg @alimaredia @alinaryan @danmcp
is the GGUF check being done because eval spins up a vLLM server? If so heads up that I have a PR to check for .bin files in addition to .safetensors files - presence of either would qualify the model as valid since .bin files can also be served via vLLM
this shouldn't break eval, and should allow us to also evaluate .bin models
#1809

@nathan-weinberg @alimaredia @alinaryan @danmcp is the GGUF check being done because eval spins up a vLLM server? If so heads up that I have a PR to check for .bin files in addition to .safetensors files - presence of either would qualify the model as valid since .bin files can also be served via vLLM this shouldn't break eval, and should allow us to also evaluate .bin models #1809

For MTBench and MTBenchBranch, yes we are spinning up a vLLM server but there is also llama-cpp support so GGUF files can be used, though for most users using vLLM and safetensors makes the most sense here

@leseb @alinaryan @booxter I've done a bit of a refactor here - know you all already approved but wanted to give you a ping in case you had any comments on the new code