Allow for separate judge backend option - #1700
Conversation
This is to allow for cases when for example your model or base model need to be served with llama-cpp but your judge needs to be served by vllm. That would work as is if backend isn't specified. But as soon as you need to force a backend, currently it would force the backend to be the same for the model and judge model Signed-off-by: Dan McPherson <dmcphers@redhat.com>
leseb
left a comment
There was a problem hiding this comment.
Please document --judge-backend to CHANGELOG.md.
We're still on the same version of ilab where evaluate hasn't been released. evaluate itself is mentioned in the release notes but not every single option. --backend for example is not listed explicitly there now. It's going to be pretty spotty if I add judge-backend as is. |
Ok sounds good to keep a general section for now and add flags once we have a release. |
This is to allow for cases when for example your model or base model need to be served with llama-cpp but your judge needs to be served by vllm. That would work as is if backend isn't specified. But as soon as you need to force a backend, currently it would force the backend to be the same for the model and judge model
Ex:
Issue resolved by this Pull Request:
Resolves #1602
Two other PRs have already addressed issues noted in #1602: #1692 and #1564
Checklist:
conventional commits.