Skip to content
This repository was archived by the owner on Apr 23, 2026. It is now read-only.

Allow for separate judge backend option - #1700

Merged
mergify[bot] merged 1 commit into
instructlab:mainfrom
danmcp:judgebackendoption
Jul 15, 2024
Merged

Allow for separate judge backend option#1700
mergify[bot] merged 1 commit into
instructlab:mainfrom
danmcp:judgebackendoption

Conversation

@danmcp

@danmcp danmcp commented Jul 12, 2024

Copy link
Copy Markdown
Contributor

This is to allow for cases when for example your model or base model need to be served with llama-cpp but your judge needs to be served by vllm. That would work as is if backend isn't specified. But as soon as you need to force a backend, currently it would force the backend to be the same for the model and judge model

Ex:

(venv) [ec2-user@ip-10-0-1-95 instructlab]$ ilab model evaluate --judge-model models/instructlab/granite-7b-lab --benchmark mt_bench --backend llama-cpp --judge-backend vllm
Generating answers...
WARNING 2024-07-12 22:22:51,358 evaluate.py:276: launch_server When using llama-cpp, we recommend setting max_workers to a maximum of 16
  0%|                                                                                                                                                     | 0/4 [00:00<?, ?it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [02:16<00:00, 34.06s/it]
Evaluating answers...
INFO 2024-07-12 22:25:16,422 vllm.py:149: run_vllm vLLM starting up on pid 3485 at http://127.0.0.1:36937/v1
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:18<00:00,  2.26s/it]
# SKILL EVALUATION REPORT

## MODEL
models/merlinite-7b-lab-Q4_K_M.gguf

### AVERAGE:
9.0 (across 1)

### TURN ONE:
9.0

### TURN TWO:
N/A

### ERROR RATE:
0.88

Issue resolved by this Pull Request:
Resolves #1602
Two other PRs have already addressed issues noted in #1602: #1692 and #1564

Checklist:

  • Commit Message Formatting: Commit titles and messages follow guidelines in the
    conventional commits.
  • Changelog updated with breaking and/or notable changes for the next minor release.
  • Documentation has been updated, if necessary.
  • Unit tests have been added, if necessary.
  • Integration tests have been added, if necessary.

This is to allow for cases when for example your model or base model need to be served with llama-cpp but your judge needs to be served by vllm.  That would work as is if backend isn't specified.  But as soon as you need to force a backend, currently it would force the backend to be the same for the model and judge model

Signed-off-by: Dan McPherson <dmcphers@redhat.com>
@mergify mergify Bot added the one-approval PR has one approval from a maintainer label Jul 12, 2024
@alimaredia
alimaredia self-requested a review July 14, 2024 13:01

@leseb leseb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please document --judge-backend to CHANGELOG.md.

@danmcp

danmcp commented Jul 15, 2024

Copy link
Copy Markdown
Contributor Author

Please document --judge-backend to CHANGELOG.md.

We're still on the same version of ilab where evaluate hasn't been released. evaluate itself is mentioned in the release notes but not every single option. --backend for example is not listed explicitly there now. It's going to be pretty spotty if I add judge-backend as is.

@leseb

leseb commented Jul 15, 2024

Copy link
Copy Markdown
Contributor

Please document --judge-backend to CHANGELOG.md.

We're still on the same version of ilab where evaluate hasn't been released. evaluate itself is mentioned in the release notes but not every single option. --backend for example is not listed explicitly there now. It's going to be pretty spotty if I add judge-backend as is.

Ok sounds good to keep a general section for now and add flags once we have a release.

@mergify mergify Bot removed the one-approval PR has one approval from a maintainer label Jul 15, 2024
@mergify
mergify Bot merged commit 301f6f9 into instructlab:main Jul 15, 2024
@ktam3 ktam3 added this to the 0.18.0 milestone Jul 15, 2024
@danmcp
danmcp deleted the judgebackendoption branch August 7, 2024 17:59
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default model and backend does match for evaluate

5 participants