This repository was archived by the owner on Apr 23, 2026. It is now read-only.
ilab model evaluate command and eval library usage - #1369
Merged
Conversation
Contributor
Author
|
this is a placeholder for now, lmk when the library is somewhere I can access and import (with the actual code) |
Contributor
|
@cdoern you can install directory from test.pypy.org for testing if you wish: https://test.pypi.org/project/instructlab-eval once we have a |
russellb
self-requested a review
June 19, 2024 00:26
leseb
suggested changes
Jun 19, 2024
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
danmcp
suggested changes
Jun 24, 2024
danmcp
reviewed
Jun 25, 2024
… model-path Serving will need to be replaced with library calls once available Signed-off-by: Dan McPherson <dmcphers@redhat.com>
… base-model-path to allow for _branch comparison Signed-off-by: Dan McPherson <dmcphers@redhat.com>
in order to avoid conflicts with other entities named model_ you need a model_config Signed-off-by: Charlie Doern <cdoern@redhat.com>
Signed-off-by: Charlie Doern <cdoern@redhat.com>
Signed-off-by: Dan McPherson <dmcphers@redhat.com>
…s benchmarks Signed-off-by: Dan McPherson <dmcphers@redhat.com>
Signed-off-by: Dan McPherson <dmcphers@redhat.com>
Signed-off-by: Dan McPherson <dmcphers@redhat.com>
Signed-off-by: Nathan Weinberg <nweinber@redhat.com>
Signed-off-by: Nathan Weinberg <nweinber@redhat.com>
Signed-off-by: Nathan Weinberg <nweinber@redhat.com>
Signed-off-by: Charlie Doern <cdoern@redhat.com>
Signed-off-by: Nathan Weinberg <nweinber@redhat.com>
Signed-off-by: Nathan Weinberg <nweinber@redhat.com>
Signed-off-by: Dan McPherson <dmcphers@redhat.com>
Signed-off-by: Dan McPherson <dmcphers@redhat.com>
danmcp
approved these changes
Jun 29, 2024
danmcp
left a comment
Contributor
There was a problem hiding this comment.
I am approving with a few notes:
- The error handling still needs a decent bit of work both here and the eval library. -> Replace stack traces with nice messages.
- This version is very chatty in the cli. Will largely be addressed with https://github.com/instructlab/instructlab/compare/main...danmcp:instructlab:evalandvllm?expand=1. Those commit(s) also address the vllm serving hack still in place in this PR.
- More test cases are needed
- Config defaulting probably still needs some work with the models selected
alinaryan
reviewed
Jun 30, 2024
alinaryan
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the strong work on this!! Just have a few comments/q's:
alinaryan
approved these changes
Jun 30, 2024
This was referenced Jun 30, 2024
Contributor
Author
|
made follow up issues! |
Contributor
Author
|
Given that tomorrow morning (7/1) is a deadline. This PR needs to be merged before then to add some form of evaluation support. That being said, there is a stale change request on this PR, some pending reviews that haven't come in yet, etc. We will be dismissing those in favor of deferring to follow up issues but if any of the reviewers have immediate follow up concerns please feel free to reach out!!!! |
3 tasks
Contributor
|
I filed #1540 as a follow-up to get this tested in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ilab model evaluatewhich allows users to run MMLU Bench MT Bench, MMLU Branch, and MT Branch Benchmarks_evaluateclass toconfig.yamlso that users can get sane evaluation defaults that they can see and modify. These funnel directly into the evaluation flags as training now does.a sample evaluation class looks like: