Skip to content
This repository was archived by the owner on Apr 23, 2026. It is now read-only.

[DRAFT] Model serve multiple backends - #1386

Closed
alimaredia wants to merge 4 commits into
instructlab:mainfrom
alimaredia:model-serve-multiple-backends
Closed

[DRAFT] Model serve multiple backends#1386
alimaredia wants to merge 4 commits into
instructlab:mainfrom
alimaredia:model-serve-multiple-backends

Conversation

@alimaredia

@alimaredia alimaredia commented Jun 17, 2024

Copy link
Copy Markdown
Contributor

Checklist:

  • Commit Message Formatting: Commit titles and messages follow guidelines in the
    conventional commits.
  • Changelog updated with breaking and/or notable changes for the next minor release.
  • Documentation has been updated, if necessary.
  • Unit tests have been added, if necessary.
  • Integration tests have been added, if necessary.

@alimaredia
ilab model chat with vllm
0dd84e1 
rewrite/copy ensure_serve logic to work
with a model served with vllm for `ilab chat`

Signed-off-by: Ali Maredia <amaredia@redhat.com>
@alimaredia
Have ilab chat working with new BackendServer classes
68f5fe3 
Signed-off-by: Ali Maredia <amaredia@redhat.com>
@alimaredia
get ilab data generate to run with new ensure_server()
0dd1625 
Signed-off-by: Ali Maredia <amaredia@redhat.com>
@alimaredia
integrated vllm into ilab model serve
1af831c 
Signed-off-by: Ali Maredia <amaredia@redhat.com>
@alimaredia alimaredia mentioned this pull request Jun 17, 2024
Closed
@mergify mergify Bot added the ci-failure PR has at least one CI failure label Jun 17, 2024
@tiran

tiran commented Jun 17, 2024

Copy link
Copy Markdown
Contributor

The vLLM package on PyPI is a CUDA build. How do you plan to support CPU-only, ROCm, and the vLLM fork for Intel Gaudi?

@alimaredia alimaredia added the hold In-progress PR. Tag should be removed before merge. label Jun 17, 2024
@alimaredia

alimaredia commented Jun 17, 2024

Copy link
Copy Markdown
Contributor Author

@tiran I understand your concerns and your comments in #1276 were insightful and will be applied/taken into consideration. The requirements.txt in this file was able to work in my development environment, and is not intended to be merged.

This PR is purely a draft, @leseb will be taking over this integrating vllm into ilab. I just wanted a place where many people can get eyes on the code and discuss the feature.

@leseb
leseb marked this pull request as draft June 17, 2024 11:38
@mergify

mergify Bot commented Jun 18, 2024

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. @alimaredia please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase This Pull Request needs to be rebased label Jun 18, 2024
@leseb

leseb commented Jun 21, 2024

Copy link
Copy Markdown
Contributor

Closing in favor of #1442. Thanks!

@leseb leseb closed this Jun 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

No one assigned

Labels

ci-failure PR has at least one CI failure hold In-progress PR. Tag should be removed before merge. needs-rebase This Pull Request needs to be rebased

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@alimaredia @tiran @leseb