Add vLLM backend - #1442
Conversation
This commit introduces the ability to configure and automatically detect the backend for serving models in InstructLab. One new field, `backend` has been added to the configuration and the CLI to allow users to specify the backend. This field is optional. Additionally, the commit includes the implementation of backend validation and auto-detection based on the model file properties, specifically targeting GGUF file format for now. The `serve` command has been updated to utilize these new configurations and perform backend validation or auto-detection as required. Unit tests for backend validation and changes to the configuration tests to include the new fields are also part of this commit. Co-authored-by: Ali Maredia <amaredia@redhat.com> Signed-off-by: Sébastien Han <seb@redhat.com>
This commit refactors the backend selection and server initialization process for both the generate and chat functionalities within the application. The changes include: - Abstracting the backend selection logic by utilizing the `backends` module, which dynamically determines the appropriate backend based on the configuration file or auto-detection if no configuration is set in the serve section of config.yaml (serve.backend) - Encapsulating the conditional logic for checking if the selected backend is LLAMA - Streamlining the exception handling for backend determination and server initialization, ensuring that any failures in these processes are clearly communicated to the user and result in a graceful exit. These modifications enhance the code's structure, making it easier to understand and extend in the future, especially as new backends or initialization requirements are introduced. Signed-off-by: Sébastien Han <seb@redhat.com>
|
This contains everything related to the vLLM integration work. |
tiran
left a comment
There was a problem hiding this comment.
The vLLM version on PyPI is limited to CUDA. AMD ROCm and Intel Gaudi needs different builds and different forks of vLLM. There is also a version conflict between vLLM and optimum-habana for Intel Gaudi.
I suggest to introduce a cuda optional dependencies and only install vLLM with pip install instructlab[cuda] for now.
backends.py now holds the common functions to be used by all the backends we will implements. These functions will be used to reduce code duplication. Now, the only supported backend 'llama-cpp' was reworked to make it of these common functions. Signed-off-by: Sébastien Han <seb@redhat.com>
|
vLLM does not support Python 3.12 and therefore does not have binaries for 3.12 on PyPI. It also has a really strange build system. vLLM build systems requires a PyTorch installation to build vLLM for a platform. Basically vLLM only works out of the box on Linux x86_64 with CUDA on Python >=3.9 <= 3.11. Everything else is not working at the moment or cannot be expressed as a simple requirement in wheels. |
|
And you also need to install a bunch of packages manually for each GPU arch, e.g. https://github.com/vllm-project/vllm/blob/main/requirements-cuda.txt for CUDA and https://github.com/vllm-project/vllm/blob/main/requirements-rocm.txt for ROCm. Except do NOT install Ray. It's only needed for multi-node inference and is an even greater PITB to install correctly. |
Previously we talked about making vllm an optional dep which people could pip install instructlab[vllm]. I think that probably makes sense still. wdyt @tiran ? |
InstructLab is now capable of running a vLLM server to serve models. The backend is auto-detected based on that fact that model_path is a directory. Signed-off-by: Sébastien Han <seb@redhat.com>
When running `ilab model chat` or `ilab model generate`, the internal
backend that will be used can be vLLM (if no server is running through
`ilab model serve`).
This code works also with the internal auto-detection backend as well as
setting the backend in the serve section of the config file.
Signed-off-by: Sébastien Han <seb@redhat.com>
The logs from the backend server is being redirected from stdout/stderr to an instance of a logger. When this happens the logger would print the current log python file name. We replace this with "serving_backend" to indicate that the log is coming from inside the uvicorn process that runs the backend (llama_cpp or vllm). Signed-off-by: Sébastien Han <seb@redhat.com>
| template = "" | ||
| # TODO: find the equivalent in vllm for eos_token and bos_token | ||
| eos_token = "<|endoftext|>" | ||
| bos_token = "" |
There was a problem hiding this comment.
@leseb are eos_token and bos_token just here for the TODO? They don't seem to lead anywhere
|
this probably should have come after #1370 |
See individual commits.
Checklist:
conventional commits.