Skip to content
This repository was archived by the owner on Apr 23, 2026. It is now read-only.

Add backend capabilities to chat and generate - #1420

Closed
leseb wants to merge 2 commits into
instructlab:mainfrom
leseb:add-backend-capabilities-chat-gen
Closed

Add backend capabilities to chat and generate#1420
leseb wants to merge 2 commits into
instructlab:mainfrom
leseb:add-backend-capabilities-chat-gen

Conversation

@leseb

@leseb leseb commented Jun 18, 2024

Copy link
Copy Markdown
Contributor

This commit refactors the backend selection and server initialization
process for both the generate and chat functionalities within the
application. The changes include:

  • Abstracting the backend selection logic by utilizing the backends
    module, which dynamically determines the appropriate backend based on
    the configuration file or auto-detection if no configuration is set in
    the serve section of config.yaml (serve.backend)

  • Encapsulating the conditional logic for checking if the selected
    backend is LLAMA

  • Streamlining the exception handling for backend determination and
    server initialization, ensuring that any failures in these processes
    are clearly communicated to the user and result in a graceful exit.

These modifications enhance the code's structure, making it easier to
understand and extend in the future, especially as new backends or
initialization requirements are introduced.

Signed-off-by: Sébastien Han seb@redhat.com

ON-TOP of #1415

Checklist:

  • Commit Message Formatting: Commit titles and messages follow guidelines in the
    conventional commits.
  • Changelog updated with breaking and/or notable changes for the next minor release.
  • Documentation has been updated, if necessary.
  • Unit tests have been added, if necessary.
  • Integration tests have been added, if necessary.

@mergify mergify Bot added testing Relates to testing ci-failure PR has at least one CI failure labels Jun 18, 2024
@leseb
leseb force-pushed the add-backend-capabilities-chat-gen branch from f4cac3d to abf76d7 Compare June 18, 2024 21:05
@mergify mergify Bot added ci-failure PR has at least one CI failure and removed ci-failure PR has at least one CI failure labels Jun 18, 2024
@russellb

Copy link
Copy Markdown
Contributor

It looks like we need to resolve this design doc first? instructlab/dev-docs#81

From the looks of that PR, it doesn't look like we have consensus on how to incorporate the backends into the CLI commands.

@russellb russellb added the hold In-progress PR. Tag should be removed before merge. label Jun 18, 2024
@leseb
leseb force-pushed the add-backend-capabilities-chat-gen branch from abf76d7 to fdb4e22 Compare June 18, 2024 21:30
@mergify mergify Bot added ci-failure PR has at least one CI failure and removed ci-failure PR has at least one CI failure labels Jun 18, 2024
@leseb
leseb force-pushed the add-backend-capabilities-chat-gen branch from fdb4e22 to ec5cf09 Compare June 18, 2024 21:50
@mergify mergify Bot removed the ci-failure PR has at least one CI failure label Jun 18, 2024
@leseb
leseb force-pushed the add-backend-capabilities-chat-gen branch from ec5cf09 to 33e08b4 Compare June 19, 2024 12:50
@mergify mergify Bot added the ci-failure PR has at least one CI failure label Jun 19, 2024
@leseb
leseb force-pushed the add-backend-capabilities-chat-gen branch from 33e08b4 to b4ed798 Compare June 19, 2024 13:06
@mergify mergify Bot added needs-rebase This Pull Request needs to be rebased and removed ci-failure PR has at least one CI failure labels Jun 19, 2024
@mergify

mergify Bot commented Jun 19, 2024

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. @leseb please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@leseb
leseb force-pushed the add-backend-capabilities-chat-gen branch from b4ed798 to 3f5591d Compare June 19, 2024 13:29
@mergify mergify Bot removed the needs-rebase This Pull Request needs to be rebased label Jun 19, 2024
This commit introduces the ability to configure and automatically detect
the backend for serving models in InstructLab. One new field, `backend`
has been added to the configuration and the CLI to allow users to
specify the backend. This field is optional.

Additionally, the commit includes the implementation of backend validation and
auto-detection based on the model file properties, specifically targeting GGUF
file format for now. The `serve` command has been updated to utilize these new
configurations and perform backend validation or auto-detection as required.

Unit tests for backend validation and changes to the configuration tests to
include the new fields are also part of this commit.

Co-authored-by: Ali Maredia <amaredia@redhat.com>
Signed-off-by: Sébastien Han <seb@redhat.com>
@leseb
leseb force-pushed the add-backend-capabilities-chat-gen branch from 3f5591d to a2c5792 Compare June 21, 2024 13:49
@mergify mergify Bot added the ci-failure PR has at least one CI failure label Jun 21, 2024
This commit refactors the backend selection and server initialization
process for both the generate and chat functionalities within the
application. The changes include:

- Abstracting the backend selection logic by utilizing the `backends`
  module, which dynamically determines the appropriate backend based on
  the configuration file or auto-detection if no configuration is set in
  the serve section of config.yaml (serve.backend)

- Encapsulating the conditional logic for checking if the selected
  backend is LLAMA

- Streamlining the exception handling for backend determination and
  server initialization, ensuring that any failures in these processes
  are clearly communicated to the user and result in a graceful exit.

These modifications enhance the code's structure, making it easier to
understand and extend in the future, especially as new backends or
initialization requirements are introduced.

Signed-off-by: Sébastien Han <seb@redhat.com>
@leseb
leseb force-pushed the add-backend-capabilities-chat-gen branch from a2c5792 to 74ed0ce Compare June 21, 2024 13:56
@mergify mergify Bot removed the ci-failure PR has at least one CI failure label Jun 21, 2024
@leseb

leseb commented Jun 21, 2024

Copy link
Copy Markdown
Contributor Author

Closing for #1442.

@leseb leseb closed this Jun 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

hold In-progress PR. Tag should be removed before merge. testing Relates to testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants