lcp[server] has been excellent. And I can host two models by running a second instance.
I'd like to be able to serve multiple models with a single instance of the OpenAI-compatible server and switch between them based on alias-able model in the query payload. My use case is to serve a code model and bakllava at the same time.
I am going to see about attempting to PR an Nginx configuration example that would reverse proxy to two instances based on the model in the POST body, but first order support would be great. If no one picks this up, I might attempt a PR of first order support early '24.
lcp[server] has been excellent. And I can host two models by running a second instance.
I'd like to be able to serve multiple models with a single instance of the OpenAI-compatible server and switch between them based on alias-able
modelin the query payload. My use case is to serve a code model and bakllava at the same time.I am going to see about attempting to PR an Nginx configuration example that would reverse proxy to two instances based on the
modelin the POST body, but first order support would be great. If no one picks this up, I might attempt a PR of first order support early '24.