You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When max-ctx-size goes from 25 -> 55 , certain prompts throw a openai.InternalServerError even though both max-ctx-sizes get rounded up to 64 in llama_cpp. Below is a breakdown of when the trimming code in chat is executed with different prompts and two small max-ctx-sizes.
M1 = "Hello"
M2 = "Hello, joe was born in 2000. How old is joe"
M3 = "Hello, I am a ci message that should not finish because I am too long for the context window, tell me about your day please?"
M4 = "Hello, I am a ci message that should not finish because I am too long for the context window, tell me about your day please? How many tokens could you take today. Could you tell me about the time you could only take 55 tokens"
M5 = "I was born in 2000. How old am I?"
25 max context window (ilab serve --max-ctx-size 25):
ilab chat -qq M1
- BadRequestError: Trims message
- Responds back fine
ilab chat -qq M2
- BadRequestError: Trims message
- Responds back fine
ilab chat -qq M3
- BadRequestError: Trims message
- BadRequestError: Final message too large for context size
ilab chat -qq M4
- BadRequestError: Trims message
- BadRequestError: Final message too large for context size
ilab chat -qq M5
- BadRequestError: Trims message
- Responds back fine
55 max context window (ilab serve --max-ctx-size 55):
ilab chat -qq M1
- Responds back fine
ilab chat -qq M2
- BadRequestError: Trims message
- Responds back fine
ilab chat -qq M3
- BadRequestError: Trims message
- Responds back fine
ilab chat -qq M4
- BadRequestError: Trims message
- InternalServerError
ilab chat -qq M5
- InternalServerError
To Reproduce
See description above
Expected behavior
BadrequestError's get thrown every time a trim happens in the situation where the max-ctx-window is 55 for M4 and M5. InternalServerError's should not get thrown when trimming occurs.
Device Info (please complete the following information):
Describe the bug
From #1161:
To Reproduce
See description above
Expected behavior
BadrequestError's get thrown every time a trim happens in the situation where the max-ctx-window is 55 for M4 and M5. InternalServerError's should not get thrown when trimming occurs.
Device Info (please complete the following information):
Python 3.12.2ilab, version 0.14.1.dev235