Bug fixed with n_ctx=0 - #1015
Merged
Merged
Conversation
If the n_ctx is set to 0 the code should use the maximum context length of the selected model, but it didn't work. There was a problem with the initialization of this parameter and a related problem with 'n_batch'.
Contributor
Author
|
Another approach can be to use the code in the llama.cpp repo to read the metadata from the gguf file before loading the model. In the json format the context length is related to the key |
Owner
|
@DanieleMorotti does the metadata value differ from |
Contributor
Author
|
I think |
K-Mistele
added a commit
to K-Mistele/llama-cpp-python
that referenced
this pull request
Jan 16, 2024
…l n_ctx_train field per abetlen#1015
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If the
n_ctxparameter is set to 0 the function should use the maximum context length of the selected model, but it didn't work. There was a problem with the initialization of this parameter and a related problem withn_batch.I know that the code is not the best one, but in order to get the model information about the context I needed to add it after the creation of the model instance at line 923 of the file
llama.py.Unfortunately, different objects were already initialized, therefore in the fix I had to change the
n_ctx,self.n_batch,self.context_params.n_ctxandself.context_params.n_batchvariables even if they already had a value.Tell me if you find a smarter or more elegant solution to change the code and i will implement it.
This change should also fix #988.
Thank you