Releases: LSDJesus/llama-cpp-python
Releases · LSDJesus/llama-cpp-python
Release list
v0.3.37-cu130-Basic-win-20260510
Add build_output.txt and update llama_cpp integration - Add build_output.txt - Update llama_cpp/llama_chat_format.py and llama_cpp/llama_cpp.py - Update pyproject.toml and vendor/llama.cpp
v0.3.37-cu130-Basic-linux-20260510
Add build_output.txt and update llama_cpp integration - Add build_output.txt - Update llama_cpp/llama_chat_format.py and llama_cpp/llama_cpp.py - Update pyproject.toml and vendor/llama.cpp
v0.3.37-cu130-Basic-win-20260508
fix image input
v0.3.27-LUNA — CUDA Wheels with Penultimate Layer API
llama-cpp-python v0.3.27-LUNA — Pre-built CUDA 12.8 Wheels
Fork of JamePeng/llama-cpp-python with custom C API extensions for penultimate layer hidden state extraction.
What's Included
Pre-built wheels with CUDA 12.8 and native SASS for:
| Target | GPU Series | Architecture |
|---|---|---|
| sm_75 | RTX 2000 | Turing |
| sm_86 | RTX 3000 | Ampere |
| sm_89 | RTX 4000 | Ada Lovelace |
| sm_120 | RTX 5000 | Blackwell |
ABI3 wheels — single build covers Python 3.9 – 3.14, no PyTorch dependency.
Wheels
| Platform | File | Size |
|---|---|---|
| Windows x64 | llama_cpp_python-0.3.27-cp39-abi3-win_amd64.whl |
~186 MB |
| Linux x86_64 | llama_cpp_python-0.3.27-cp39-abi3-manylinux_2_28_x86_64.whl |
~280 MB |
Installation
# Windows
pip install https://github.com/LSDJesus/llama-cpp-python/releases/download/v0.3.27-cuda-6396cf3/llama_cpp_python-0.3.27-cp39-abi3-win_amd64.whl
# Linux
pip install https://github.com/LSDJesus/llama-cpp-python/releases/download/v0.3.27-cuda-6396cf3/llama_cpp_python-0.3.27-cp39-abi3-manylinux_2_28_x86_64.whl
Custom API Extensions
This fork adds layer-level hidden state access to the llama.cpp C API:
llama_get_embeddings_penultimate_ith(ctx, i) — Extract pre-normalization penultimate layer output per token (equivalent to HuggingFace hidden_states[-2])
llama_set_layer_capture(ctx, mask, n_layers) — Enable per-layer hidden state capture
llama_get_embeddings_layer_ith(ctx, layer, i) — Retrieve captured hidden state for any layer + token position
llama_set_layer_skip(ctx, mask, n_layers) — Skip specific transformer layers during inference
These are exposed in Python via Llama.get_penultimate_embeddings(), Llama.set_layer_capture(), Llama.get_layer_embeddings(), and Llama.set_layer_skip().
Use Case
Primary use case: quantized text encoding for diffusion models (e.g., Z-Image) where penultimate layer hidden states serve as conditioning input. A Q4_K_M GGUF runs at ~2.5 GB VRAM vs ~8 GB for full bf16, with negligible quality difference on base model variants.
Requirements
NVIDIA GPU (sm_75+) with CUDA 12.8-compatible driver
No CUDA Toolkit installation needed — runtime libs are loaded from driver