Build software better, together

Luce-Org / lucebox

LLM speculative inference server for heterogeneous hardware & consumer GPUs

spark kernel cuda cuda-kernels rocm heterogeneous-computing luce poolside rtx3090 llama-cpp local-ai qwen speculative-decoding strix-halo dflash megakernel speculative-prefill pflash r9700

Updated Sep 18, 2026
C++

SafeAILab / EAGLE

Star

Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).

large-language-models llm-inference speculative-decoding

Updated Feb 20, 2026
Python

The fastest way to run Qwen 3.8 Flash Next and Qwen 3.8 27B on a Mac: 125 tok/s in OpenCode on an M5 Max. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.

Updated Sep 19, 2026
Python

intel / intel-extension-for-transformers

Star

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

retrieval chatbot rag habana large-language-model chatpdf llm-inference 4-bits speculative-decoding llm-cpu streamingllm intel-optimized-llamacpp neural-chat neural-chat-7b autoround gaudi3

Updated Oct 8, 2024
Python

dphnAI / sonar

Star

Large-scale LLM inference engine

machine-learning cuda intel api-rest lora rocm inference-engine tpu inferentia speculative-decoding

Updated Sep 11, 2026
Python

Tencent / AngelSlim

Star

Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.

audio eagle quantization diffusion vlm llm qwen speculative-decoding llm-compression hunyuan deepseek fp4 dflash

Updated Sep 4, 2026
Python

syv-ai / HyperQwen

Star

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

cuda quantization inference-optimization kv-cache vllm local-llm llm-inference qwen speculative-decoding qwen3 consumer-gpu rtx-3090

Updated Sep 19, 2026
Python

chrisliu298 / awesome-on-policy-distillation

Star

A curated collection of papers, technical reports, frameworks, and tools for on-policy distillation (OPD) of large language models

awesome reinforcement-learning rl awesome-list knowledge-distillation post-training opd distillation self-distillation llm rlhf gkd llm-training speculative-decoding on-policy-distillation minillm llm-distillation

Updated Aug 30, 2026

Avarok-Cybersecurity / atlas

Star

Pure Rust Inference Engine

rust cuda transformers ssm mamba dgx openai-api llm-inference speculative-decoding gb10 nvfp4 dgx-spark

Updated Sep 20, 2026
Rust

ARahim3 / mlx-dspark

Star

Up to 4× faster LLM decoding on Apple Silicon, lossless. Native MLX port of DeepSeek's DSpark & z-lab's DFlash speculative decoding — Gemma-4, Qwen3.8, Muse-Glimmer, Nemotron, LFM2.5, Ornith-1.0, ternary Bonsai-27B.

macos metal pi codex mlx inference-engine apple-silicon local-llm llm-inference speculative-decoding claude-code dspark qwen3-8 dflash2

Updated Sep 10, 2026
Python

peonist-ai / halogen-flash-server

Star

The fastest way to run Qwen3.8-Flash-Next on Strix Halo (gfx1151)

amd hip rocm radeon inference-engine mixture-of-experts openai-api llm long-context llm-serving gpu-inference local-llm qwen speculative-decoding ryzen-ai qwen3 strix-halo gfx1151

Updated Sep 20, 2026
Shell

zengxiao-he / tessera

Star

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

rust cuda pytorch triton quantization knowledge-distillation inference-engine jax kv-cache ml-systems llm mechanistic-interpretability fsdp flash-attention speculative-decoding paged-attention

Updated Jun 5, 2026
Python

ARahim3 / kaggle-tpu-lab

Star

Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~64 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.

Updated Sep 15, 2026
Python

AEON-7 / Qwen3.6-27B-AEON-Ultimate-Uncensored-DFlash

Star

Fully uncensored, capability-enhanced abliteration of Qwen3.6-27B. NVFP4 + z-lab DFlash speculative decoding (n=12) on the unified ghcr.io/aeon-7/aeon-vllm-ultimate:latest container, tuned for long-context draft acceptance on DGX Spark. 6 HF variants (BF16/NVFP4/MTP/MTP-XS), docker-compose, and QuickStart.

quantization uncensored blackwell llm vllm qwen speculative-decoding abliteration qwen3 nvfp4 dgx-spark dflash

Updated Sep 15, 2026
Python

incoai / splash

Star

A local inference engine for Apple silicon, built around the model.

macos metal apple-silicon llm-inference speculative-decoding coding-agents

Updated Sep 20, 2026
Python

Nehanth / swarmllm

Star

Every device brings a slice. Together they run the whole model. Peer-to-peer LLM inference across browser tabs: a from-scratch WebGPU engine and a WebRTC runtime that split a 27B model over the devices in a room.

browser webrtc peer-to-peer inference webgpu wgsl llm local-llm speculative-decoding distributed-inference