QuantizedModelLoader¶

Loader for quantized models saved by OneComp.

On macOS, load_quantized_model() places the model on MPS when available (CUDA > MPS > CPU via get_default_device()). Use Transformers generate() for inference; vLLM requires Linux with an NVIDIA GPU. See the macOS / MPS guide.

QuantizedModelLoader ¶

Loader for quantized models saved by onecomp (GPTQ, DBF, OneBit, etc.).

load_quantized_model `classmethod` ¶

load_quantized_model(save_directory: str, *, torch_dtype: Optional[dtype] = None, device_map: Optional[str] = 'auto', trust_remote_code: bool = True, local_files_only: bool = True) -> Tuple[Any, Any]

Load a quantized model and tokenizer from a safetensors directory.

The directory must contain: - config.json (with quantization_config) - tokenizer files - model.safetensors (quantized layers: qweight/scales for GPTQ, scaling0/bp for DBF)

Quantization parameters (quant_method, bits, group_size, etc.) are read from config.json and quantized layers are reconstructed directly from the safetensors state_dict. No quantization_results.pt is needed.

If the directory additionally contains a PEFT-format LoRA adapter sidecar (adapter_model.safetensors + adapter_config.json), the matching GPTQLinear layers are automatically re-wrapped with LoRAGPTQLinear populated from the sidecar. This lets runner.save_quantized_model → load_quantized_model round-trip models produced by a LoRA post-process such as PostProcessLoraSFT.

For legacy models saved via torch.save (.pt format), use :meth:load_quantized_model_pt instead.

Parameters:

Name	Type	Description	Default
`save_directory`	`str`	Path to the saved model directory.	required
`torch_dtype`	`Optional[dtype]`	Model dtype (default: torch.float16).	`None`
`device_map`	`Optional[str]`	Device placement (default: "auto"). Set to `None` or `""` to leave the model on CPU.	`'auto'`
`trust_remote_code`	`bool`	Passed to from_pretrained.	`True`
`local_files_only`	`bool`	Passed to from_pretrained.	`True`

Returns:

Type	Description
`Tuple[Any, Any]`	(model, tokenizer)

Raises:

Type	Description
`FileNotFoundError`	If `save_directory` or its `config.json` is missing.
`ValueError`	If `quantization_config` is missing or not a dict, or lacks the required `quant_method` / `modules_in_block_to_quantize` keys. Validated by the same :func:`onecomp.utils.quant_config.validate_quant_config` used by the save path, so saving and loading enforce identical required keys and raise the same exception type.

Example

model, tokenizer = QuantizedModelLoader.load_quantized_model("./tinyllama_gptq3")

load_quantized_model_pt `classmethod` ¶

load_quantized_model_pt(save_directory: str, *, device_map: Optional[str] = 'auto', local_files_only: bool = True, allow_unsafe_deserialization: bool = False) -> Tuple[Any, Any]

Load a quantized model and tokenizer saved as a PyTorch .pt file.

Use this method to load models saved by :meth:Runner.save_quantized_model_pt, which preserves custom module types (e.g. LoRAGPTQLinear from LoRA post-processing).

.. note:: This .pt path is intended for research and development use only -- for example, to rapidly experiment with a new post-process before it has a safetensors-compatible :meth:load_quantized_model implementation. It is not recommended for general or production use, both because of the unsafe-deserialization risk described below and because the .pt format is not HF-compatible. Prefer the safetensors-based :meth:load_quantized_model whenever possible.

The directory must contain: - model.pt (serialized with torch.save) - Tokenizer files

.. warning:: This method deserializes model.pt with torch.load(..., weights_only=False). Because PyTorch .pt checkpoints use Python's pickle, a maliciously crafted model.pt can execute arbitrary code during deserialization (CWE-502). weights_only=False is required here because the .pt format preserves full custom module objects (e.g. LoRAGPTQLinear) that cannot be reconstructed from tensors alone. Only load model.pt files that you produced yourself or obtained from a fully trusted source. For untrusted or third-party models, prefer the safetensors-based :meth:load_quantized_model, which does not execute code.

Parameters:

Name	Type	Description	Default
`save_directory`	`str`	Path to the saved model directory.	required
`device_map`	`Optional[str]`	Device placement (default: `"auto"`). Set to `""` or `None` to skip device placement.	`'auto'`
`local_files_only`	`bool`	Passed to `AutoTokenizer.from_pretrained`.	`True`
`allow_unsafe_deserialization`	`bool`	Must be explicitly set to `True` to acknowledge the unsafe-deserialization risk described above and permit loading. Defaults to `False`, in which case this method raises before any code can be executed.	`False`

Returns:

Type	Description
`Tuple[Any, Any]`	(model, tokenizer)

Raises:

Type	Description
`ValueError`	If `allow_unsafe_deserialization` is not `True`.

Example

model, tokenizer = QuantizedModelLoader.load_quantized_model_pt( ... "./quantized_model_lora", ... allow_unsafe_deserialization=True, # trusted source only ... )

Convenience Functions¶

The top-level aliases provide shortcuts for both formats:

from onecomp import load_quantized_model, load_quantized_model_pt

# Load a safetensors model, including BlockWisePTQ / GlobalPTQ / GlobalPTQDistributed
# outputs. If a LoRA adapter sidecar (lora_adapter/) is present it is auto-detected
# and applied, so LoRA models saved with save_quantized_model() load here too.
model, tokenizer = load_quantized_model("./saved_model")

# Keep the loaded model on CPU before running additional post-processes
model, tokenizer = load_quantized_model("./saved_model", device_map=None)

# Load a legacy PyTorch .pt model (whole-object torch.save; prefer the
# safetensors load_quantized_model() above, including for LoRA).
# Requires explicit opt-in: the .pt loader uses torch.load(weights_only=False),
# which can execute code from a malicious file (CWE-502). Only enable this for
# model.pt files from a fully trusted source.
model, tokenizer = load_quantized_model_pt(
    "./saved_model_lora", allow_unsafe_deserialization=True
)

Unsafe deserialization (.pt loader)

load_quantized_model_pt() loads model.pt with torch.load(..., weights_only=False). Because PyTorch .pt checkpoints use Python pickle, a maliciously crafted model.pt can execute arbitrary code during loading (CWE-502). The method refuses to load unless you pass allow_unsafe_deserialization=True. Only opt in for models you produced yourself or obtained from a fully trusted source. For untrusted or third-party models, prefer the safetensors-based load_quantized_model(), which does not execute code.

Research/development use only

load_quantized_model_pt() (and the .pt save/load path in general) is intended for research and development only -- for example, to quickly experiment with a new post-process before it has a safetensors-compatible load_quantized_model() implementation. It is not recommended for general or production use; prefer the safetensors-based load_quantized_model(), which is HF-compatible and does not execute code.

QuantizedModelLoader¶