AI / Local LLM

What's in a GGUF? Understanding the GGUF Format for Local LLMs

If you've downloaded a local LLM, you've probably seen .gguf files. But what's actually inside them? A deep dive into the format that powers llama.cpp, Ollama, LM Studio, and the entire local LLM ecosystem.

What Is GGUF?

GGUF stands for GGML Universal Format. It's a binary file format designed specifically for storing large language models (LLMs) for local inference. If you've ever downloaded a model from Hugging Face to run on your own machine — whether through llama.cpp, Ollama, LM Studio, or text-generation-webui — you've encountered GGUF files.

Before GGUF, the ecosystem used GGML files. The shift from GGML to GGUF happened in mid-to-late 2023, driven by a critical limitation: GGML files had no standard metadata. Tokenizer configuration, model architecture, hyperparameters — all of this was hardcoded into the inference engine or required separate configuration files. Every new model architecture meant a code change. GGUF fixed this by making the model self-describing.

Key insight: GGUF's killer feature is self-description. A single .gguf file contains everything needed to load and run a model — weights, architecture type, tokenizer config, hyperparameters, even rope scaling settings. No sidecar config files needed.

Why GGUF Became the Standard

GGUF won because it solved a real problem: portability. Before GGUF, running a model meant juggling multiple files — model weights in one format, tokenizer files from Hugging Face, a separate config.json for parameters. If you wanted to switch inference engines, you often had to reconvert the model.

GGUF packages everything into a single file. Download one file, point your inference engine at it, and you're running. This simplicity is why GGUF became the lingua franca of local LLM inference.

The format is deeply tied to the llama.cpp ecosystem, but it's supported far beyond it. Ollama uses GGUF internally. LM Studio ships with GGUF models. KoboldCPP, text-generation-webui, GPT4All — they all speak GGUF.

GGUF File Structure: What's Inside the Binary

A GGUF file has three main sections:

1. Header

The header is a fixed-size block at the beginning of the file. It contains:

  • Magic number (GGUF in ASCII, 0x46554747) — identifies the file as GGUF
  • Version number — current version is 3, with v2 and v1 considered legacy
  • Tensor count — how many weight tensors are stored
  • Metadata key-value count — how many metadata entries follow

2. Metadata (Key-Value Pairs)

This is the innovation that sets GGUF apart from GGML. Metadata is stored as a sequence of key-value pairs. Keys are strings, values can be: integers, floats, booleans, strings, or arrays. The metadata section encodes everything the inference engine needs to load the model correctly.

Key metadata fields include:

  • general.architecture — model architecture (e.g., llama, mistral, gemma, qwen2, phi3, starcoder2)
  • general.name — model name
  • llama.block_count — number of transformer layers
  • llama.context_length — maximum context window size
  • llama.embedding_length — hidden dimension size
  • llama.feed_forward_length — FFN intermediate dimension
  • llama.attention.head_count — number of attention heads
  • llama.attention.head_count_kv — number of KV heads (for GQA/MQA)
  • llama.rope.freq_base — RoPE frequency base (important for long context models)
  • llama.rope.scaling.type — RoPE scaling type (linear, yarn, etc.)
  • general.file_type — quantization type used
  • tokenizer.ggml.model — tokenizer type (e.g., llama, gpt2, bert)
  • tokenizer.ggml.tokens — the full vocabulary list
  • tokenizer.ggml.scores — token scores
  • tokenizer.ggml.token_type — token type IDs
  • tokenizer.ggml.merges — BPE merges (for BPE tokenizers)
  • tokenizer.ggml.bos_token_id, eos_token_id — special token IDs

The architecture-specific keys (prefixed like llama., mistral., gemma.) change depending on the general.architecture value. Each model family has its own set of expected metadata keys.

3. Tensor Data

After metadata comes the actual weight data. Each tensor is stored as:

  • Tensor name — e.g., blk.0.attn_q.weight, output.weight
  • Dimensions — shape of the weight matrix
  • Quantization type — how the tensor is quantized (can differ per tensor in some cases)
  • Raw quantized data — the actual bytes

Quantization Types in GGUF: A Practical Guide

Quantization is what makes local LLMs practical. Without it, a 70B-parameter model requires ~140GB of VRAM at FP16. With 4-bit quantization, that drops to ~35GB. GGUF supports a wide range of quantization types, each with different tradeoffs between size and quality.

K-Quant Types (The Standard)

K-quants (named after the K suffix) divide transformer layers into importance tiers. The attention layers get more bits, FFN layers get fewer. This gives better quality-per-bit than uniform quantization.

  • Q2_K — 2-bit, extreme compression. Approx 2.50 bpw (bits per weight). Useable for small demos but quality loss is significant.
  • Q3_K_S / Q3_K_M / Q3_K_L — 3-bit variants. Small (S), Medium (M), Large (L). Q3_K_M is a common "low RAM" choice.
  • Q4_K_S / Q4_K_M — 4-bit variants. Q4_K_M is the gold standard for most users. Excellent quality, ~4.2 bpw, fits most 7B models in ~4.5GB of RAM. This is the default recommendation for local LLM users.
  • Q5_K_S / Q5_K_M — 5-bit variants. Q5_K_M delivers near-FP16 quality, ~5.3 bpw. Recommended if you have the RAM.
  • Q6_K — 6-bit. Very close to FP16 quality. ~6.6 bpw.
  • Q8_0 — 8-bit. Essentially lossless compared to FP16. ~8.5 bpw. Good for CPU inference where memory bandwidth is the bottleneck.

IQ Quantization Types (Newer, Higher Quality)

IQ (Importance-aware Quantization) types are a newer family introduced in llama.cpp in 2024. They use importance matrices to allocate bits more intelligently across tensors.

  • IQ1_S — 1.5 bpw, experimental. Can be useful for extreme compression.
  • IQ2_XXS / IQ2_XS / IQ2_S — 2-bit variants with importance weighting. Better quality than Q2_K at similar sizes.
  • IQ3_XXS — 3-bit with importance weighting. Competes with Q3_K.
  • IQ4_NL / IQ4_XS — 4-bit variants with importance weighting. IQ4_NL (no list) is a non-listed variant that can be faster on some hardware.

How to Choose the Right Quantization

Here's a practical decision tree:

  • You have plenty of VRAM (24GB+): Use Q8_0 or Q6_K. Quality is nearly indistinguishable from FP16.
  • You have 8-16GB VRAM: Use Q5_K_M or Q4_K_M. Q4_K_M is the safe default.
  • You have 4-8GB VRAM: Use Q4_K_M for 7B models, Q3_K_M or Q2_K for larger models.
  • You have less than 4GB VRAM: Q2_K or IQ2_S. Expect quality degradation but it will still run.
  • You're running on CPU only: Q4_K_M or Q5_K_M. CPU inference is memory-bandwidth-limited, so smaller quants help with speed too.

General rule: Q4_K_M is the sweet spot for 90% of users. It's the default recommendation for a reason.

GGUF vs GGML: What Changed?

GGML was the predecessor format. The key differences:

  • Metadata: GGML had none. GGUF stores tokenizer, architecture type, and all parameters inline.
  • Extensibility: GGML required new inference code for each new model architecture. GGUF's metadata system allows new architectures without engine changes.
  • Versioning: GGUF has proper versioning. GGML was essentially unversioned.
  • Data types: GGUF supports more quantization types and is designed to be extensible for future types.

GGML is effectively deprecated. All current tools and models use GGUF.

How GGUF Enables Portability Across Tools

The self-describing nature of GGUF means you can download a single .gguf file and use it with virtually any local LLM tool:

  • llama.cpp — the reference implementation. Command-line inference, server mode with OpenAI-compatible API.
  • Ollama — wraps GGUF models with a user-friendly CLI and REST API. Manages model downloading, modelfiles, and tag-based versioning.
  • LM Studio — a GUI application for Mac, Windows, and Linux. Browse, download, and chat with GGUF models.
  • KoboldCPP — GGUF-focused backend designed for roleplaying and creative writing.
  • text-generation-webui (oobabooga) — full-featured web UI with GGUF loading support.
  • GPT4All — local chat application using GGUF under the hood.
  • Continue.dev — VS Code extension for AI coding assistants, supports GGUF through Ollama or llama.cpp.

This cross-tool compatibility is GGUF's greatest strength. You're not locked into any single application.

How to Inspect GGUF Files

Want to peek inside a GGUF file? Here are the tools:

gguf-dump (Part of llama.cpp)

The quickest way to inspect a GGUF file:

# Install llama.cpp and build the tools
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make gguf-dump

# Dump everything
./gguf-dump model.gguf

# Dump just the metadata (JSON)
./gguf-dump --json model.gguf

This will show you the header info, all metadata key-value pairs, and a list of every tensor with its shape and quantization type.

Python gguf Library

The official Python library lets you inspect GGUF files programmatically:

pip install gguf

from gguf import GGUFReader

reader = GGUFReader("model.gguf")

# Print all metadata
for key, value in reader.fields.items():
    print(f"{key}: {value.value}")

# List all tensors
for tensor in reader.tensors:
    print(f"  {tensor.name}: shape={tensor.shape}, type={tensor.tensor_type}")

Quick Command-Line Check

You can also use head and xxd to quickly verify a file is GGUF:

# Check magic bytes
head -c 4 model.gguf
# Should output: GGUF

# Or with hex dump
xxd -l 16 model.gguf
# First 4 bytes: 47 47 55 46 = "GGUF"

What's Still Missing from the GGUF Format?

The Hacker News discussion (and ongoing community discourse) highlights several gaps in the current format:

1. No Standard Safety Metadata

There's no standardized way to mark a model as "fine-tuned" vs "base," or to include safety evaluation results. If you download a GGUF model from a random Hugging Face repo, you have no reliable way to know what it is — is it the base Llama 3.1? A fine-tune? A fine-tune of a fine-tune that went off the rails? Safety metadata would help.

2. Missing Dataset and Training Provenance

While GGUF can store arbitrary metadata, there's no standard for tracking the training data, base model, or fine-tuning recipe. The machine learning reproducibility problem remains unsolved within the format.

3. No License Field Standard

Some GGUF converters add a general.license field, but it's not standardized. Model licenses (Apache 2.0, Llama 3, CC-BY-NC, etc.) aren't machine-queryable in a reliable way from GGUF metadata.

4. Tokenizer Inconsistencies

Different models use different tokenizers (Llama, Gemma, GPT-2, tiktoken, SentencePiece, etc.), and GGUF stores them all differently. Some tokenizer types require custom code in the inference engine. A more uniform tokenizer encoding would reduce bugs.

5. No Built-In Auditing or Integrity

GGUF files have no checksum system built in. If a file gets corrupted during download, you won't know until inference produces garbage. Some download platforms include checksums, but the format itself doesn't enforce this.

6. Multi-Format Model Support

GGUF is designed for transformers-based language models. It doesn't natively support diffusion models (Stable Diffusion), vision models, or multi-modal architectures. The community would like a format that can handle these too, but that's a much harder problem.

7. Sparse and MoE Model Support

Mixture of Experts (MoE) models pose challenges for quantization in GGUF. While llama.cpp supports MoE inference, the quantization quality for expert weights is an active research area.

Practical Tips for GGUF Users

  • Start with Q4_K_M. It's the safest default. Good quality-to-size ratio, works on most hardware.
  • Check the model's requirements. A 7B model in Q4_K_M needs ~4.5GB RAM. A 13B model in Q4_K_M needs ~8GB. A 70B model needs ~40GB. Don't rely on parameter count alone — always check the file size.
  • Use imatrix (importance matrix) quants when available. Some model creators release imatrix-calibrated quants (suffix: -IQ4_NL, -Q4_K_M with imatrix). These tend to have better quality at the same bitrate.
  • Trust but verify with gguf-dump. Before spending time with a model, run gguf-dump --json your-model.gguf to see its architecture, context length, and metadata. This can catch download errors and model mismatches early.
  • Use the right context length. The metadata tells you the model's native context length. Don't exceed it without proper RoPE scaling settings.
  • Watch for tokenizer divergence. If a model generates garbled text, it might be a tokenizer mismatch in the GGUF conversion. Re-download from a trusted source.
  • For CPU inference, prioritize speed. Use Q4_K_M or Q5_0. Higher quants actually run slower on CPU because there's more data to transfer from memory.

The Bottom Line

GGUF is a pragmatic, well-designed file format that solved a real ecosystem problem. It's not perfect — the missing features around safety, provenance, and integrity are real concerns — but it's hard to argue with the results. Before GGUF, running an LLM locally was a multi-step process with fragile configuration. Now it's "download one file, run one command."

As the local LLM ecosystem matures, expect the format to evolve. Community proposals for safety metadata, checksumming, and improved tokenizer handling are already being discussed. But even as it stands, GGUF has quietly become one of the most important file formats in AI infrastructure.