A personal research fork of ggml-org/llama.cpp exploring BF12K15 — a 12-bit weight quantization format for LLM inference. This is purely a learning project to understand llama.cpp internals better.
Inspired by brianbell-x/weight-compression.
BF12K15 packs 4 BF16 weights into 6 bytes (12 bits per weight) using a code-word + low-byte scheme. Two variants exist:
- BF12K15 — on-disk format with sparse sidecar tensors for escape corrections. Requires reconstruction to BF16 at load time.
- BF12K15B — self-contained super-block format (256 weights per block, ~394 bytes) with a shared base exponent and up to 8 escape corrections. Runs natively in the CPU backend with a dedicated
vec_dotkernel.
The converter targets 2D weight tensors; 1D tensors (biases, norms) and already-quantized tensors are kept as-is.
| Area | Files | What |
|---|---|---|
| GGML core | ggml/include/ggml.h, ggml/src/ggml.c, ggml/src/ggml-common.h |
GGML_TYPE_BF12K15 and GGML_TYPE_BF12K15B type definitions, block structs |
| CPU backend | ggml/src/ggml-cpu/quants.c, quants.h, ops.cpp, ggml-cpu.c |
dequantize_row_bf12k15_to_bf16, dequantize_row_bf12k15b, scalar ggml_vec_dot_bf12k15b_f32 |
| Metal backend | ggml/src/ggml-metal/ggml-metal-device.m |
Exclude BF12K15B from Metal get_rows and mul_mat_id (falls back to CPU) |
| Model loader | src/llama-model-loader.cpp, llama-model-loader.h |
Sidecar tensor discovery, mmap disable, BF12K15-to-BF16 reconstruction at load |
| gguf-py | gguf-py/gguf/constants.py |
GGMLQuantizationType.BF12K15 enum + block size mapping |
| llama API | include/llama.h |
LLAMA_FTYPE_MOSTLY_BF12K15 |
| Tools | convert_gguf_bf12k15.py, mem_compare.sh |
GGUF conversion script and memory comparison helper |
This is the "barebones working BF12K15 inference" baseline: every 4-weight
sub-block is decoded lane-by-lane with scalar bit manipulation, then accumulated
with a scalar sumf += decoded[j] * y[yi++]. Measured on Apple M4 Pro
(Qwen3-0.6B, batch=1, CPU -ngl 0): ~13 t/s generation — slower than BF16
(~24–30 t/s CPU) despite reading ~25% less weight memory, proving the cost was
decode compute, not bandwidth. On Metal, BF16 runs at ~157 t/s via its GPU
kernel; BF12K15B has no Metal kernel and falls back to this CPU path regardless
of -ngl.
The 4-lane sub-block maps 1:1 to float32x4_t, so the decode+FMA was rewritten
with NEON, guarded by #if defined(__ARM_NEON) inside the existing
ggml_vec_dot_bf12k15b_f32 (scalar #else preserved unchanged for non-ARM).
No header, CMake, or traits change; one symbol, no duplicates.
| File | What |
|---|---|
ggml/src/ggml-cpu/quants.c |
New decode_bf12k15b_subblock_neon (full code→exponent→BF16→F32 decode in uint32x4_t lanes, bit-identical to the scalar decoder); FMA with four independent float32x4_t accumulators (unrolled 16-wide) to break the serial dependency chain; escapes still patched by the unchanged patch_bf12k15b_escapes |
tests/test-quantize-fns.cpp |
New test_vec_dot_bf12k15b — exercises the NEON path with 0–8 escape positions across 5 seeds × {256, 512} weights; asserts |result − ref|/n < 1e-4f |
Result, same CPU backend (-ngl 0): ~30 t/s generation — ~2.3× over the
scalar baseline, reaching parity with CPU BF16 at ~25% smaller weight
memory. Correctness: all test cases pass at ~1e-10 error (threshold 1e-4), and
greedy --temp 0.0 output is unchanged from phase 1.
No special build flags. The NEON path activates via the toolchain's
__ARM_NEON definition (default on Apple Silicon); GGML_NATIVE=ON (the
default) suffices. Non-ARM platforms compile the scalar #else path with no
flags.
cmake --build build --target test-quantize-fns -j # correctness
ctest --test-dir build -R 'quantize-fns' --output-on-failure
cmake --build build --target llama-cli -j # perf
./build/bin/llama-cli -m model-bf12k15.gguf -ngl 0 \
-p "hey ratbag" -n 64 --temp 0.0 --single-turn --no-conversationScope: a CPU-inference win. dequantize_row_bf12k15b (the to_float /
get_rows path) is left scalar — it is not on the dense-model generation hot
path. A Metal kernel for BF12K15B is the remaining lever for GPU inference,
where BF16 (~157 t/s) still leads because BF12K15B falls back to CPU.
# Convert a BF16 GGUF to BF12K15 (requires gguf_convert.py in the same directory)
python3 convert_gguf_bf12k15.py model-bf16.gguf model-bf12k15.gguf
# Run inference (BF12K15B super-block reconstructs natively on CPU)
./build/bin/llama-cli -m model-bf12k15.gguf -p "hello" -n 16
# Compare memory usage
./mem_compare.sh model-bf16.gguf model-bf12k15.ggufAll credit for llama.cpp goes to the ggml-org team. This fork is not intended for upstream contribution.