Metal ESCHA_LINEAR kernel + dense qwen3.5 escha support: 0.037 to 6 t/s

[Manojbhat09]

Summary

Adds a native Metal compute kernel for GGML_OP_ESCHA_LINEAR — the dense
(linear) escha projection used by Qwen3.5 W2-dense models — and wires it through
the ggml backend, model loader, and graph builder. Includes a Q4_K head
conversion path, several hard-won infrastructure fixes, and a verification
toolkit. End-to-end decode goes from 0.037 t/s to ~6 t/s on M4 16GB.

---

What changed

Core kernel (ggml-metal.metal)

kernel_escha_linear — a single Metal compute kernel that fuses:

stage	description
Decode GEMV	2-bit code lookup + per-channel scale + tanh squashing + matmul against input
Staging (f32 to f16)	writes fused (rout*xT) * rin output as f16 for the downstream RoPE / attention graph
Output H	one more code to LUT to sigmoid + softplus product, written back in-place

The kernel is dispatched from the Metal backend (GGML_OP_ESCHA_LINEAR case in
ggml-metal-ops.cpp), registered in the Metal device's supports_op, and
declared in the ggml op enum + ggml_cpu_supports_op fallback.

The decode GEMV path (M=1) is the hot path for token generation. Staging and
output-H are sub-kernels that run for M>=1.

Graph builder (src/models/qwen35.cpp)

• build_escha_mm() — new helper: when llm_escha_dense is populated, calls
  ggml_escha_linear(code, rin, rout, s_in, s_out, bias, input); falls back to
  standard ggml_mul_mat when the sidecar is absent.
• All attention/FFN projections (wq, wk, wv, wo, wg, wu, wd,
  wqkv, wqkv_gate, ssm_out) now route through build_escha_mm().

Loader (src/llama-model-loader.cpp, src/models/qwen35.cpp)

• llm_escha_dense struct: code, rin, rout, s_in, s_out, bias.
• load_escha_linear() — loads the 6 sidecar tensors per projection; reads
  ne[0] of the code tensor to determine bit-width (32 or 48).
• Loader dispatches GGML_OP_ESCHA_LINEAR (dense) vs GGML_OP_ESCHA_MOE
  (routed) based on arch name.

mmap span guard (src/llama-model.cpp)

When mixed code/dense models place GPU tensors far apart in the file with
CPU-only weights in between, BytesNoCopy was wrapping a multi-gigabyte span
for a few hundred MB of actual weights — exhausting the unified-memory wired
limit. The guard now compares the mmap span against the actual tensor bytes and
falls back to a real allocation when the span exceeds the tensors by >64 MiB.

Q4_K head quantization

output.weight (2.4 GB F16) is read every token. Quantizing it to Q4_K saves
~1.8 GB of reads per decode step. The escha codes are byte-identical; logits
change by ~0.1-0.45% — within expected quantization noise. Achieved via:

llama-quantize .../Qwen38-27B-Escha-W2-f16.gguf .../Qwen38-27B-Escha-W2-q4head.gguf F16 Q4_K 1

Output logits verified against f16 baseline via eval-callback checksums.

Instrumentation (env-gated, zero overhead when unset)

env var	effect
GGML_ESCHA_DBG=1	Escha kernels do staging-only (near-zero work, wrong results); for timing bisection
GGML_ESCHA_PERF=1	Per-call escha logging (batch#, M, wall us)
GGML_METAL_SKIP_OPS=X,Y	Skips listed ops during Metal encoding (for timing bisection)
GGML_METAL_PROFILE=1	Encode-time Metal command buffer profiling
GGML_SCHED_TIMING=1	Per-split wall time in ggml_backend_sched_graph_compute
GGML_METAL_DENY_OPS=X,Y	Forces listed ops to CPU backend

GGML_SCHED_TIMING=1 is what ultimately found the root cause of the 9 s/tok
regression (see below).

---

Benchmark

Mac mini M4 16GB, Qwen38-27B-Escha-W2-q4head.gguf (10 GB), -ngl 99 -np 1 -c 2048:

metric	before (MoE-sidecar only)	after (this PR)
prompt eval	~9400 ms/tok	115 ms/tok (8.7 t/s)
token generation	~27000 ms/tok	~165 ms/tok (6.0 t/s)
model file size	12.7 GB (f16 head)	10 GB (Q4_K head)

Cross-check with GGML_ESCHA_DBG=1 (escha approx. free):

• decode drops to 54 ms/tok — escha accounts for ~68% of decode time.
• Non-escha overhead (attn, norms, head GEMV, RoPE) = ~54 ms/tok.

---

The 9-second mystery (solved)

With -ngl 48 (the default set by the previous fit-ladder), every token took
~9.5 s. GGML_SCHED_TIMING=1 revealed the decode graph had two splits:

• Split #0: CPU backend, 1041 nodes — 9509 ms (the entire cost)
• Split Metal ESCHA_LINEAR kernel + dense qwen3.5 escha support: 0.037 to 6 t/s #1: Metal backend, 2854 nodes — 2 ms

The 1041 CPU nodes were the last 16 layers + output head, which -ngl 48 never
offloaded. Switching to -ngl 99 puts everything on the GPU. This single flag
accounts for the vast majority of the speedup.

---

Verification

• Standalone kernel test (tests/test-escha-linear.cpp): compares Metal
  output against C reference at every stage (decode, staging, output-H) for 8
  cases varying IC, OC, M, and code width. All pass; residual norms are fp16
  quantization noise only.
• Model checksums via llama-eval-callback:
  • CPU f16 head: result_norm = -105.259483
  • Metal ngl=99 q4head: result_norm = -105.252533 (6.6e-5 relative)
  • Logits sum: CPU -432089 vs q4head GPU -431636 (0.1%)
• NumPy parity pipeline (tools/escha-numpy-forward.py): stage-wise
  reference at fp32, used during kernel development to catch wire-ups and
  broadcast bugs.

---

Files changed

area	files
Metal kernel	ggml-metal.metal, ggml-metal-ops.cpp/h, ggml-metal-context.m, ggml-metal-device.cpp/m/h, ggml-metal.cpp
ggml core	ggml.h, ggml.c, ggml-backend.cpp
CPU fallback	ggml-cpu/ops.cpp/h, ggml-cpu.c
Graph builder	llama-graph.h, models/qwen35.cpp, models/qwen3next.cpp, models/qwen3moe.cpp, models/models.h
Loader	llama-model-loader.cpp, llama-model.cpp, llama-model.h
Tests	tests/test-escha-linear.cpp, tests/CMakeLists.txt
Tools	tools/escha-convert-dense.py, tools/escha-numpy-forward.py, tools/escha-numpy-goodvar.py, tools/escha-dump-linear-cases.py, tools/diff-cb-numpy.py