Skip to content

upstream-bump: patch 0001-turboquant.patch no longer applies #71

Open
Open
upstream-bump: patch 0001-turboquant.patch no longer applies#71
Labels
upstream-bumpAutomated llama.cpp upstream replay

Description

@github-actions
github-actions
bot
opened on Aug 31, 2026

Automated weekly upstream replay failed.
upstream ref master
upstream commit 010be9683afabe14ce299197b38c329f94bae568
previously pinned 3f545beccee69d9975f466ec7e45fd9aacd8ba90
failing patch 0001-turboquant.patch
run https://github.com/mjwsolo/localcode/actions/runs/33414421434

Upstream changed the code this patch touches, so the patch needs rebasing.
See docs/upstream-fork.md → Rebasing a patch. If upstream has
merged the equivalent change, delete the patch instead.

git apply output and reject hunks 
Patch that failed:  0001-turboquant.patch
Upstream commit:    010be9683afabe14ce299197b38c329f94bae568
Previously pinned:  3f545beccee69d9975f466ec7e45fd9aacd8ba90

git apply output:
Checking patch common/arg.cpp...
Checking patch ggml/include/ggml.h...
Hunk #4 succeeded at 2677 (offset 12 lines).
Checking patch ggml/src/CMakeLists.txt...
Checking patch ggml/src/ggml-common.h...
Checking patch ggml/src/ggml-cpu/ggml-cpu.c...
Hunk #6 succeeded at 3048 (offset 2 lines).
Hunk #7 succeeded at 3496 (offset 2 lines).
Checking patch ggml/src/ggml-cpu/ops.cpp...
Hunk #5 succeeded at 4675 (offset 133 lines).
Hunk #6 succeeded at 4954 (offset 133 lines).
Hunk #7 succeeded at 5181 (offset 133 lines).
Hunk #8 succeeded at 5267 (offset 133 lines).
Hunk #9 succeeded at 5947 (offset 133 lines).
Hunk #10 succeeded at 11251 (offset 143 lines).
Hunk #11 succeeded at 11503 (offset 143 lines).
Checking patch ggml/src/ggml-cpu/ops.h...
Checking patch ggml/src/ggml-cuda/CMakeLists.txt...
Checking patch ggml/src/ggml-cuda/convert.cu...
Checking patch ggml/src/ggml-cuda/dequantize.cuh...
Checking patch ggml/src/ggml-cuda/fattn-common.cuh...
Checking patch ggml/src/ggml-cuda/fattn-mma-f16.cuh...
Checking patch ggml/src/ggml-cuda/fattn-tile.cu...
Checking patch ggml/src/ggml-cuda/fattn-tile.cuh...
Checking patch ggml/src/ggml-cuda/fattn-vec.cuh...
Checking patch ggml/src/ggml-cuda/fattn.cu...
Checking patch ggml/src/ggml-cuda/ggml-cuda.cu...
Hunk #1 succeeded at 62 (offset 1 line).
Hunk #2 succeeded at 785 (offset 1 line).
error: while searching for:
        : ggml_nbytes(tensor);
    int64_t ne0 = tensor->ne[0];

    if (ggml_is_quantized(tensor->type)) {
        if (ne0 % MATRIX_ROW_PADDING != 0) {
            GGML_ASSERT(tensor->nb[0] == ggml_element_size(tensor));

error: patch failed: ggml/src/ggml-cuda/ggml-cuda.cu:914
Hunk #4 succeeded at 1836 (offset -5 lines).
Hunk #5 succeeded at 1879 (offset -5 lines).
Hunk #6 succeeded at 1974 (offset -5 lines).
Hunk #7 succeeded at 2143 (offset -5 lines).
Hunk #8 succeeded at 2627 (offset 1 line).
Hunk #9 succeeded at 5054 (offset 11 lines).
Hunk #10 succeeded at 5090 (offset 11 lines).
Hunk #11 succeeded at 5108 (offset 11 lines).
Hunk #12 succeeded at 5294 (offset 11 lines).
Checking patch ggml/src/ggml-cuda/mmvq-tq.cu...
Checking patch ggml/src/ggml-cuda/mmvq-tq.cuh...
Checking patch ggml/src/ggml-cuda/set-rows.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-mma-f16-instance-ncols1_1-ncols2_16.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-mma-f16-instance-ncols1_2-ncols2_16.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-mma-f16-instance-ncols1_4-ncols2_16.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-tile-instance-dkq640-dv512.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-q8_0-turbo2_0.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-q8_0-turbo3_0.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-q8_0-turbo4_0.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo2_0-q8_0.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo2_0-turbo2_0.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo2_0-turbo3_0.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo2_0-turbo4_0.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo3_0-q8_0.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo3_0-turbo2_0.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo3_0-turbo3_0.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo3_0-turbo4_0.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo4_0-q8_0.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo4_0-turbo2_0.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo4_0-turbo3_0.cu...
Checking patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo4_0-turbo4_0.cu...
Checking patch ggml/src/ggml-cuda/turbo-innerq.cu...
Checking patch ggml/src/ggml-cuda/turbo-innerq.cuh...
Checking patch ggml/src/ggml-cuda/turbo-quant.cuh...
Checking patch ggml/src/ggml-cuda/turbo-wht.cu...
Checking patch ggml/src/ggml-cuda/turbo-wht.cuh...
Checking patch ggml/src/ggml-cuda/vendors/hip.h...
Checking patch ggml/src/ggml-hip/CMakeLists.txt...
Checking patch ggml/src/ggml-metal/ggml-metal-device.cpp...
Hunk #1 succeeded at 726 (offset 23 lines).
Hunk #2 succeeded at 911 (offset 23 lines).
Hunk #3 succeeded at 1046 (offset 23 lines).
error: while searching for:
        res = ggml_metal_library_compile_pipeline(lib, base, name, nullptr);
    }

    res.smem = (size_t) ne02*ne20*sizeof(uint16_t);

    return res;
}

error: patch failed: ggml/src/ggml-metal/ggml-metal-device.cpp:1006
Hunk #5 succeeded at 1233 (offset 17 lines).
Hunk #6 succeeded at 1634 (offset 17 lines).
error: while searching for:

    const char * type = use_kv_f16 ? "f16" : ggml_type_name(op->src[1]->type);

    snprintf(base, 256, "kernel_%s_%s_dk%d_dv%d",
            "flash_attn_ext_vec",
            type,
            dk,
            dv);

    snprintf(name, 256, "%s_mask=%d_sink=%d_bias=%d_scap=%d_kvpad=%d_ns10=%d_ns20=%d_nsg=%d_nwg=%d",
            base,
            has_mask,

error: patch failed: ggml/src/ggml-metal/ggml-metal-device.cpp:1559
Checking patch ggml/src/ggml-metal/ggml-metal-device.h...
Hunk #1 succeeded at 134 (offset 1 line).
Checking patch ggml/src/ggml-metal/ggml-metal-device.m...
error: while searching for:
                [prep setObject:@"1" forKey:@"GGML_METAL_EMBED_LIBRARY"];
#endif

                MTLCompileOptions * options = [MTLCompileOptions new];
                options.preprocessorMacros = prep;


error: patch failed: ggml/src/ggml-metal/ggml-metal-device.m:226
Hunk #2 succeeded at 1647 (offset 320 lines).
error: while searching for:
            return true;
        case GGML_OP_GATED_DELTA_NET:
            return has_simdgroup_reduction && op->src[2]->ne[0] % 32 == 0;
        case GGML_OP_SOLVE_TRI:
        case GGML_OP_MUL_MAT:
        case GGML_OP_MUL_MAT_ID:

error: patch failed: ggml/src/ggml-metal/ggml-metal-device.m:1385
Hunk #4 succeeded at 1817 (offset 324 lines).
Hunk #5 succeeded at 1855 (offset 324 lines).
Checking patch ggml/src/ggml-metal/ggml-metal-impl.h...
Hunk #3 succeeded at 973 (offset 7 lines).
Checking patch ggml/src/ggml-metal/ggml-metal-ops.cpp...
Hunk #1 succeeded at 173 (offset 1 line).
Hunk #2 succeeded at 230 (offset 1 line).
Hunk #3 succeeded at 381 (offset 1 line).
Hunk #4 succeeded at 1954 (offset 31 lines).
Hunk #5 succeeded at 2618 (offset 22 lines).
error: while searching for:

    const uint32_t r2 = 1;
    const uint32_t r3 = 1;

    // find the break-even point where the matrix-matrix kernel becomes more efficient compared
    // to the matrix-vector kernel
    // ne20 = n_used_experts
    // ne21 = n_rows (batch size)
    const int ne21_mm_id_min = 32;

    if (props_dev->has_simdgroup_mm && ne00 >= 64 && (ne21 >= ne21_mm_id_min)) {
        // some Metal matrix data types require aligned pointers

error: patch failed: ggml/src/ggml-metal/ggml-metal-ops.cpp:2590
Hunk #7 succeeded at 2733 (offset 15 lines).
Hunk #8 succeeded at 2875 (offset 15 lines).
Hunk #9 succeeded at 2940 (offset 15 lines).
Hunk #10 succeeded at 3170 (offset 15 lines).
error: while searching for:
        // ne20*(nsg)
        // each simdgroup has a full f32 head vector in shared mem to accumulate results
        //
#define FATTN_SMEM(nsg) (GGML_PAD(((GGML_PAD(ne00, 128) + 4*ncpsg + 2*GGML_PAD(ne20, 128))*(nsg))*(sizeof(float)/2), 16))

        int64_t nsg = 1;


error: patch failed: ggml/src/ggml-metal/ggml-metal-ops.cpp:3410
Checking patch ggml/src/ggml-metal/ggml-metal-ops.h...
Checking patch ggml/src/ggml-metal/ggml-metal.metal...
error: ggml/src/ggml-metal/ggml-metal.metal: No such file or directory
Checking patch ggml/src/ggml-metal/turbo-matrices.h...
Checking patch ggml/src/ggml-metal/turbo-wht.h...
Checking patch ggml/src/ggml-quants.c...
Checking patch ggml/src/ggml-quants.h...
Checking patch ggml/src/ggml-turbo-quant.c...
Checking patch ggml/src/ggml-vulkan/ggml-vulkan.cpp...
Hunk #1 succeeded at 5505 (offset 94 lines).
Hunk #2 succeeded at 5535 (offset 94 lines).
Hunk #3 succeeded at 5565 (offset 94 lines).
Hunk #4 succeeded at 5638 (offset 94 lines).
Hunk #5 succeeded at 5651 (offset 94 lines).
Hunk #6 succeeded at 5669 (offset 94 lines).
Hunk #7 succeeded at 18791 (offset 517 lines).
Hunk #8 succeeded at 18818 (offset 517 lines).
Hunk #9 succeeded at 18844 (offset 517 lines).
Hunk #10 succeeded at 18862 (offset 517 lines).
Checking patch ggml/src/ggml-vulkan/vulkan-shaders/copy_to_quant.comp...
Checking patch ggml/src/ggml-vulkan/vulkan-shaders/dequant_funcs.glsl...
Checking patch ggml/src/ggml-vulkan/vulkan-shaders/dequant_funcs_cm2.glsl...
Checking patch ggml/src/ggml-vulkan/vulkan-shaders/dequant_turbo3_0.comp...
Checking patch ggml/src/ggml-vulkan/vulkan-shaders/types.glsl...
Checking patch ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp...
Checking patch ggml/src/ggml.c...
Hunk #6 succeeded at 6561 (offset 27 lines).
Hunk #7 succeeded at 8089 (offset 27 lines).
Checking patch gguf-py/gguf/constants.py...
Hunk #1 succeeded at 5511 (offset 127 lines).
Hunk #2 succeeded at 5570 (offset 127 lines).
Hunk #3 succeeded at 5709 (offset 127 lines).
Checking patch include/llama.h...
Checking patch src/llama-context.cpp...
Hunk #1 succeeded at 3687 (offset 81 lines).
Hunk #2 succeeded at 3706 (offset 81 lines).
Hunk #3 succeeded at 3724 (offset 81 lines).
Checking patch src/llama-graph.cpp...
Hunk #1 succeeded at 2553 (offset -6 lines).
Hunk #2 succeeded at 2583 (offset -6 lines).
Hunk #3 succeeded at 2663 (offset -6 lines).
Hunk #4 succeeded at 2692 (offset -6 lines).
Hunk #5 succeeded at 2857 (offset -6 lines).
Hunk #6 succeeded at 2981 (offset -6 lines).
Hunk #7 succeeded at 3175 (offset -6 lines).
Checking patch src/llama-kv-cache.cpp...
Hunk #1 succeeded at 59 (offset 2 lines).
Hunk #2 succeeded at 154 (offset 3 lines).
error: while searching for:
            throw std::runtime_error("failed to create ggml context for kv cache");
        }

        const bool has_k = true;
        const bool has_v = !is_mla;

        ggml_tensor * k = has_k ? ggml_new_tensor_3d(ctx, type_k, n_embd_k_gqa, kv_size, n_stream) : nullptr;
        ggml_tensor * v = has_v ? ggml_new_tensor_3d(ctx, type_v, n_embd_v_gqa, kv_size, n_stream) : nullptr;

        has_k && ggml_format_name(k, "cache_k_l%d", il);
        has_v && ggml_format_name(v, "cache_v_l%d", il);

error: patch failed: src/llama-kv-cache.cpp:225
Hunk #4 succeeded at 281 (offset -93 lines).
Hunk #5 succeeded at 345 (offset -93 lines).
Hunk #6 succeeded at 388 (offset -93 lines).
Hunk #7 succeeded at 458 (offset -93 lines).
Hunk #8 succeeded at 1367 (offset -79 lines).
Hunk #9 succeeded at 1398 (offset -79 lines).
Hunk #10 succeeded at 1435 (offset -79 lines).
Hunk #11 succeeded at 1471 (offset -79 lines).
Hunk #12 succeeded at 1490 (offset -79 lines).
Hunk #13 succeeded at 1524 (offset -79 lines).
Hunk #14 succeeded at 2435 (offset 62 lines).
Hunk #15 succeeded at 2460 (offset 62 lines).
Hunk #16 succeeded at 2707 (offset 100 lines).
Hunk #17 succeeded at 2748 (offset 100 lines).
Hunk #18 succeeded at 2913 (offset 100 lines).
Hunk #19 succeeded at 2956 (offset 100 lines).
Checking patch src/llama-kv-cache.h...
Hunk #1 succeeded at 189 (offset 13 lines).
Hunk #2 succeeded at 319 (offset 24 lines).
Hunk #3 succeeded at 414 (offset 25 lines).
Checking patch src/llama-memory-hybrid.cpp...
Checking patch src/llama-memory-hybrid.h...
Checking patch src/llama-memory.h...
Checking patch src/llama-model-loader.cpp...
Hunk #2 succeeded at 762 (offset 5 lines).
Checking patch src/llama-quant.cpp...
Hunk #1 succeeded at 854 (offset 18 lines).
Checking patch src/turbo-rotation-data-32.h...
Checking patch src/turbo-rotation-data.h...
Checking patch tools/llama-bench/llama-bench.cpp...
Hunk #1 succeeded at 516 (offset 16 lines).
Checking patch tools/quantize/quantize.cpp...
Applied patch common/arg.cpp cleanly.
Applied patch ggml/include/ggml.h cleanly.
Applied patch ggml/src/CMakeLists.txt cleanly.
Applied patch ggml/src/ggml-common.h cleanly.
Applied patch ggml/src/ggml-cpu/ggml-cpu.c cleanly.
Applied patch ggml/src/ggml-cpu/ops.cpp cleanly.
Applied patch ggml/src/ggml-cpu/ops.h cleanly.
Applied patch ggml/src/ggml-cuda/CMakeLists.txt cleanly.
Applied patch ggml/src/ggml-cuda/convert.cu cleanly.
Applied patch ggml/src/ggml-cuda/dequantize.cuh cleanly.
Applied patch ggml/src/ggml-cuda/fattn-common.cuh cleanly.
Applied patch ggml/src/ggml-cuda/fattn-mma-f16.cuh cleanly.
Applied patch ggml/src/ggml-cuda/fattn-tile.cu cleanly.
Applied patch ggml/src/ggml-cuda/fattn-tile.cuh cleanly.
Applied patch ggml/src/ggml-cuda/fattn-vec.cuh cleanly.
Applied patch ggml/src/ggml-cuda/fattn.cu cleanly.
Applying patch ggml/src/ggml-cuda/ggml-cuda.cu with 1 reject...
Hunk #1 applied cleanly.
Hunk #2 applied cleanly.
Rejected hunk #3.
Hunk #4 applied cleanly.
Hunk #5 applied cleanly.
Hunk #6 applied cleanly.
Hunk #7 applied cleanly.
Hunk #8 applied cleanly.
Hunk #9 applied cleanly.
Hunk #10 applied cleanly.
Hunk #11 applied cleanly.
Hunk #12 applied cleanly.
Applied patch ggml/src/ggml-cuda/mmvq-tq.cu cleanly.
Applied patch ggml/src/ggml-cuda/mmvq-tq.cuh cleanly.
Applied patch ggml/src/ggml-cuda/set-rows.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-mma-f16-instance-ncols1_1-ncols2_16.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-mma-f16-instance-ncols1_2-ncols2_16.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-mma-f16-instance-ncols1_4-ncols2_16.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-tile-instance-dkq640-dv512.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-q8_0-turbo2_0.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-q8_0-turbo3_0.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-q8_0-turbo4_0.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo2_0-q8_0.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo2_0-turbo2_0.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo2_0-turbo3_0.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo2_0-turbo4_0.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo3_0-q8_0.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo3_0-turbo2_0.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo3_0-turbo3_0.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo3_0-turbo4_0.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo4_0-q8_0.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo4_0-turbo2_0.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo4_0-turbo3_0.cu cleanly.
Applied patch ggml/src/ggml-cuda/template-instances/fattn-vec-instance-turbo4_0-turbo4_0.cu cleanly.
Applied patch ggml/src/ggml-cuda/turbo-innerq.cu cleanly.
Applied patch ggml/src/ggml-cuda/turbo-innerq.cuh cleanly.
Applied patch ggml/src/ggml-cuda/turbo-quant.cuh cleanly.
Applied patch ggml/src/ggml-cuda/turbo-wht.cu cleanly.
Applied patch ggml/src/ggml-cuda/turbo-wht.cuh cleanly.
Applied patch ggml/src/ggml-cuda/vendors/hip.h cleanly.
Applied patch ggml/src/ggml-hip/CMakeLists.txt cleanly.
Applying patch ggml/src/ggml-metal/ggml-metal-device.cpp with 2 rejects...
Hunk #1 applied cleanly.
Hunk #2 applied cleanly.
Hunk #3 applied cleanly.
Rejected hunk #4.
Hunk #5 applied cleanly.
Hunk #6 applied cleanly.
Rejected hunk #7.
Applied patch ggml/src/ggml-metal/ggml-metal-device.h cleanly.
Applying patch ggml/src/ggml-metal/ggml-metal-device.m with 2 rejects...
Rejected hunk #1.
Hunk #2 applied cleanly.
Rejected hunk #3.
Hunk #4 applied cleanly.
Hunk #5 applied cleanly.
Applied patch ggml/src/ggml-metal/ggml-metal-impl.h cleanly.
Applying patch ggml/src/ggml-metal/ggml-metal-ops.cpp with 2 rejects...
Hunk #1 applied cleanly.
Hunk #2 applied cleanly.
Hunk #3 applied cleanly.
Hunk #4 applied cleanly.
Hunk #5 applied cleanly.
Rejected hunk #6.
Hunk #7 applied cleanly.
Hunk #8 applied cleanly.
Hunk #9 applied cleanly.
Hunk #10 applied cleanly.
Rejected hunk #11.
Applied patch ggml/src/ggml-metal/ggml-metal-ops.h cleanly.
Applied patch ggml/src/ggml-metal/turbo-matrices.h cleanly.
Applied patch ggml/src/ggml-metal/turbo-wht.h cleanly.
Applied patch ggml/src/ggml-quants.c cleanly.
Applied patch ggml/src/ggml-quants.h cleanly.
Applied patch ggml/src/ggml-turbo-quant.c cleanly.
Applied patch ggml/src/ggml-vulkan/ggml-vulkan.cpp cleanly.
Applied patch ggml/src/ggml-vulkan/vulkan-shaders/copy_to_quant.comp cleanly.
Applied patch ggml/src/ggml-vulkan/vulkan-shaders/dequant_funcs.glsl cleanly.
Applied patch ggml/src/ggml-vulkan/vulkan-shaders/dequant_funcs_cm2.glsl cleanly.
Applied patch ggml/src/ggml-vulkan/vulkan-shaders/dequant_turbo3_0.comp cleanly.
Applied patch ggml/src/ggml-vulkan/vulkan-shaders/types.glsl cleanly.
Applied patch ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp cleanly.
Applied patch ggml/src/ggml.c cleanly.
Applied patch gguf-py/gguf/constants.py cleanly.
Applied patch include/llama.h cleanly.
Applied patch src/llama-context.cpp cleanly.
Applied patch src/llama-graph.cpp cleanly.
Applying patch src/llama-kv-cache.cpp with 1 reject...
Hunk #1 applied cleanly.
Hunk #2 applied cleanly.
Rejected hunk #3.
Hunk #4 applied cleanly.
Hunk #5 applied cleanly.
Hunk #6 applied cleanly.
Hunk #7 applied cleanly.
Hunk #8 applied cleanly.
Hunk #9 applied cleanly.
Hunk #10 applied cleanly.
Hunk #11 applied cleanly.
Hunk #12 applied cleanly.
Hunk #13 applied cleanly.
Hunk #14 applied cleanly.
Hunk #15 applied cleanly.
Hunk #16 applied cleanly.
Hunk #17 applied cleanly.
Hunk #18 applied cleanly.
Hunk #19 applied cleanly.
Applied patch src/llama-kv-cache.h cleanly.
Applied patch src/llama-memory-hybrid.cpp cleanly.
Applied patch src/llama-memory-hybrid.h cleanly.
Applied patch src/llama-memory.h cleanly.
Applied patch src/llama-model-loader.cpp cleanly.
Applied patch src/llama-quant.cpp cleanly.
Applied patch src/turbo-rotation-data-32.h cleanly.
Applied patch src/turbo-rotation-data.h cleanly.
Applied patch tools/llama-bench/llama-bench.cpp cleanly.
Applied patch tools/quantize/quantize.cpp cleanly.

Reject hunks:

--- ggml/src/ggml-cuda/ggml-cuda.cu.rej
diff a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu	(rejected hunks)
@@ -914,6 +956,13 @@ static size_t ggml_backend_cuda_buffer_type_get_alloc_size(ggml_backend_buffer_t
         : ggml_nbytes(tensor);
     int64_t ne0 = tensor->ne[0];
 
+    // TQ4_1S → q8_0 load-time conversion: allocate q8_0-sized space if opted in
+    if (ggml_tq_convert_q8() && tensor->type == GGML_TYPE_TQ4_1S) {
+        // q8_0 block: 34 bytes per 32 elements. TQ4_1S block: 20 bytes per 32 elements.
+        const int64_t n_blocks = ggml_nelements(tensor) / QK_TQ4_1S;
+        size = n_blocks * sizeof(block_q8_0);
+    }
+
     if (ggml_is_quantized(tensor->type)) {
         if (ne0 % MATRIX_ROW_PADDING != 0) {
             GGML_ASSERT(tensor->nb[0] == ggml_element_size(tensor));

--- ggml/src/ggml-metal/ggml-metal-device.cpp.rej
diff a/ggml/src/ggml-metal/ggml-metal-device.cpp b/ggml/src/ggml-metal/ggml-metal-device.cpp	(rejected hunks)
@@ -1006,7 +1110,14 @@ ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_mul_mm_id_map0(g
         res = ggml_metal_library_compile_pipeline(lib, base, name, nullptr);
     }
 
+    // Graph reservation may pass worst-case ne20=ne02 (e.g. 256*256*2=128KB).
+    // At runtime ne20 is the actual n_expert_used (e.g. 8), keeping shmem within limits.
+    // Cap to 32KB (Apple Silicon threadgroup memory limit) to prevent reservation assert
+    // on high-expert-count MoE models (Qwen3.5-35B with 256 experts).
     res.smem = (size_t) ne02*ne20*sizeof(uint16_t);
+    if (res.smem > 32768) {
+        res.smem = 32768;
+    }
 
     return res;
 }
@@ -1559,12 +1686,15 @@ ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_flash_attn_ext_v
 
     const char * type = use_kv_f16 ? "f16" : ggml_type_name(op->src[1]->type);
 
-    snprintf(base, 256, "kernel_%s_%s_dk%d_dv%d",
+    // Asymmetric K/V: always encode both K and V types in the pipeline name.
+    // Uses k/v prefix to avoid ambiguity with type names containing underscores.
+    snprintf(base, 256, "kernel_%s_k%s_v%s_dk%d_dv%d",
             "flash_attn_ext_vec",
             type,
             dk,
             dv);
 
+
     snprintf(name, 256, "%s_mask=%d_sink=%d_bias=%d_scap=%d_kvpad=%d_ns10=%d_ns20=%d_nsg=%d_nwg=%d",
             base,
             has_mask,

--- ggml/src/ggml-metal/ggml-metal-device.m.rej
diff a/ggml/src/ggml-metal/ggml-metal-device.m b/ggml/src/ggml-metal/ggml-metal-device.m	(rejected hunks)
@@ -226,6 +226,43 @@ ggml_metal_library_t ggml_metal_library_init(ggml_metal_device_t dev) {
                 [prep setObject:@"1" forKey:@"GGML_METAL_EMBED_LIBRARY"];
 #endif
 
+                // TurboQuant: auto-select dequant path based on hardware
+                // M1/M2/M3/M4 (no tensor API): 4-mag LUT (+38-45% decode at long ctx)
+                // M5+ (has tensor API): 8-entry full LUT (best decode speed)
+                {
+                    const char * force_4mag = getenv("TURBO_FORCE_4MAG");
+                    // Always compile with 4-mag support. The dispatch code selects
+                    // 4-mag vs 8-LUT based on context depth at runtime.
+                    // Pre-M5: always 4-mag (constant cache too slow)
+                    // M5+: 4-mag for mid-context (8K-20K), 8-LUT otherwise
+                    if (!ggml_metal_device_get_props(dev)->has_tensor || (force_4mag && force_4mag[0] == '1')) {
+                        [prep setObject:@"1" forKey:@"TURBO_USE_4MAG"];
+                        GGML_LOG_INFO("%s: turbo3 using 4-mag LUT%s\n", __func__,
+                            force_4mag ? " (forced)" : " (pre-M5 hardware)");
+                    }
+                    // Sparse V dequant: skip V for negligible attention weights
+                    // Enabled by default on all Metal (validated: PPL identical, NIAH 9/9, 30+ testers)
+                    // Opt-out via TURBO_SPARSE_V=0
+                    const char * sparse_v_env = getenv("TURBO_SPARSE_V");
+                    const bool sparse_v_disabled = sparse_v_env && sparse_v_env[0] == '0';
+                    if (!sparse_v_disabled) {
+                        [prep setObject:@"1" forKey:@"TURBO_SPARSE_V"];
+                        GGML_LOG_INFO("%s: turbo3 sparse V dequant enabled (opt-out: TURBO_SPARSE_V=0)\n", __func__);
+                    }
+                    // TODO: context-adaptive dispatch — compile both 4-mag and 8-LUT
+                    // FA kernel instantiations, select based on ne11 (KV cache size)
+                    // at dispatch time in ggml_metal_op_flash_attn_ext()
+                }
+
+                // TurboQuant profiling: set TURBO_PROFILE_MODE env var (0-4)
+                {
+                    const char * pm = getenv("TURBO_PROFILE_MODE");
+                    if (pm && pm[0] >= '0' && pm[0] <= '4') {
+                        [prep setObject:[NSString stringWithUTF8String:pm] forKey:@"TURBO_PROFILE_MODE"];
+                        GGML_LOG_INFO("%s: TURBO_PROFILE_MODE=%s\n", __func__, pm);
+                    }
+                }
+
                 MTLCompileOptions * options = [MTLCompileOptions new];
                 options.preprocessorMacros = prep;
 
@@ -1385,6 +1438,8 @@ bool ggml_metal_device_supports_op(ggml_metal_device_t dev, const struct ggml_te
             return true;
         case GGML_OP_GATED_DELTA_NET:
             return has_simdgroup_reduction && op->src[2]->ne[0] % 32 == 0;
+        case GGML_OP_TURBO_WHT:
+            return op->src[0]->ne[0] % 128 == 0;
         case GGML_OP_SOLVE_TRI:
         case GGML_OP_MUL_MAT:
         case GGML_OP_MUL_MAT_ID:

--- ggml/src/ggml-metal/ggml-metal-ops.cpp.rej
diff a/ggml/src/ggml-metal/ggml-metal-ops.cpp b/ggml/src/ggml-metal/ggml-metal-ops.cpp	(rejected hunks)
@@ -2590,12 +2658,13 @@ int ggml_metal_op_mul_mat_id(ggml_metal_op_t ctx, int idx) {
 
     const uint32_t r2 = 1;
     const uint32_t r3 = 1;
+    const bool is_tq_weight = (op->src[0]->type == GGML_TYPE_TQ3_1S || op->src[0]->type == GGML_TYPE_TQ4_1S);
 
     // find the break-even point where the matrix-matrix kernel becomes more efficient compared
     // to the matrix-vector kernel
     // ne20 = n_used_experts
     // ne21 = n_rows (batch size)
-    const int ne21_mm_id_min = 32;
+    const int ne21_mm_id_min = is_tq_weight ? 1 : 32;
 
     if (props_dev->has_simdgroup_mm && ne00 >= 64 && (ne21 >= ne21_mm_id_min)) {
         // some Metal matrix data types require aligned pointers
@@ -3410,7 +3563,8 @@ int ggml_metal_op_flash_attn_ext(ggml_metal_op_t ctx, int idx) {
         // ne20*(nsg)
         // each simdgroup has a full f32 head vector in shared mem to accumulate results
         //
-#define FATTN_SMEM(nsg) (GGML_PAD(((GGML_PAD(ne00, 128) + 4*ncpsg + 2*GGML_PAD(ne20, 128))*(nsg))*(sizeof(float)/2), 16))
+// Extra 128 floats (512 bytes) for TurboQuant pre-dequantized block cache in threadgroup memory
+#define FATTN_SMEM(nsg) (GGML_PAD(((GGML_PAD(ne00, 128) + 4*ncpsg + 2*GGML_PAD(ne20, 128))*(nsg))*(sizeof(float)/2) + 128*sizeof(float), 16))
 
         int64_t nsg = 1;
 

--- src/llama-kv-cache.cpp.rej
diff a/src/llama-kv-cache.cpp b/src/llama-kv-cache.cpp	(rejected hunks)
@@ -225,11 +265,107 @@ llama_kv_cache::llama_kv_cache(
             throw std::runtime_error("failed to create ggml context for kv cache");
         }
 
+        // TurboQuant zero-padding: for models with non-128-aligned head_dim (e.g. DeepSeek
+        // head_dim_k=192), pad each head to the next multiple of 128. The padded zeros don't
+        // affect dot products since WHT preserves inner products:
+        //   <WHT(Q_padded), WHT(K_padded)> = <Q_padded, K_padded> = <Q, K> + <0, 0> = <Q, K>
+        const uint32_t n_embd_head_k = hparams.n_embd_head_k(il);
+
+
         const bool has_k = true;
         const bool has_v = !is_mla;
 
-        ggml_tensor * k = has_k ? ggml_new_tensor_3d(ctx, type_k, n_embd_k_gqa, kv_size, n_stream) : nullptr;
-        ggml_tensor * v = has_v ? ggml_new_tensor_3d(ctx, type_v, n_embd_v_gqa, kv_size, n_stream) : nullptr;
+        // Layer-adaptive: use higher precision for quality-sensitive layers
+        // Config: TURBO_LAYER_ADAPTIVE env var controls the strategy
+        //   0 = uniform (default)
+        //   1 = q8_0 K+V for first+last 4 layers
+        //   2 = q8_0 K+V for last 8 layers
+        //   5 = Boundary V: first2+last2 V=turbo4, rest V=turbo2 (K unchanged)
+        //   6 = V-only: last 8 V=turbo4, rest V=turbo2 (K unchanged)
+        //   7 = Boundary V (recommended): first2+last2 V=q8_0, rest V=turbo2 (K unchanged)
+        ggml_type layer_type_k = type_k;
+        ggml_type layer_type_v = type_v;
+        {
+            static const int adaptive_mode = [&]() {
+                const char * env = getenv("TURBO_LAYER_ADAPTIVE");
+                if (env) {
+                    int mode = atoi(env);
+                    if (mode > 0) {
+                        LLAMA_LOG_INFO("llama_kv_cache: layer-adaptive mode %d enabled (env)\n", mode);
+                    }
+                    return mode;
+                }
+                // Auto-enable Boundary V (mode 7) when V is turbo2
+                if (type_v == GGML_TYPE_TURBO2_0 && hparams.n_layer >= 8) {
+                    LLAMA_LOG_INFO("llama_kv_cache: Boundary V auto-enabled for turbo2-V (opt-out: TURBO_LAYER_ADAPTIVE=0)\n");
+                    return 7;
+                }
+                return 0;
+            }();
+            const bool is_turbo = (type_k == GGML_TYPE_TURBO3_0 || type_k == GGML_TYPE_TURBO4_0 || type_k == GGML_TYPE_TURBO2_0);
+            const bool v_is_turbo = (type_v == GGML_TYPE_TURBO3_0 || type_v == GGML_TYPE_TURBO4_0 || type_v == GGML_TYPE_TURBO2_0);
+            const uint32_t n_layer = hparams.n_layer;
+            if (adaptive_mode == 1 && is_turbo && n_layer >= 8) {
+                if (il < 4 || il >= n_layer - 4) {
+                    layer_type_k = GGML_TYPE_Q8_0;
+                    layer_type_v = GGML_TYPE_Q8_0;
+                }
+            } else if (adaptive_mode == 2 && is_turbo && n_layer >= 8) {
+                if (il >= n_layer - 8) {
+                    layer_type_k = GGML_TYPE_Q8_0;
+                    layer_type_v = GGML_TYPE_Q8_0;
+                }
+            } else if (adaptive_mode == 5 && v_is_turbo && n_layer >= 8) {
+                // Boundary V (turbo4 boundaries): first2+last2 V=turbo4, rest V=turbo2
+                const bool is_boundary = (il < 2 || il >= n_layer - 2);
+                layer_type_v = is_boundary ? GGML_TYPE_TURBO4_0 : GGML_TYPE_TURBO2_0;
+                if (il == 0) {
+                    LLAMA_LOG_INFO("llama_kv_cache: Boundary V mode 5: first2+last2 V=turbo4, rest V=turbo2\n");
+                }
+            } else if (adaptive_mode == 6 && v_is_turbo && n_layer >= 8) {
+                // V-only: last 8 V=turbo4, rest V=turbo2
+                layer_type_v = (il >= n_layer - 8) ? GGML_TYPE_TURBO4_0 : GGML_TYPE_TURBO2_0;
+                if (il == 0) {
+                    LLAMA_LOG_INFO("llama_kv_cache: V-only LA mode 6: last8 V=turbo4, rest V=turbo2\n");
+                }
+            } else if (adaptive_mode == 7 && v_is_turbo && n_layer >= 8) {
+                // Boundary V (recommended): first2+last2 V=q8_0, rest V=turbo2
+                const bool is_boundary = (il < 2 || il >= n_layer - 2);
+                layer_type_v = is_boundary ? GGML_TYPE_Q8_0 : GGML_TYPE_TURBO2_0;
+                if (il == 0) {
+                    LLAMA_LOG_INFO("llama_kv_cache: Boundary V mode 7: first2+last2 V=q8_0, rest V=turbo2\n");
+                }
+            }
+        }
+        // For turbo types, pad K head_dim to next multiple of 128 for full WHT groups
+        uint32_t n_embd_k_gqa_eff = n_embd_k_gqa;
+        const bool k_is_turbo = (layer_type_k == GGML_TYPE_TURBO3_0 || layer_type_k == GGML_TYPE_TURBO4_0 || layer_type_k == GGML_TYPE_TURBO2_0);
+        if (k_is_turbo && n_embd_head_k % 128 != 0) {
+            const uint32_t padded_head_k = ((n_embd_head_k + 127) / 128) * 128;
+            const uint32_t n_head_kv = n_embd_k_gqa / n_embd_head_k;
+            n_embd_k_gqa_eff = n_head_kv * padded_head_k;
+            if (il == 0) {
+                LLAMA_LOG_INFO("%s: turbo zero-padding K head_dim %u -> %u (cache %u -> %u)\n",
+                               __func__, n_embd_head_k, padded_head_k, n_embd_k_gqa, n_embd_k_gqa_eff);
+            }
+        }
+
+        // For turbo types, pad V head_dim to next multiple of 128 if needed
+        const uint32_t n_embd_head_v = hparams.n_embd_head_v(il);
+        uint32_t n_embd_v_gqa_eff = n_embd_v_gqa;
+        const bool v_is_turbo = (layer_type_v == GGML_TYPE_TURBO3_0 || layer_type_v == GGML_TYPE_TURBO4_0 || layer_type_v == GGML_TYPE_TURBO2_0);
+        if (v_is_turbo && !is_mla && n_embd_head_v % 128 != 0) {
+            const uint32_t padded_head_v = ((n_embd_head_v + 127) / 128) * 128;
+            const uint32_t n_head_kv = n_embd_v_gqa / n_embd_head_v;
+            n_embd_v_gqa_eff = n_head_kv * padded_head_v;
+            if (il == 0) {
+                LLAMA_LOG_INFO("%s: turbo zero-padding V head_dim %u -> %u (cache %u -> %u)\n",
+                               __func__, n_embd_head_v, padded_head_v, n_embd_v_gqa, n_embd_v_gqa_eff);
+            }
+        }
+
+        ggml_tensor * k = has_k ? ggml_new_tensor_3d(ctx, layer_type_k, n_embd_k_gqa_eff, kv_size, n_stream) : nullptr;
+        ggml_tensor * v = has_v ? ggml_new_tensor_3d(ctx, layer_type_v, n_embd_v_gqa_eff, kv_size, n_stream) : nullptr;
 
         has_k && ggml_format_name(k, "cache_k_l%d", il);
         has_v && ggml_format_name(v, "cache_v_l%d", il);
Last 200 lines of the run log 
@@ -226,6 +226,43 @@ ggml_metal_library_t ggml_metal_library_init(ggml_metal_device_t dev) {
                 [prep setObject:@"1" forKey:@"GGML_METAL_EMBED_LIBRARY"];
 #endif
 
+                // TurboQuant: auto-select dequant path based on hardware
+                // M1/M2/M3/M4 (no tensor API): 4-mag LUT (+38-45% decode at long ctx)
+                // M5+ (has tensor API): 8-entry full LUT (best decode speed)
+                {
+                    const char * force_4mag = getenv("TURBO_FORCE_4MAG");
+                    // Always compile with 4-mag support. The dispatch code selects
+                    // 4-mag vs 8-LUT based on context depth at runtime.
+                    // Pre-M5: always 4-mag (constant cache too slow)
+                    // M5+: 4-mag for mid-context (8K-20K), 8-LUT otherwise
+                    if (!ggml_metal_device_get_props(dev)->has_tensor || (force_4mag && force_4mag[0] == '1')) {
+                        [prep setObject:@"1" forKey:@"TURBO_USE_4MAG"];
+                        GGML_LOG_INFO("%s: turbo3 using 4-mag LUT%s\n", __func__,
+                            force_4mag ? " (forced)" : " (pre-M5 hardware)");
+                    }
+                    // Sparse V dequant: skip V for negligible attention weights
+                    // Enabled by default on all Metal (validated: PPL identical, NIAH 9/9, 30+ testers)
+                    // Opt-out via TURBO_SPARSE_V=0
+                    const char * sparse_v_env = getenv("TURBO_SPARSE_V");
+                    const bool sparse_v_disabled = sparse_v_env && sparse_v_env[0] == '0';
+                    if (!sparse_v_disabled) {
+                        [prep setObject:@"1" forKey:@"TURBO_SPARSE_V"];
+                        GGML_LOG_INFO("%s: turbo3 sparse V dequant enabled (opt-out: TURBO_SPARSE_V=0)\n", __func__);
+                    }
+                    // TODO: context-adaptive dispatch — compile both 4-mag and 8-LUT
+                    // FA kernel instantiations, select based on ne11 (KV cache size)
+                    // at dispatch time in ggml_metal_op_flash_attn_ext()
+                }
+
+                // TurboQuant profiling: set TURBO_PROFILE_MODE env var (0-4)
+                {
+                    const char * pm = getenv("TURBO_PROFILE_MODE");
+                    if (pm && pm[0] >= '0' && pm[0] <= '4') {
+                        [prep setObject:[NSString stringWithUTF8String:pm] forKey:@"TURBO_PROFILE_MODE"];
+                        GGML_LOG_INFO("%s: TURBO_PROFILE_MODE=%s\n", __func__, pm);
+                    }
+                }
+
                 MTLCompileOptions * options = [MTLCompileOptions new];
                 options.preprocessorMacros = prep;
 
@@ -1385,6 +1438,8 @@ bool ggml_metal_device_supports_op(ggml_metal_device_t dev, const struct ggml_te
             return true;
         case GGML_OP_GATED_DELTA_NET:
             return has_simdgroup_reduction && op->src[2]->ne[0] % 32 == 0;
+        case GGML_OP_TURBO_WHT:
+            return op->src[0]->ne[0] % 128 == 0;
         case GGML_OP_SOLVE_TRI:
         case GGML_OP_MUL_MAT:
         case GGML_OP_MUL_MAT_ID:

--- ggml/src/ggml-metal/ggml-metal-ops.cpp.rej
diff a/ggml/src/ggml-metal/ggml-metal-ops.cpp b/ggml/src/ggml-metal/ggml-metal-ops.cpp	(rejected hunks)
@@ -2590,12 +2658,13 @@ int ggml_metal_op_mul_mat_id(ggml_metal_op_t ctx, int idx) {
 
     const uint32_t r2 = 1;
     const uint32_t r3 = 1;
+    const bool is_tq_weight = (op->src[0]->type == GGML_TYPE_TQ3_1S || op->src[0]->type == GGML_TYPE_TQ4_1S);
 
     // find the break-even point where the matrix-matrix kernel becomes more efficient compared
     // to the matrix-vector kernel
     // ne20 = n_used_experts
     // ne21 = n_rows (batch size)
-    const int ne21_mm_id_min = 32;
+    const int ne21_mm_id_min = is_tq_weight ? 1 : 32;
 
     if (props_dev->has_simdgroup_mm && ne00 >= 64 && (ne21 >= ne21_mm_id_min)) {
         // some Metal matrix data types require aligned pointers
@@ -3410,7 +3563,8 @@ int ggml_metal_op_flash_attn_ext(ggml_metal_op_t ctx, int idx) {
         // ne20*(nsg)
         // each simdgroup has a full f32 head vector in shared mem to accumulate results
         //
-#define FATTN_SMEM(nsg) (GGML_PAD(((GGML_PAD(ne00, 128) + 4*ncpsg + 2*GGML_PAD(ne20, 128))*(nsg))*(sizeof(float)/2), 16))
+// Extra 128 floats (512 bytes) for TurboQuant pre-dequantized block cache in threadgroup memory
+#define FATTN_SMEM(nsg) (GGML_PAD(((GGML_PAD(ne00, 128) + 4*ncpsg + 2*GGML_PAD(ne20, 128))*(nsg))*(sizeof(float)/2) + 128*sizeof(float), 16))
 
         int64_t nsg = 1;
 

--- src/llama-kv-cache.cpp.rej
diff a/src/llama-kv-cache.cpp b/src/llama-kv-cache.cpp	(rejected hunks)
@@ -225,11 +265,107 @@ llama_kv_cache::llama_kv_cache(
             throw std::runtime_error("failed to create ggml context for kv cache");
         }
 
+        // TurboQuant zero-padding: for models with non-128-aligned head_dim (e.g. DeepSeek
+        // head_dim_k=192), pad each head to the next multiple of 128. The padded zeros don't
+        // affect dot products since WHT preserves inner products:
+        //   <WHT(Q_padded), WHT(K_padded)> = <Q_padded, K_padded> = <Q, K> + <0, 0> = <Q, K>
+        const uint32_t n_embd_head_k = hparams.n_embd_head_k(il);
+
+
         const bool has_k = true;
         const bool has_v = !is_mla;
 
-        ggml_tensor * k = has_k ? ggml_new_tensor_3d(ctx, type_k, n_embd_k_gqa, kv_size, n_stream) : nullptr;
-        ggml_tensor * v = has_v ? ggml_new_tensor_3d(ctx, type_v, n_embd_v_gqa, kv_size, n_stream) : nullptr;
+        // Layer-adaptive: use higher precision for quality-sensitive layers
+        // Config: TURBO_LAYER_ADAPTIVE env var controls the strategy
+        //   0 = uniform (default)
+        //   1 = q8_0 K+V for first+last 4 layers
+        //   2 = q8_0 K+V for last 8 layers
+        //   5 = Boundary V: first2+last2 V=turbo4, rest V=turbo2 (K unchanged)
+        //   6 = V-only: last 8 V=turbo4, rest V=turbo2 (K unchanged)
+        //   7 = Boundary V (recommended): first2+last2 V=q8_0, rest V=turbo2 (K unchanged)
+        ggml_type layer_type_k = type_k;
+        ggml_type layer_type_v = type_v;
+        {
+            static const int adaptive_mode = [&]() {
+                const char * env = getenv("TURBO_LAYER_ADAPTIVE");
+                if (env) {
+                    int mode = atoi(env);
+                    if (mode > 0) {
+                        LLAMA_LOG_INFO("llama_kv_cache: layer-adaptive mode %d enabled (env)\n", mode);
+                    }
+                    return mode;
+                }
+                // Auto-enable Boundary V (mode 7) when V is turbo2
+                if (type_v == GGML_TYPE_TURBO2_0 && hparams.n_layer >= 8) {
+                    LLAMA_LOG_INFO("llama_kv_cache: Boundary V auto-enabled for turbo2-V (opt-out: TURBO_LAYER_ADAPTIVE=0)\n");
+                    return 7;
+                }
+                return 0;
+            }();
+            const bool is_turbo = (type_k == GGML_TYPE_TURBO3_0 || type_k == GGML_TYPE_TURBO4_0 || type_k == GGML_TYPE_TURBO2_0);
+            const bool v_is_turbo = (type_v == GGML_TYPE_TURBO3_0 || type_v == GGML_TYPE_TURBO4_0 || type_v == GGML_TYPE_TURBO2_0);
+            const uint32_t n_layer = hparams.n_layer;
+            if (adaptive_mode == 1 && is_turbo && n_layer >= 8) {
+                if (il < 4 || il >= n_layer - 4) {
+                    layer_type_k = GGML_TYPE_Q8_0;
+                    layer_type_v = GGML_TYPE_Q8_0;
+                }
+            } else if (adaptive_mode == 2 && is_turbo && n_layer >= 8) {
+                if (il >= n_layer - 8) {
+                    layer_type_k = GGML_TYPE_Q8_0;
+                    layer_type_v = GGML_TYPE_Q8_0;
+                }
+            } else if (adaptive_mode == 5 && v_is_turbo && n_layer >= 8) {
+                // Boundary V (turbo4 boundaries): first2+last2 V=turbo4, rest V=turbo2
+                const bool is_boundary = (il < 2 || il >= n_layer - 2);
+                layer_type_v = is_boundary ? GGML_TYPE_TURBO4_0 : GGML_TYPE_TURBO2_0;
+                if (il == 0) {
+                    LLAMA_LOG_INFO("llama_kv_cache: Boundary V mode 5: first2+last2 V=turbo4, rest V=turbo2\n");
+                }
+            } else if (adaptive_mode == 6 && v_is_turbo && n_layer >= 8) {
+                // V-only: last 8 V=turbo4, rest V=turbo2
+                layer_type_v = (il >= n_layer - 8) ? GGML_TYPE_TURBO4_0 : GGML_TYPE_TURBO2_0;
+                if (il == 0) {
+                    LLAMA_LOG_INFO("llama_kv_cache: V-only LA mode 6: last8 V=turbo4, rest V=turbo2\n");
+                }
+            } else if (adaptive_mode == 7 && v_is_turbo && n_layer >= 8) {
+                // Boundary V (recommended): first2+last2 V=q8_0, rest V=turbo2
+                const bool is_boundary = (il < 2 || il >= n_layer - 2);
+                layer_type_v = is_boundary ? GGML_TYPE_Q8_0 : GGML_TYPE_TURBO2_0;
+                if (il == 0) {
+                    LLAMA_LOG_INFO("llama_kv_cache: Boundary V mode 7: first2+last2 V=q8_0, rest V=turbo2\n");
+                }
+            }
+        }
+        // For turbo types, pad K head_dim to next multiple of 128 for full WHT groups
+        uint32_t n_embd_k_gqa_eff = n_embd_k_gqa;
+        const bool k_is_turbo = (layer_type_k == GGML_TYPE_TURBO3_0 || layer_type_k == GGML_TYPE_TURBO4_0 || layer_type_k == GGML_TYPE_TURBO2_0);
+        if (k_is_turbo && n_embd_head_k % 128 != 0) {
+            const uint32_t padded_head_k = ((n_embd_head_k + 127) / 128) * 128;
+            const uint32_t n_head_kv = n_embd_k_gqa / n_embd_head_k;
+            n_embd_k_gqa_eff = n_head_kv * padded_head_k;
+            if (il == 0) {
+                LLAMA_LOG_INFO("%s: turbo zero-padding K head_dim %u -> %u (cache %u -> %u)\n",
+                               __func__, n_embd_head_k, padded_head_k, n_embd_k_gqa, n_embd_k_gqa_eff);
+            }
+        }
+
+        // For turbo types, pad V head_dim to next multiple of 128 if needed
+        const uint32_t n_embd_head_v = hparams.n_embd_head_v(il);
+        uint32_t n_embd_v_gqa_eff = n_embd_v_gqa;
+        const bool v_is_turbo = (layer_type_v == GGML_TYPE_TURBO3_0 || layer_type_v == GGML_TYPE_TURBO4_0 || layer_type_v == GGML_TYPE_TURBO2_0);
+        if (v_is_turbo && !is_mla && n_embd_head_v % 128 != 0) {
+            const uint32_t padded_head_v = ((n_embd_head_v + 127) / 128) * 128;
+            const uint32_t n_head_kv = n_embd_v_gqa / n_embd_head_v;
+            n_embd_v_gqa_eff = n_head_kv * padded_head_v;
+            if (il == 0) {
+                LLAMA_LOG_INFO("%s: turbo zero-padding V head_dim %u -> %u (cache %u -> %u)\n",
+                               __func__, n_embd_head_v, padded_head_v, n_embd_v_gqa, n_embd_v_gqa_eff);
+            }
+        }
+
+        ggml_tensor * k = has_k ? ggml_new_tensor_3d(ctx, layer_type_k, n_embd_k_gqa_eff, kv_size, n_stream) : nullptr;
+        ggml_tensor * v = has_v ? ggml_new_tensor_3d(ctx, layer_type_v, n_embd_v_gqa_eff, kv_size, n_stream) : nullptr;
 
         has_k && ggml_format_name(k, "cache_k_l%d", il);
         has_v && ggml_format_name(v, "cache_v_l%d", il);

Patch 0001-turboquant.patch no longer applies to upstream 010be9683afabe14ce299197b38c329f94bae568.

This is not a bug in the automation. Upstream changed the code this patch
touches, so the patch needs rebasing. See docs/upstream-fork.md
("Rebasing a patch"). Nothing was built and nothing was synced.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    upstream-bumpAutomated llama.cpp upstream replay

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions