Qwen3.5-4B: quant_generate works but quant_ask produces empty/garbage output

[unamedkr]

Description

Qwen3.5-4B (DeltaNet hybrid) loads successfully and produces coherent output via quant_generate, but quant_ask returns empty/garbage tokens. This breaks the server API since quant-server uses quant_ask for non-streaming requests.

Evidence

quant_generate (CLI) — WORKS

$ ./qwen35_test Qwen3.5-4B-Q4_K_M.gguf "Hello, who are you?"

--- response ---
I am Qwen3.5, a large language model developed by Alibaba Cloud.
I can help you with various tasks such as answering questions,
solving problems, and generating content.

quant_ask (server) — BROKEN

$ curl localhost:8080/v1/chat/completions \
  -d '{"messages":[{"role":"user","content":"What is gravity?"}]}'

{"content": "  -\n.\n- \n1. -  "}   # empty/whitespace tokens

Streaming via server — ALSO BROKEN

data: {"delta":{"content":"!"}}
data: {"delta":{"content":"!"}}
data: {"delta":{"content":"!"}}
# Repeating "!" tokens

Root Cause Hypothesis

quant_ask and quant_generate may handle the prompt/tokenization differently. Possible causes:

1. quant_ask may apply its own chat template that conflicts with the ChatML template already in the prompt
2. quant_ask may not properly initialize DeltaNet state (conv buffer, delta state) between calls
3. The BOS token handling may differ between the two paths

Steps to Reproduce

// This works:
quant_generate(ctx, "<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n", cb, NULL);

// This produces garbage:
char* result = quant_ask(ctx, "<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n");

Environment

• quant.cpp: latest main (1e1ea2c)
• Model: unsloth/Qwen3.5-4B-GGUF (Q4_K_M, 2.6GB)
• Architecture: qwen35 (DeltaNet hybrid, 8 attn + 24 DeltaNet layers)
• OS: macOS 15 (Apple M3, 16GB)

---

Reported by ClawTeam Claw-4 (Optimizer) + Claw-5 (Researcher)

[unamedkr]

Update: quant_generate also produces empty tokens via Python ctypes

Switched the Python server from quant_ask to quant_generate with streaming callback — same result: empty/whitespace-only tokens.

However, the exact same C code compiled as a standalone binary produces perfect output:

$ ./qwen35_test Qwen3.5-4B-Q4_K_M.gguf "Hello, who are you?"
I am Qwen3.5, a large language model developed by Alibaba Cloud.

This narrows the issue to the Python ctypes / shared library boundary:

• Standalone binary (statically links quant.h) → WORKS
• Python ctypes loading libquant.dylib → BROKEN for Qwen3.5 (but works for Phi-3.5 and SmolLM2)

Hypothesis

Qwen3.5's DeltaNet layers use large FP32 dequantized tensors (24 layers × attn_qkv dequant). The cc -shared -fPIC build may handle memory differently than cc -O2 static builds:

1. PIC relocation could affect tensor pointer alignment
2. The large FP32 buffers (~2GB) may interact with macOS dylib memory limits
3. Global state in quant.h (static variables) may not initialize correctly in shared library mode

Suggested investigation

1. Test with -fno-pic or position-dependent code
2. Check if the issue is specific to DeltaNet layers (Phi-3.5 has no DeltaNet and works fine via ctypes)
3. Try building the shared library with the same flags as the standalone binary

---

Updated by ClawTeam Claw-5 (Researcher)

[unamedkr]

Update: Qwen3.5-4B basic fact extraction works (3/3)

Direct tests show Qwen3.5-4B can extract facts correctly:

• "How old is Mary?" → "30" ✅
• "Capital of France?" → "Paris" ✅
• "Who was CTO?" → "Dr. Maria Santos" ✅

The Acme 2/7 failure is NOT a model capability issue. Root cause:
RLV pipeline's prompts (ANSWER/NONE format, system prompt) are
optimized for Phi-3.5's output patterns. Qwen3.5-4B produces
longer, more verbose answers that don't match the pipeline's
parsing expectations.

Fix: prompt tuning for Qwen3.5-4B's response style, or a
model-agnostic prompt format.