OpenAI-compatible Chat Completions: top-level reasoning_effort ignored, only chat_template_kwargs controls reasoning

[teichertw]

Summary

When using the OpenAI-compatible /v3/chat/completions endpoint with Qwen3 reasoning models, OVMS ignores the standard top-level reasoning_effort field and only honors reasoning controls passed through chat_template_kwargs.

This creates interoperability issues with OpenAI-compatible clients that correctly send reasoning_effort but have no knowledge of OVMS-specific chat-template parameters.

Additionally, with --reasoning_parser qwen3, reasoning still appears inside message.content instead of being returned separately as reasoning_content.

I'm not sure whether both behaviors are intentional, so this issue is primarily a request for clarification.

---

Environment

• OVMS 2026.2.1 (Windows binary)
• REST API (/v3/chat/completions)
• Windows 11
• GPU target
• Model: OpenVINO/Qwen3-8B-int4-ov
• --tool_parser hermes3
• --reasoning_parser qwen3

---

What I verified

Using direct curl requests against OVMS:

Request	Result
reasoning_effort: "none"	❌ ignored
reasoning_effort: "high"	❌ ignored
chat_template_kwargs.enable_thinking=false	✅ works
chat_template_kwargs.enable_thinking=true	✅ works
chat_template_kwargs.reasoning_effort="low"	✅ works

This confirms that reasoning is currently controlled exclusively through:

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "reasoning_effort": "low"
  }
}

while the OpenAI-style top-level

{
  "reasoning_effort": "low"
}

appears to have no effect.

---

Interoperability concern

OVMS advertises an OpenAI-compatible Chat Completions API.

Many OpenAI-compatible clients expose reasoning through the standard reasoning_effort field.

Some OpenAI-compatible inference servers translate this internally into the chat-template parameters required by the underlying model.

OVMS currently appears to require every client to explicitly know about chat_template_kwargs, reducing interoperability despite exposing an OpenAI-compatible endpoint.

If this behavior is intentional, documenting it explicitly would help client implementations avoid silent incompatibilities.

---

Questions

1. Is ignoring top-level reasoning_effort intentional?

If yes, I'll report this as an OVMS-specific capability that client implementations need to support.

If no, would OVMS consider automatically mapping

{
  "reasoning_effort": "low"
}

into

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "reasoning_effort": "low"
  }
}

for reasoning-capable models?

This would improve interoperability with existing OpenAI-compatible clients while remaining backward compatible.

---

2. What is the intended behavior of --reasoning_parser qwen3?

Even with the parser enabled, reasoning remains inside

choices[0].message.content

instead of being emitted separately as

choices[0].message.reasoning_content

Is this expected for Qwen3 models?

Or does it indicate a parser/template mismatch?

---

Why this matters

This is not specific to one client implementation.

Any OpenAI-compatible client that only knows the standard reasoning_effort field currently cannot dynamically control thinking on OVMS.

If appropriate, a server-side mapping would improve interoperability without requiring every client to implement OVMS-specific request rewriting.

---

Related

Client-side discussion:

NousResearch/hermes-agent#99820

That issue discusses how Hermes Agent could work around the current OVMS behavior.

This issue asks whether the current OVMS behavior is intentional and whether improving compatibility at the server level would be desirable.

[mzegla]

Correct, OVMS does not support reasoning_effort on the top request level as described in OpenAI API spec which is a gap in support.
I suppose we could plan adding it with some sort of mapping of reasoning_effort value to particular models needs.

As for the reasoning parser not doing the work for qwen3, could you run problematic request in OVMS launched with --log_level TRACE and provide server logs from this run? It would help us establish what went wrong. We've had qwen3 reasoning parser for a while now and did not have issues with it.

[dtrawins]

@teichertw The reasoning parser is expected to retrieve model reasoning phase and place it in reasoning_content field in the response. If that is not working, TRACE logs would be helpful to determine the root cause. It should be potentially wrong model raw response or some unknown defect in the parser. Could you capture the logs with --log_level TRACE including the server initialization part and request processing?
Model OpenVINO/Qwen3-8B-int4-ov should use --tool_parser hermes3 and --reasoning_parser qwen3. Not that starting from version 2026.3, tool parsers are detected automatically based on the model chat template.

Handling reasoning_effort field is separate issues. Indeed currently reasoning configuration can be done only via chat_template_kwargs. It can be added in near future. We have a plan to include automatic detection of the reasoning tuning parameters and automatically map it to chat_template_kwargs. That should simplify client configuration.

[teichertw]

Thanks for the replies, @mzegla and @dtrawins.

I ran the tests with --log_level TRACE as requested (server startup + request processing included), on OVMS 2026.2.1 (Windows binary) with OpenVINO/Qwen3-8B-int4-ov, --tool_parser hermes3, --reasoning_parser qwen3. The trace excerpt (reduced, without hardware/device details) is embedded below.

Findings:

1. The reasoning parser works correctly — when generation passes the thinking phase, reasoning_content and content are properly separated (test 3).
2. The problem reproduces only when generation ends inside the thinking phase (finish_reason: length with a small max_tokens budget). Qwen3-8B spends a lot of tokens on thinking; with max_tokens=200 the model never reaches the response phase. In that case OVMS returns the raw thinking segment inside message.content — including the visible thinking marker — and no reasoning_content field at all.

Test	Config	Result
1	enable_thinking=true, max_tokens=200	content = " thinking\nOkay, let's see..." — no reasoning_content, finish_reason=length
2	enable_thinking=true, reasoning_effort=low, max_tokens=200	same
3	enable_thinking=true, max_tokens=1024	reasoning_content + content correctly separated

Conclusion: This is not a parser defect but a behavior question: when generation is truncated (finish_reason=length) while still in the thinking phase, OVMS emits the raw thinking text (with the thinking marker) into message.content instead of reasoning_content.

Would it be possible for OVMS to classify the thinking segment as reasoning_content even when the response phase is never reached? That would make truncated generations much safer for OpenAI-compatible clients, which currently receive thinking as regular content and may display it to end users.

If additional targeted tests are useful, I can run them on request.

# ============================================================
# OVMS TRACE-Log (bereinigt) — Issue #4495
# OVMS 2026.2.1, Modell OpenVINO/Qwen3-8B-int4-ov
# Ziel: Untersuchung reasoning_parser qwen3 + Tool-Parser hermes3
# Hinweis: Hardware-/Pfad-Details wurden entfernt.
# ============================================================

# -------- SERVER-INIT (gekürzt) --------
[2026-09-01 23:55:32.069][26884][serving][info][server.cpp:117] OpenVINO Model Server 2026.2.1.1122f03bf
[2026-09-01 23:55:32.069][26884][serving][info][server.cpp:118] OpenVINO backend 2026.2.1-21919-ede283a88e3-releases/2026/2
[2026-09-01 23:55:32.069][26884][serving][info][server.cpp:121] OpenVINO GenAI backend 2026.2.1.0-3123-7dea0459b2a
[2026-09-01 23:55:32.069][26884][serving][debug][server.cpp:131] model_path: models\OpenVINO\Qwen3-8B-int4-ov
[2026-09-01 23:55:32.069][26884][serving][debug][server.cpp:132] model_name: OpenVINO/Qwen3-8B-int4-ov
[2026-09-01 23:55:32.070][26884][serving][debug][server.cpp:137] target_device: GPU
[2026-09-01 23:55:32.070][26884][serving][debug][server.cpp:149] REST port: 8000
[2026-09-01 23:55:32.070][26884][serving][debug][server.cpp:155] log level: TRACE
[2026-09-01 23:55:32.070][26884][serving][debug][server.cpp:160] model_repository_path: models
[... Init-Abschnitte für Python/MediaPipe/Device-Detection entfernt ...]
[2026-09-01 23:55:43.640][26884][modelmanager][info][mediapipegraphdefinition.cpp:261] Mediapipe: OpenVINO/Qwen3-8B-int4-ov inputs:
[2026-09-01 23:55:43.640][26884][modelmanager][info][mediapipegraphdefinition.cpp:263] Mediapipe: OpenVINO/Qwen3-8B-int4-ov kfs pass through: false
[2026-09-01 23:55:43.640][26884][modelmanager][info][pipelinedefinitionstatus.hpp:60] Mediapipe: OpenVINO/Qwen3-8B-int4-ov state changed to: AVAILABLE

# ============================================================
# TEST 1 — enable_thinking=true, max_tokens=200
# ============================================================
[2026-09-01 23:57:02.943][31280][serving][debug][http_server.cpp:174] REST request /v3/chat/completions
[2026-09-01 23:57:02.946][27948][llm_calculator][debug][servable.cpp:370] Request body: {"model":"OpenVINO/Qwen3-8B-int4-ov","messages":[{"role":"user","content":"..."}],"max_tokens":200,"chat_template_kwargs":{"enable_thinking":true}}
[2026-09-01 23:57:14.880][27948][llm_calculator][trace][output_parser.cpp:268] Raw model output:  thinking
[2026-09-01 23:57:14.881][27948][llm_calculator][debug][servable.cpp:298] Complete unary response: {"choices":[{"finish_reason":"length","index":0,"logprobs":null,"message":{"content":" thinking\nOkay, let's see...","role":"assistant","tool_calls":[]}}],...}
# -> KEIN reasoning_content-Feld; roher Thinking-Marker; finish_reason=length

# ============================================================
# TEST 2 — enable_thinking=true + reasoning_effort=low, max_tokens=200
# ============================================================
[2026-09-01 23:57:15.125][11328][llm_calculator][debug][servable.cpp:370] Request body: {"model":"OpenVINO/Qwen3-8B-int4-ov","messages":[{"role":"user","content":"..."}],"max_tokens":200,"chat_template_kwargs":{"enable_thinking":true,"reasoning_effort":"low"}}
[2026-09-01 23:57:27.506][11328][llm_calculator][trace][output_parser.cpp:268] Raw model output:  thinking
# -> ebenfalls kein reasoning_content, finish_reason=length

# ============================================================
# TEST 3 — enable_thinking=true, max_tokens=1024 (ausreichendes Budget)
# ============================================================
[2026-09-01 23:59:33.733][18552][llm_calculator][trace][output_parser.cpp:268] Raw model output:  thinking
[2026-09-01 23:59:33.734][18552][llm_calculator][debug][servable.cpp:298] Complete unary response: {"choices":[{"finish_reason":"length","index":0,"logprobs":null,"message":{"content":"\n\nUm 29 * 17 zu berechnen...","reasoning_content":"\nOkay, let's see. I need to calculate 29 multiplied by 17...","role":"assistant","tool_calls":[]}}],...}
# -> Parser separiert: reasoning_content + content getrennt!