fix(batch): deepseek-v4 serves concurrent requests (one-row prompt batches, CacheList promotion) - #38
Merged
Merged
Conversation
to_batch_cache has no PoolingCache arm, so any multi-row prompt batch on a pooling-cache model raises before prefill and every request in it fails. Concurrency above 2 was unserveable on deepseek-v4 as a result. Cap prompt batches at one row for these models. Upstream's single-row fast path builds the model's own scalar caches and never reaches the conversion, and the decode side already joins those: _extend_cache promotes a cache carrying merge with no left_padding, which is exactly PoolingCache, and BatchPoolingCache implements extend/filter/extract. The result is serial prefill with batched decode, which is where batching pays; B>1 prefill is not a throughput win at depth. Same lever the ckpt tier pulls in spec_engine, for a different reason. Kill switch GMLX_POOLED_PREFILL_B1=0.
_extend_cache lifts a scalar cache into a batch one when it carries a merge and no left_padding. A CacheList has neither of its own, so a decode batch built from CacheLists (deepseek-v4: rotating window plus two pools per layer) looked scalar on every admission and was merged a second time. The second merge reached BatchRotatingKVCache.merge, which calls c._temporal_order(c.keys) against rows that are already batch caches, and every request in flight died with a TypeError. Own the promotion test and look through CacheList members. Also give BatchPoolingCache the left_padding attribute the other batch caches carry (always zero; the constructor rejects anything else), so the test reads the same way for pooled rows. With this and the B=1 prompt-batch gate, deepseek-v4 serves concurrent requests: c=1..4 all complete, realized decode width 3.48 of 4.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.