Making KV Cache a Dynamic Short-Term Memory for LLMs
DMKV (Dynamic Memory KV Cache) is a research and implementation project that rethinks the KV Cache of large language models (LLMs) as a dynamic short-term memory managed by the AI itself, rather than merely a cache for reusing past computations.
Modern LLMs can process increasingly long contexts, but longer context does not necessarily mean better memory utilization.
In particular, information placed near the middle of a long context can be less effectively used than information placed near the beginning or end. This phenomenon is commonly known as Lost in the Middle.
DMKV approaches this problem from a different perspective:
What if an LLM treated its context as a memory with different levels of importance, instead of treating all past information as an equally ordered sequence?
In a conventional Transformer, past information is primarily handled as:
Token
↓
Key / Value
↓
KV Cache
↓
Attention
DMKV extends this concept into:
Information
↓
Importance
↓
Memory Policy
↓
Dynamic KV Cache
↓
Attention
↑
│
Re-evaluation
The core idea is simple:
Turn KV Cache from a passive cache into a dynamic short-term memory.
DMKV does not assume that all information in a context should be treated equally.
Each memory unit can be associated with properties such as:
Memory A → Importance 0.95
Memory B → Importance 0.21
Memory C → Importance 0.73
Based on these values, the memory system can decide:
- where the information should be placed,
- how precisely it should be retained,
- when its importance should be reconsidered,
- and which information should be discarded when memory is limited.
The first DMKV experiment focuses on changing the logical position of memories according to their importance.
An initial policy is:
High Importance
→ Beginning / End of Context
Medium Importance
→ Intermediate Area
Low Importance
→ Central Area
Conceptually:
┌──────────────────────────────────────┐
│ HIGH │ HIGH │ MID │ LOW │ LOW │ MID │ HIGH │ HIGH │
└──────────────────────────────────────┘
↑ ↑
important important
However, DMKV does not assume that putting important information at the edges is always optimal.
This is only the first Position Policy to be tested.
The broader research question is:
What is the optimal relationship between memory importance and logical position within an LLM context?
One of the most important research questions in DMKV is:
How should an AI decide what is important?
Importance should not necessarily be a fixed number.
Instead, it may depend on the current reasoning state:
Importance
=
f(
Content,
Relevance,
Recency,
Current Goal,
Future Utility
)
For example, the importance of the same memory may change during an interaction:
t1
A = 0.90
B = 0.20
Later:
t2
A = 0.40
B = 0.95
This means short-term memory should be treated as a dynamic state that can be re-evaluated during inference, rather than as a fixed transcript.
A key design principle of DMKV is to separate the logical order of memory from its physical storage location.
For example:
Logical Order
A → D → G → B → C → E → F
while the physical storage may remain:
Physical Storage
Page 01 → B
Page 07 → G
Page 13 → A
Page 25 → F
Page 31 → D
Page 42 → C
Page 51 → E
The goal is to avoid physically copying and moving KV data whenever possible.
Instead, logical ordering may be changed through:
- mappings,
- indices,
- page tables,
- or similar memory-management mechanisms.
This design makes DMKV potentially compatible with paged KV-cache systems.
DMKV does not organize memory once and leave it unchanged.
When new information arrives, the system may re-evaluate existing memories:
New Input
↓
Importance Re-evaluation
↓
Position Recalculation
↓
Memory Reorganization
↓
Attention
A memory that was previously unimportant may become highly relevant later.
Likewise, information that was previously important may gradually become less useful.
This makes DMKV a dynamic memory system, rather than a static context reordering mechanism.
A future extension of DMKV is to control not only where information is stored, but also how much information is retained.
For example:
High Importance
→ Full KV
Medium Importance
→ Compressed KV
Low Importance
→ Strong Compression
This introduces an additional dimension to memory management:
Important memories should be retained with higher resolution, while less important memories can be represented more compactly.
The goal is to use limited VRAM more intelligently instead of simply increasing the context length.
Ultimately, DMKV aims to allow the AI to control multiple aspects of its own short-term memory:
Importance
Position
Compression
Eviction
A conceptual architecture is:
Current State
↓
Importance Model
↓
Memory Controller
↓
┌────────────┼────────────┐
↓ ↓ ↓
Position Compression Eviction
Policy Policy Policy
└────────────┼────────────┘
↓
Dynamic KV Cache
↓
Attention
↓
New State
│
└────→ Re-evaluation
At this stage, the KV Cache becomes more than a performance optimization.
It becomes a short-term memory architecture for the LLM.
DMKV focuses on the following questions.
Can importance-aware memory placement reduce the degradation associated with Lost in the Middle?
Can AI-based importance estimation outperform fixed or manually assigned importance values?
Can dynamic memory reorganization outperform static context ordering?
Can importance-aware memory resolution reduce KV Cache requirements while maintaining model performance?
Can these mechanisms be integrated directly into KV Cache management and eventually into the internal architecture of Transformer models?
DMKV does not require training a new LLM at the beginning.
The first stage can be implemented as an external memory scheduler around an existing open-weight model:
Conversation / Documents
↓
DMKV Layer
↓
Reconstructed Context
↓
Existing LLM
The initial experiment compares:
A. Original Context
B. Random Context
C. Importance-Aware Context
The first goal is intentionally simple:
Does changing memory placement alone measurably improve the model's ability to use information from long contexts?
DMKV evaluates both model performance and memory efficiency.
Main metrics include:
Accuracy
Context Length
KV Cache Size
VRAM Usage
Latency
Throughput
Reorganization Cost
Information Retention
Two particularly important metrics are:
Performance / KV Memory
and:
Performance / Context Token
The objective is not simply to maximize context length.
The objective is to use available memory more intelligently.
Phase 0
External Memory Scheduler
↓
Importance + Position Reordering
Phase 1
AI-based Importance Evaluation
↓
Dynamic Importance
Phase 2
Dynamic Memory Reorganization
↓
Continuous Re-evaluation
Phase 3
Native KV Cache Integration
↓
Dynamic KV Memory
Phase 4
Compression + Eviction
↓
Memory-efficient LLM
Treat KV Cache as short-term memory rather than only as a computation cache.
Not all memories should be treated as equally valuable.
Memory importance may change as the reasoning state changes.
Logical memory order should be independent from physical memory layout whenever possible.
Position, compression, and eviction policies should remain modular rather than being hard-coded into the architecture.
The first stage should validate the hypothesis without modifying the underlying LLM.
Long-context LLM research often asks:
How can we make the context longer?
DMKV asks a different question:
How can we make the context behave more like a memory?
Instead of treating context as a uniform token sequence, DMKV treats it as:
Token Sequence
↓
Weighted Memory
↓
Dynamic Memory
The goal is to move from:
more context
to:
better-managed memory.
Research / Experimental
The current objective is to validate the smallest DMKV hypothesis using existing open-weight LLMs.
Initial target:
Importance
↓
Position Reordering
↓
LLM
If the hypothesis is supported, the project will progressively investigate:
- AI-based importance estimation
- dynamic reorganization
- memory compression
- memory eviction
- native KV Cache integration
- model-level short-term memory architectures
DMKV is released under the MIT License.
Copyright (c) 2026 Yoshiharu Uematsu
DMKV (Dynamic Memory KV Cache) is a dynamically managed short-term memory architecture that allows an AI to evaluate, reorganize, and retain information in KV Cache according to its importance.
In simpler terms:
DMKV turns KV Cache into a dynamic short-term memory for LLMs.