fix(agentmemory): raise memory limit to 8Gi as staircase measurement - #1489
Merged
Conversation
mem::consolidate-pipeline retains ~1130 MiB every 120-minute tick (~960 MiB in glibc per-thread malloc arenas) and never gives it back. The 4Gi limit was sized from an incomplete model; a single tick from a clean baseline already peaks at 3871 MiB, and later ticks start ~1100 MiB higher. 8Gi (request 4Gi, same 1:2 ratio) is headroom to measure whether the staircase converges, not a declared fix. Leave MALLOC_ARENA_MAX, AUTO_FORGET_INTERVAL_MS, the image, and the chart version unchanged so this result stays interpretable. Live workload verified 2026-08-29: ghcr.io/joryirving/agentmemory 0.9.29@sha256:74f1175c..., iii 0.11.2, request 2Gi / limit 4Gi, AUTO_FORGET_INTERVAL_MS=86400000, 14 restarts, last OOMKilled at 2026-08-29T10:18:35Z. Upstream (report in PR, not applied): - agentmemory latest tagged/npm release is still 0.9.29. main is one commit ahead (portable CLI install paths, #892), not a memory fix. - iii 0.22.1 is current stable (0.23.0-rc.8 is a prerelease) but agentmemory still pins iii 0.11.2; 0.11.6+ is an incompatible architecture. No changelog/issue/commit on either project describes a fix for per-tick arena retention. Open issue #1153 is a related but different OOM (61s unbounded spike). - Renovate tracks the joryirving docker tag, not iii. ignoreUnstable default would skip 0.23.0-rc.* the same way other prereleases are skipped. A future 0.9.30 would be proposed (patch automerge); a 0.10.0 would wait 3 days then automerge as minor. Watch container_memory_usage_bytes after merge, not working_set (the ~570 MiB gap is the reserve the kernel spends before it kills).
Contributor
--- kubernetes/apps/base/ai/agentmemory/app Kustomization: ai/agentmemory HelmRelease: ai/agentmemory
+++ kubernetes/apps/base/ai/agentmemory/app Kustomization: ai/agentmemory HelmRelease: ai/agentmemory
@@ -111,16 +111,16 @@
httpGet:
path: /agentmemory/livez
port: 3111
periodSeconds: 5
resources:
limits:
- memory: 4Gi
+ memory: 8Gi
requests:
cpu: 100m
- memory: 2Gi
+ memory: 4Gi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
replicas: 1
|
Contributor
--- HelmRelease: ai/agentmemory Deployment: ai/agentmemory
+++ HelmRelease: ai/agentmemory Deployment: ai/agentmemory
@@ -125,16 +125,16 @@
port: 3111
initialDelaySeconds: 0
periodSeconds: 30
timeoutSeconds: 5
resources:
limits:
- memory: 4Gi
+ memory: 8Gi
requests:
cpu: 100m
- memory: 2Gi
+ memory: 4Gi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
startupProbe:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Raise the ai/agentmemory memory limit to 8Gi as a measurement of whether the mem::consolidate-pipeline staircase converges, and separately establish (report only, do not upgrade) whether a newer upstream release already fixes the underlying per-tick arena retention.
Why 8Gi and not 5Gi: a completed investigation measured that mem::consolidate-pipeline permanently retains roughly 1130 MiB every 120-minute tick and never gives it back. A single consolidation tick from the cleanest baseline took the cgroup 2123 -> 3871 MiB in about 4 seconds, leaving only 225 MiB of the current 4096 MiB unused. Later ticks start from a plateau roughly 1100 MiB higher. talos-2 never dropped below 57 GiB available. The captain chose 8Gi deliberately. A limit that lands just under the real peak is actively worse than the current state: it converts a clean 14-second restart into a 90-second liveness reclaim stall. Do not shave this number down. Keep the request-to-limit relationship sane and consistent with this repo (the previous pair was request 2Gi / limit 4Gi, 1:2; this change uses request 4Gi / limit 8Gi to keep that ratio).
This is a measurement, not a declared fix. Update the existing explanatory comment block in kubernetes/apps/base/ai/agentmemory/app/helmrelease.yaml so it tells the truth: the prior comment said "Not a leak" and described a 120-minute timer coincidence with a ~3588 MiB floor - that model is incomplete. Record per-tick retention of roughly 1130 MiB, of which about 960 MiB sits in glibc per-thread malloc arenas, so the floor rises every tick rather than once. Record that both earlier remedies worked and neither could fix this - de-harmonisation lowered the starting floor by ~560 MiB and raised mean container lifetime from 1h58m to 5h04m, measured across 27 container incarnations. Point at the 2026-08-29 recurrence investigation for the full evidence.
Explicitly do NOT: set MALLOC_ARENA_MAX (running the limit raise and the arena cap together makes neither result interpretable; the arena cap remains the next step if the high-water mark proves effectively unbounded); change consolidation or auto-forget intervals (AUTO_FORGET_INTERVAL_MS=86400000 stays); change the image or the chart version.
Second deliverable, reported in the PR description, not applied: verify we run agentmemory 0.9.29 (ghcr.io/joryirving/agentmemory) and iii 0.11.2 against the live workload; whether any newer release of either exists and whether its changelog, commits, or issues mention memory retention, arena growth, consolidation memory, or an equivalent fix; whether Renovate would ever propose that release or skip it the way prereleases are. Do not upgrade anything. If nothing newer addresses it, say that plainly.
Do not apply this change to the cluster by hand - it reaches the cluster through Flux when the PR merges. State in the PR description how to read the result afterwards: watch container_memory_usage_bytes, not working_set, because the gap here is about 570 MiB and that gap is exactly the reserve the kernel spends before it kills.
What Changed
ai/agentmemorycontainer memory from request2Gi/ limit4Gito request4Gi/ limit8Gi(same 1:2 ratio) inkubernetes/apps/base/ai/agentmemory/app/helmrelease.yamlas headroom to measure whether themem::consolidate-pipelineper-tick staircase converges.MALLOC_ARENA_MAXunset and does not change image, chart, orAUTO_FORGET_INTERVAL_MS.Upstream check (report only; nothing upgraded)
ghcr.io/joryirving/agentmemorytag0.9.29(digestsha256:74f1175c…); comments and wiring still describe iii 0.11.2 (agentmemorypackage.jsonpinsiii-sdk@0.11.2; changelog audit text matches).rohitg00/agentmemorylatest tag/release and npm@agentmemory/agentmemorylatest are still 0.9.29 (2026-08-16). No newer release exists to upgrade to. Changelog "memory leak" entries (e.g. 0.9.28#1057, 0.9.27#817) are cross-agent isolation / context filtering, not RSS, glibc arena growth, or consolidate-tick retention. Nothing in 0.9.27–0.9.29 addresses per-tick arena retention.iii-sdklatestis 0.22.1 (alsorc/nextlines past that). agentmemory has not moved off the 0.11.2 pin, so a newer iii is not part of any published agentmemory release that fixes this. No public iii changelog/issue evidence was found that an arena/consolidate retention fix already shipped in a release agentmemory consumes.ghcr.io/joryirving/agentmemorytag appears, Renovate can open a version PR; digest-only updates auto-merge. Unstable/prerelease tags are not treated like normal stable bumps (same class as other prerelease skips). Today there is simply no newer agentmemory tag for it to propose.How to read the result after merge
Watch
container_memory_usage_bytes, not working_set: the gap is ~570 MiB and is the reserve the kernel spends before it kills. Goal is whether the consolidate-tick staircase plateaus under 8Gi or keeps climbing toward the new limit.Risk Assessment
✅ Low: Single bounded measurement change: request/limit 4Gi/8Gi with an accurate comment rewrite, no forbidden knobs, image, or interval changes.
Testing
Parsed and kustomize-built the agentmemory HelmRelease to confirm the 4Gi request / 8Gi limit measurement (1:2 preserved), unchanged 0.9.29 image, unchanged AUTO_FORGET_INTERVAL_MS, and no MALLOC_ARENA_MAX, with the operator comment rewritten to the staircase model; independently re-verified npm/GitHub/GHCR/Renovate that nothing newer fixes per-tick arena retention. No kubeconfig here so live pod image was not re-probed (commit already records that check).
Evidence: Before/after resources and invariants transcript
Evidence: Semantic acceptance checks (PASS)
Evidence: kustomize-built agentmemory manifests showing 4Gi/8Gi
Evidence: Upstream report-only re-verification (npm/GitHub/GHCR/Renovate)
Evidence: GHCR joryirving/agentmemory tag list
Evidence: agentmemory v0.9.29 GitHub release body
Evidence: Rendered resources after change
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
git diff 83344f78..d78a2739 -- kubernetes/apps/base/ai/agentmemory/app/helmrelease.yaml(before/after resources + single-file scope)semantic YAML parse of HEAD HelmRelease values (4Gi/8Gi 1:2, image 0.9.29 digest, AUTO_FORGET_INTERVAL_MS=86400000, no MALLOC_ARENA_MAX, operator comment contract)baseline semantic parse of83344f78:.../helmrelease.yaml(2Gi/4Gi)mise exec -- kustomize build kubernetes/apps/base/ai/agentmemory/appthen semantic assert on emitted HelmReleasenpm registry:@agentmemory/agentmemory@0.9.29latest; depiii-sdk@0.11.2;iii-sdklatest stable0.22.1, prerelease head0.23.0-rc.8GitHubrohitg00/agentmemory: tags/releases,v0.9.29...HEADahead-by-1 (#892), issue #1153, release notes scan for arena/retention fixGHCRjoryirving/agentmemorytags/list digest match for 0.9.29Renovate.renovate/autoMerge.json5patch/minor docker rules for upgrade-proposal behaviour✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.