fix(mcp): page the boot-time tool load instead of holding two copies - #537
Merged
Conversation
The cloud instance crash-looped nine times on 10 Sep between 05:51 and 06:13 UTC, each time with "FATAL ERROR: Ineffective mark-compacts near heap limit", taking sign-in, sign-up and every tenant's MCP endpoint down with it. loadAllTools pulled every active connector with all of its enabled tools in one findMany. On cloud that is 578 connectors and 22,215 tools — about 118 MB of raw JSON before V8 object overhead — and the whole result set stayed live while the registry it fed was being built, so peak heap at boot was roughly twice the steady state. Steady state alone is ~1.6 GB against a 2048 MB old space, which left nothing for the transient copy. Read a page of connectors at a time so the raw rows for a page are collectable once registered. The registry itself is unchanged and still holds every tenant's tools; this only removes the duplicate. The registry growing linearly with tenants is the real ceiling and wants a bounded, per-organization cache — that is a larger change and is not attempted here. While extracting the per-connector registration, reloadConnectorTools turns out to have been a verbatim copy of the same 45 lines. Both now call registerConnectorTools, so they cannot drift.
start.sh already honours NODE_MAX_OLD_SPACE_MB, but the cloud compose never passed it through, so the only way to give the backend more heap than the 2048 MB default was to rebuild the image. Plumb it, defaulting to the same 2048 so nothing changes for anyone who does not set it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
The cloud instance crash-looped nine times on 10 Sep between 05:51 and 06:13 UTC:
Each crash took sign-in, sign-up and every tenant's MCP endpoint down. It has been up since, but at 1.74 GB against a 2048 MB old space — ~300 MB of headroom — so it will happen again.
Why
loadAllTools()pulled every active connector with all of its enabled tools in a singlefindMany. On the cloud instance that is:The whole result set stays live while the registry it feeds is being built, so peak heap at boot is roughly twice the steady state. Steady state alone is ~1.6 GB, which leaves nothing for the transient copy.
What this does
Reads a page of connectors at a time (25) so a page's raw rows become collectable as soon as they're registered. The registry is unchanged and still holds every tenant's tools — this only removes the duplicate.
Tests cover the paging loop directly, since cursor pagination is easy to get subtly wrong: every connector visited exactly once and in order, never more than one page in flight, termination on an exact multiple of the page size (the classic infinite-loop case), and the empty-table case.
Extracting the per-connector registration also revealed that
reloadConnectorToolswas a verbatim copy of the same 45 lines. Both now callregisterConnectorTools, so they can't drift.What this does NOT fix
The registry grows linearly with tenants and is never evicted. Every tool of every organization is held in
ToolRegistryfor the process lifetime, including each connector's decryptedauthConfig. Halving the boot peak buys headroom; it doesn't move the ceiling. The real fix is a bounded, per-organization cache — a much larger change toToolRegistry,getToolForOrg,countByNameand the upstream single-tenant mcp-nest registry, and not something to land as a hotfix.Until that exists the cloud droplet (4 GB total, 2.7 GB in use, swapping) needs more RAM and a larger
--max-old-space-size.Full suite green: 3973 passed, 235 suites.