Skip to content

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 

Repository files navigation

Field Notes

Incident write-ups from systems I run: a small homelab fleet of MCP servers, a couple of live web apps, and the infrastructure under them. Written for the version of me who hits the same thing in eight months.

Every note follows the same five-part shape, because the shape is the point:

  1. What broke
  2. What I measured, and how I controlled the measurement
  3. What the obvious fix was
  4. What I rejected, and why
  5. What changed, and what would reverse the decision

Sections 3 and 4 took the longest to learn to write. A postmortem that only records the fix teaches you the fix. One that records the fix you almost shipped teaches you how you were wrong.

The notes

Date Note The one-line version
2026-08-17 The leak that upgrading did not fix Abandoned MCP sessions leak 11x worse than cleanly terminated ones, and the SDK upgrade that obviously fixes it does not.
2026-08-17 Green tests, broken build, and a screenshot that lied A passing suite is not evidence the project compiles, and a rendered page is not evidence your code is running.
2026-08-12 The read-only mode that wasn't My own spec classified tools by name prefix. Twelve mutating tools matched no prefix, including the one that commits a queued delete.
2026-08-12 Redaction fails two ways, and only one shows in a diff The wiring gap you can grep for, and the pattern gap that reads as coverage in review while changing zero bytes of output.
2026-07-25 The mutable tag, and the cron line you cannot grep for A nightly cron force-recreated production 25 minutes after a local image build, from a crontab line matching none of the words you would search for.

A note on how these are built, since it is visible in the commit history

I define the requirements, the constraints, and what counts as proof. Claude writes most of the implementation.

Worth being precise about, because "AI-assisted" covers everything from autocomplete to abdication. What I own is the decision record.

The numbers here exist because I decided what would settle a question and then went and got it: 37.6 KB per probe, 57,548 bytes per abandoned session against 5,201 for a terminated one, 134 tools split into 62 mutating and 72 read. Each one has its control written down next to it, because the control is the part that makes it evidence.

The rejections are the same. Prefix-gating the write path was my own written spec and I threw it out after enumerating the tool surface. Upgrading the SDK was the obvious fix for the session leak and I disproved it before shipping it. Widening the redaction patterns to key or auth was the tempting fix, and each rejection is now pinned by a negative test so nobody rediscovers it. Every note ends with what would change my mind.

In all five, the mistake was mine first and a measurement is what corrected me. Where an agent got something wrong, the note says so and says what I changed.

The two mistakes that keep coming back

A rule that only names the cases I thought of, and a signal I read as proof of more than it attests to. Each note says which one it was.

Related

The architecture decisions behind most of these live in mcp-unifi, which is where the read-only gate, the audit log, and the redaction work all ship.

License

CC BY 4.0. Use them, quote them, no attribution rituals required beyond a link.

About

Incident write-ups from systems I run: what broke, what I measured, what I rejected, and what would change my mind.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors