diff --git a/agentic-skills/README.md b/agentic-skills/README.md index e81806b..e8cbf62 100644 --- a/agentic-skills/README.md +++ b/agentic-skills/README.md @@ -16,6 +16,7 @@ What is here so far: | `trivial-pr` | The same, cut down for a one-or-two-line change made from a fresh branch. | | `review` | Review the current branch through several independent lenses, then try to disprove each finding before reporting it. | | `trim-comments` | Push a changeset toward self-documenting code, moving decision history out of comments and into the commit message. | +| `incident-report` | Find the mechanism behind a failed or flaky run, and write it up as a blame-free report that marks every claim as certain or inferred. | | `review-principles` | Reference notes behind the `review` lenses. Not a skill. | If you use one, make it yours. Fork it, rename it, rewrite it, gut it. Change the voice, the steps, diff --git a/agentic-skills/incident-report.md b/agentic-skills/incident-report.md new file mode 100644 index 0000000..9310f3e --- /dev/null +++ b/agentic-skills/incident-report.md @@ -0,0 +1,289 @@ +--- +name: incident-report +description: Investigate a failed or flaky test run, deploy or environment problem down to its mechanism, and write it up as a blame-free incident report that marks every claim as certain or inferred. Use when asked for "the real reason" a build failed, an incident write-up, a postmortem, or before accepting a timeout bump, retry or rerun as the fix. Human-driven: expect many rounds of questions, not a one-shot answer. +--- + +# Incident report + +The goal is one sentence that says what raced what, or what was missing when, backed by +timestamps from more than one source. The report is how that sentence reaches people who were not +in the investigation. + +A report that ends in "probably flaky", "infra hiccup" or "slow CI" has not found the cause. Keep +going, or say plainly that the cause is unknown and which record would find it. + +Think of how an onion rots: from the center outward. To find the source you peel it layer by +layer. Halfway in you will find rot, and it is tempting to stop there and call it the cause. It +usually isn't. It is rot that spread from further in. A timeout that expired, a container that +wasn't there, a queue with no consumer: each is a layer. Keep peeling until you reach the layer +nothing else explains, and fix that one. + +## Who drives this + +This is not an autonomous skill. It will not solve an incident in one go. A human drives the +investigation, and the agent assists. + +That comes from experience. Left alone, the agent writes a shallow report that stops at the first +or second layer and sounds finished. It is not good at doubting its own results. What gets to the +center is a developer who keeps pushing: who asks hard questions, doubts each answer, and works to +understand the processes involved themselves. Each layer usually takes another round of that. + +So expect many rounds and many questions. For the agent, that means: + +- Treat each answer as one layer, not the end. Say which layer you think you reached and what + could still sit under it. +- Say what you did not check and which claims are weakest, so the developer knows where to push. +- When the developer pushes back, take it as the next step, not something to defend against. Go + back to the records. +- Explain the processes involved plainly, so the developer can judge the mechanism themselves. + They need to understand it, not only read the conclusion. +- Don't declare the investigation done. The developer decides when the center is reached. + +## How to use this + +This is a guide, not a script. Every incident is different. Pick the parts that fit, skip the +ones that don't, and change the order when the evidence points somewhere else. The developer +asking for the investigation can steer: "just the timeline", "only check the deploy", "skip the +frequency analysis", "we already know the trigger, find the gap". Follow that over anything here. + +The same goes for the report. The format below lists sections that have been useful. A short +incident may need four of them. Leave out what adds nothing. + +## Principles + +- **A failure is a bug until proven otherwise.** Timeouts, retries and reruns hide it. The report + exists so nobody has to rediscover it. +- **The cause is usually only visible in the correlation.** One source rarely shows it. The test + output says *what* failed, the deploy log says *when* things changed, container logs and metrics + say *what the system was doing*, config and code say *why it could happen*. Put them on one + timeline. +- **Certain and inferred never mix.** Every claim is either read directly from a record, or it is + a conclusion with its basis and a confidence. A reader must be able to tell which is which + without asking. +- **Raise certainty before you write.** When a claim is inferred and a record exists that would + prove or refute it, read that record first. +- **Measured beats argued.** A count of events in the logs over weeks beats a conclusion drawn + from how the config ought to behave. When the two disagree, trust the count. +- **Try to break your own claims.** Look for a counterexample before you publish a + generalisation. "X always happens" needs every case checked, not the one you looked at. +- **Explain the trigger, not only the gap.** A fix that holds whatever the trigger was is + valuable, but it does not close the investigation. The trigger still needs a cause, or an + explicit "unknown" plus the record that would answer it. +- **Blame-free.** No personal names. Attribute to artifacts: PR numbers, build numbers, commits, + config files. A PR that proposed the wrong fix is "proposed in #1234", not a person. Don't shame + anyone, including in examples of what not to do. +- **Plain language.** Short sentences, simple words. A reader who was not there should follow it. + +## Ways in + +Pick what the incident needs. Roughly in the order they usually pay off. + +### Pin the failure + +- Get the exact failure text of every failed step, not just the first one. Strip color codes. +- Group failures by signature. Six failures with the same message in one two-minute window are + one event, not six. Failures with a different signature in the same run may be a second, + unrelated incident. Say so and keep them apart. +- Find when each failure happened. Live progress lines give the real time; summaries printed at + the end of a spec only give the time the spec finished. +- Work out when each failed wait started (failure time minus its budget). The window that matters + is when the waits began, not when they gave up. +- Read the error text for *which* clock ran out. A wait can have more than one phase (e.g. waiting + for a request vs. waiting for its response), and the message usually says which. +- Some budgets are implicit: a poll count times a delay, plus round trips. Measure the real budget + from the run instead of reading it off the code. +- Don't trust a green summary. A report overview can drop a failed feature. Check the raw + per-test results too. + +### Check what was actually tested + +- Which commit did the failing run build and test? A PR run may test a branch that is behind + main, or a job may run main's tests instead of the PR's. Get the SHA from the run itself. +- Which version was deployed where? Claims like "it is redeployed every day, so it has the fix" + need the deploy record, not an assumption. + +### Build the window + +- Find what ran before the failure: the deploy or build that triggered the run, when it finished, + and what the test harness waited on before starting. +- Convert every timestamp to one zone and say which. Sources disagree: CI consoles, container logs + and dashboards often use different zones. +- Pull the system side for the same window: which containers or tasks started, stopped or were + replaced (a new task id in a log stream name means a new container), capacity and scaling + metrics, queue depth and consumer counts, connection open and close events. +- Read ephemeral records first. Some records, such as task stop reasons or instance termination + causes, expire within an hour or a few weeks. Note what has already expired. + +### Diff against green + +- Find the last green run and list every commit between it and the red one. Read each diff. Rule + each one in or out with a reason. +- Compare the same measurement across green and red runs: startup times, scaling steps, queue + depths, step durations. The variable that differs on the red run is the lead. +- Check whether the check that failed is new. A new, stricter assertion can make an old problem + visible for the first time. That is the assertion doing its job. + +### Test the known explanations + +- List the explanations already on the table: earlier incidents, the reasoning behind a proposed + fix, "the CI machine was overloaded", "the instance was reclaimed", the requester's own theory. + Test each one against this incident's records and rule it in or out with the record that decides + it. +- A matching symptom is not a matching cause. Two incidents can fail the same assertion for + different reasons. +- Look at where each piece actually runs in *this* environment. Config differs between environment + types, and a fix in one may not apply to another. +- Rule things out with numbers where you can: memory estimates against limits, request sizes, rates + against published figures. + +### Find the mechanism + +- Read the config and code that decide the behaviour: sizing, placement, scaling, health checks, + readiness gates, entrypoint scripts, retry and reconnect logic. Cite file names. +- State the mechanism as a chain of facts, each with its source, ending at the failure. +- Break a gap into measured phases where you can (placed, pulled, started, ready). "It took four + minutes" hides which phase to fix. +- Beware of misleading metrics. Container memory for a JVM sized by a heap percentage looks full + by design; CPU and heap after garbage collection say more. + +### Reproduce it + +The strongest evidence there is. When the mechanism can be reproduced locally, do it. + +- Aim for the exact failure signature, not just "a failure". +- Find the threshold: how long a pause, how many readers, how tight a race before it fails. +- Show before/after in a small table: runs, outcomes by kind, with and without the fix. +- Prove any new or changed check can go red: break the thing it guards once and watch it fail + when and how you expect. +- To find a race, make it fail more, not less: remove fixed waits, shorten timeouts. + +### Measure how often + +- Go back as far as the records allow, not just the last few runs. +- Show the margin, not only pass/fail. A table of how close each run came shows a race that + pass/fail hides. +- Keep an inventory with explicit exclusions: which failures look similar but are something else, + and why. +- Watch for selection bias: events that happen when no test is running go unnoticed, so "it + always happens during tests" is often "we only notice it during tests". +- For a single event, say so. One occurrence supports a very wide range of rates. + +### Raise certainty, then self-check + +Before writing, go through every inferred claim: + +- Is there a record that would prove it? Read it. +- Is there a config file or code path that would explain it? Read it. +- Does any run in the data contradict it? Check them all. +- Would someone challenge it? Add the link or the exact source. + +Then list what is still inferred and why it can't be proven from here: the record expired, the +source is not reachable, nothing logs that event. When a record held by someone else would settle +it, give them the exact query and the string in the result that decides the question. Asking +others for records costs them time, so do it when you have a strong lead. + +### Decide the fix + +- The Fix section holds what would have prevented *this* incident. Show what it would have done + here, with timestamps, and in any related incident. +- Prefer fixes that make the system predictable over fixes that tolerate unpredictability. Waiting + for a known-good state beats waiting longer. +- Say where it lands (which repo or system) and roughly what it costs. +- Don't propose capacity or infrastructure changes before the mechanism is confirmed. +- List what does not fix it and why: longer timeouts, retries, reruns, workarounds that move the + race. Be concrete: "90s covers today; the gap is set by X, not by Y". +- If the trigger could not be found, add what would make the next one answerable: capture the + records that expired this time, add the log line that was missing. + +## Certainty in the report + +- **Certain:** read directly from a log line, a metric, a report, a reproduction, the config or the + code. Say where. +- **Inferred:** a conclusion. Say what it rests on, and give a confidence in words (high, medium, + low). Never give a percentage unless it was measured. +- **Best guess:** when the cause stays unknown, give one best guess, clearly labelled, and what + would confirm or refute it. Don't weight several guesses against each other without data; that + is guessing dressed as analysis. +- **Open question:** anything unexplained that doesn't fit, even if it doesn't change the fix. Keep + it named instead of smoothing it away. +- Word links to earlier reports carefully. "Would also explain" is honest; "fully resolves" needs + proof. + +## Report format + +Markdown. Draft it as a local file first, named the way it should appear once published. Publish +only when the developer asks, where they ask. Publishing makes it visible to others. + +Pick from these sections. Keep the order; drop what adds nothing. + +``` +# - incident + + ended with failed . + +All times UTC. **Certain** means read directly from a log line, a metric, the config or the code. +**Inferred** means a conclusion drawn from those, with what it rests on. + + + +## TL;DR + +- Cause: +- Fix: +- Root-caused: +- Impact outside test environments: + +## Summary + + + + +## What happened on + +| time | event | source | +|---|---|---| +| hh:mm:ss | | | + + + +**Certain** +- + +**Inferred** +- . It rests on . Confidence: . + +## +## (comparison table against green runs) +## Reproduction (when there is one) +## How often (every run the records cover, with the margin) +## Considered and ruled out (each with the record or number that rules it out) +## Fix +## What does not fix it +## Open questions / best guess (when the trigger is unknown) +## Where to look next time (the one comparison that finds this fastest) + +Not checked: . +``` + +Rules for the format: + +- Timeline rows are records. Anything derived goes in the Certain/Inferred lists or a "Derived" + subsection, never in the timeline. +- Bold the two or three rows that carry the argument. +- Quote log text exactly, in backticks. +- Tables over prose for anything compared across runs. +- Keep it as short as the incident allows. Trim before publishing. +- Never copy secrets, passwords, API keys or tokens from logs into the report, even when the + source prints them. Report the leak separately. +- A closing line on how the report was built (which sources were correlated) is worth adding when + that correlation was itself the finding. + +## Before publishing + +- Every Certain claim: can you point at the record? If not, move it to Inferred. +- Every Inferred claim: does it say what it rests on and how confident it is, in words? +- Every "always", "never", "every time": checked against all runs in the data? +- Every number in a table: recomputed from the source, not from memory? +- Does every item under Fix prevent this incident? +- Is the impact outside test environments stated? +- No names, no secrets, nothing that shames anyone.