Field-failure issues close themselves when the signature stops occurring by vivekchand · Pull Request #5790 · vivekchand/clawmetry

vivekchand · 2026-09-10T03:53:01Z

Closes #5739.

The bug is in the pipeline, not the failure

field-failure-issues.yml files one deduped issue per failure signature and refreshes it daily while it keeps happening. It had no third state. A signature that stopped simply aged out of the 30-day aggregate window, and its issue stayed open forever with the last comment still reading "Still occurring".

#5739 is the example, and it is the whole of the current backlog:

{"signatures":[{"failure_class":"no_distribution","os":"Windows",
  "bootstrap_python":"3.11","installs":1,"events":1,
  "last_seen":"2026-09-07T11:23:32Z","latest_desktop_version":"0.12.826"}],
 "window_days":30}

One install, one event, silent since — and nothing in the pipeline would ever have closed it.

I checked whether #5739 itself was fixable first, and it is not: no_distribution / Windows / 3.11 is the designed fallback in _retry_on_known_good_python, reached when _known_good_python() returns None both before and after _winget_install_python(). Its splash message is already specific and already correct — the "Python too old" wrongness was fixed for #5628. So the issue is real, unfixable from here, and permanent. That last property is the defect worth shipping.

What changed

A reconcile step closes any open field-failure issue whose signature is absent from the snapshot the filing step just took. Absence is a real signal, not missing data: the endpoint reports every signature seen in the window, so an absent title has had no event on any machine for the whole window.

Three guards keep it from sweeping the tracker on bad data:

condition	behaviour
payload declares no `window_days`	malformed — file nothing, close nothing
no snapshot file (endpoint unreachable, filing step bailed)	close nothing
well-formed payload, zero signatures	close everything — the honest reading

The closure comment states what it cannot know: from an aggregate, a shipped fix and the last affected machine going away are indistinguishable, so it says the failure stopped and explicitly not that anything was repaired. A recurrence files a fresh issue on the next tick.

Verification

tests/test_field_failure_autoclose.py extracts the reconcile step's real shell from the workflow and executes it against a fake gh. Asserting on the YAML text would pass just as happily for a step that parses and does nothing — the exact failure mode this workflow exists to catch (ADR-005).

case	expected	result
both signatures still live	close nothing	✅ nothing
5739 stopped, 5628 still live	close 5739 only	✅ `CLOSE 5739`
zero live signatures	close both	✅ `CLOSE 5628`, `CLOSE 5739`
endpoint unreachable, no snapshot	close nothing	✅ nothing, logs `closing nothing`

Proven red on the unfixed workflow:

AssertionError: the workflow has no step that closes resolved signatures

6 passed after. Named in ci.yml — CI here runs explicit file lists, so a test that is not listed runs in no job at all.

CI-only change: no package code, so no [RELEASE].

No-PRD: closes an auto-filed field-failure issue by fixing the filer that cannot retire its own alarms; no product surface changes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xb6A5G74JiMe3zHFs1JZEP

An alarm that cannot clear is not an alarm. field-failure-issues.yml files one deduped issue per failure signature and refreshes it daily while it keeps happening, and had no third state. A signature that STOPPED aged out of the 30-day aggregate window and left its issue open forever, with the last comment still reading 'Still occurring'. #5739 is the example: no_distribution / Windows / py3.11, one distinct install, one event, last seen 2026-09-07, and nothing in the pipeline would ever have closed it. The endpoint reports every signature seen in the window, so absence from it is a real signal -- no event on any machine for the whole window. A reconcile step now closes an open field-failure issue whose signature is absent from the snapshot the filing step just took. Three things keep it from sweeping the tracker on bad data: - the filing step refuses a payload that declares no window_days, so a shape change reads as malformed rather than as 'nothing is failing'; - no snapshot file (endpoint unreachable, filing bailed) closes nothing; - zero live signatures DOES close everything, which is the honest reading of a well-formed empty payload. The closure comment says what it does not know: from an aggregate, a shipped fix and the last affected machine going away are indistinguishable, so it claims the failure stopped, never that anything was repaired. A recurrence files a fresh issue on the next tick. tests/test_field_failure_autoclose.py executes the reconcile step's real shell against a fake gh rather than asserting on YAML text -- a text assertion would pass for a step that parses and does nothing, which is the exact failure mode this workflow exists to catch. Proven red on the unfixed workflow. Named in ci.yml, because CI here runs explicit file lists. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xb6A5G74JiMe3zHFs1JZEP

8090-software-factory · 2026-09-10T03:53:57Z

⚠️ Drift Bot (ClawMetry): 3 potential drift finding(s)

1. Blueprint: Field Failure Reporting and Auto-Triage

File: .github/workflows/field-failure-issues.yml:148

The workflow now includes a reconcile step that closes issues when their signatures stop occurring, but the blueprint documents only the filing and refreshing behavior, not the closure mechanism or the three guards that protect it.

2. Blueprint: Field Failure Reporting and Auto-Triage

File: .github/workflows/field-failure-issues.yml:65

The filing step now validates that the payload declares window_days before writing the snapshot file, but the blueprint does not document this malformed-payload guard or its behavior.

3. Blueprint: Field Failure Reporting and Auto-Triage

File: .github/workflows/field-failure-issues.yml:190

The closure comment specifies exact language including "not why" and "not a claim that anything was repaired", but the blueprint does not document what closure comments should say or their constraints.

8090-software-factory · 2026-09-10T03:54:00Z

              echo "filed: $TITLE"
            fi
          done
+


⚠️ Drift Bot (ClawMetry) — Blueprint: Field Failure Reporting and Auto-Triage

The workflow now includes a reconcile step that closes issues when their signatures stop occurring, but the blueprint documents only the filing and refreshing behavior, not the closure mechanism or the three guards that protect it.

8090-software-factory · 2026-09-10T03:54:02Z

          echo "signatures reported: $count"
-          [ "$count" = "0" ] && exit 0
+
+          # A well-formed payload declares its own window. Without that we


⚠️ Drift Bot (ClawMetry) — Blueprint: Field Failure Reporting and Auto-Triage

The filing step now validates that the payload declares window_days before writing the snapshot file, but the blueprint does not document this malformed-payload guard or its behavior.

Drift Bot found three: the reconcile step, the malformed-payload guard, and the closure comment's constraints were all live in the workflow and absent from the Field Failure Reporting and Auto-Triage blueprint. A capability the record does not carry is one the next change can quietly undo. The blueprint now holds the seven contracts (absence is a positive signal; the snapshot is taken before filing; no window_days means malformed; no snapshot closes nothing; zero signatures closes everything; a closure states what it does not know; a recurrence files fresh) and an ADR for why the guard executes the step instead of asserting on its YAML. MIN_INSTALLS=1 is recorded as deliberate and as safe only because the alarm can now retire itself. Placed near the top of that record, per FLYWHEEL 0c. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xb6A5G74JiMe3zHFs1JZEP

8090-software-factory · 2026-09-10T04:04:57Z

✅ Drift Bot (ClawMetry): no drift detected

Drift Bot analyzed the changed files against this project's blueprints and requirements and found no drift.

Merging main brought in #5790, whose reconcile step closes a field-failure issue when its signature stops being reported. Both branches touch the same workflow; the merge was textually clean and the autoclose tests still pass, but it exposed a collision this PR introduces. /api/desktop/_failures groups by stage x class x OS x Python. The issue title is the dedupe key and carried only class/OS/Python, so a daemon that stopped ingesting and an install that never started could share a title. The second to arrive would merely refresh the first, under a lede describing the wrong failure and pointing the fixer at the wrong file -- the exact confusion the new per-stage lede exists to prevent. Demonstrated: same class, OS and Python, two stages, one title. A stage other than bootstrap_failed now prefixes the title. bootstrap_failed keeps the historical shape byte for byte, because renaming it orphans every open issue, which the title contract forbids; a new stage takes a new prefix rather than reshaping the existing key. Pinned by three tests that compute the title with the workflow's OWN shell rather than a reimplementation, and proven red against the colliding version. Recorded on the Field Failure Reporting and Auto-Triage blueprint as a contract on the key. docs/MODULE_MAP.md was stale after the merge, which is what reddened Syntax & Lint AND produced Drift Bot's only finding (field_report.py 'missing' from the map). One root cause, both cleared by regenerating. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xb6A5G74JiMe3zHFs1JZEP

…5752) * feat(daemon): the daemon reports the failures that stop it working We already run this loop end to end for ONE class of failure: the desktop shell classifies a bootstrap that never completed, the cloud sink counts signatures, field-failure-issues.yml files a deduped issue, and the hourly fixer routine picks it up. It is live and it works (a Windows machine on a blocked index surfaced as no_distribution/Windows/3.11 with nobody reporting it). That pipeline had one producer, and it was the installer. The daemon, which runs on every node and is the single point of failure for all customer data, reported nothing when it could not run. On 2026-09-08 a Pro node's daemon was locked out of its own pid lock and retried every 30s for twelve hours: it exits 0 so launchd recorded successes, install_daemon_error_event_handler (sync.py:23972) and start_update_check_thread (sync.py:24459) both initialise AFTER the gate at 23832 so neither ever ran, and with no daemon there was no heartbeat, so the node just went quiet. Quiet is not an event. The failure existed in one place: a log file on the customer's laptop. clawmetry/field_report.py makes the daemon the second producer: - daemon_lock_refused, reported from ABOVE the gate (inline, not a thread: the process exits on the next line and would take a background thread with it) and only when ingest is actually stale, because refusing the lock is normally correct and frequent. - daemon_ingest_stalled, from the watchdog thread that keeps running when the ingest loop does not. This is detectors.py's no_progress question, asked about ourselves for the first time. Same privacy contract as the shell's, asserted key by key over the serialised body: a closed enum, platform names, a version. Same opt-outs, and the CANONICAL egress gate: endpoints.egress_suppressed(), not the narrower is_custom_endpoint(), because an air-gapped node sets no endpoint and a self-hosted server IS the endpoint. Verified live: air-gapped sends nothing. Throttled by an on-disk stamp, since a restart loop is a new process every 30 seconds and in-memory throttling would throttle nothing. The session id is derived as <class>-<UTC date> because the sink is keyed UNIQUE (install_id, session_id, stage) with ON CONFLICT DO NOTHING: a constant would have recorded the first daemon failure an install ever had and muted every one after it. Verified end to end against a local stand-in for the sink, reproducing the customer's machine: before, the failure reached nobody; after, one classified aggregate arrives, and three more restarts into the same failure add none. Needs clawmetry-cloud PR (new stage + enum entries + the aggregate view) to reach the issue filer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYQo5mXvPp3zT9tjq5mEaj * Record the product decision Drift Bot correctly objected to Drift Bot was right: this extends "Field Failure Reporting and Auto-Triage" from install-time to run-time failures, and that scope change existed only in the code. The parent requirement scopes AC-FFR-001 to the desktop shell's bootstrap and lists daemon reporting as NOT BUILT. Written as a LIVE child requirement rather than an edit to the parent, since a pending suggestion is not a record: "Daemon Field-Failure Reporting" (AC-FFR-005.1 through .9), carrying the reason the daemon reached nobody -- every self-repair mechanism it has initialises after the gate that failed, it exits 0 so the supervisor counts successes, and a dead daemon sends no heartbeat -- and the boundary that this bounds DETECTION latency and repairs nothing. Tests now name the criteria they cover. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYQo5mXvPp3zT9tjq5mEaj * Regenerate the module map after the docstring edit The map carries each module's first docstring line, so rewording one drifts it. Caught by CI's own guard, which is the guard working. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYQo5mXvPp3zT9tjq5mEaj * Point the code at the decisions that explain it Drift Bot's remaining objection was that both blueprints were still template placeholders while the code carried real architecture. Both are now written: the child documents this feature (five ADRs), and the parent documents the pipeline it joins, which had shipped and been live for weeks against an empty blueprint. The three choices here that look arbitrary and are not now name their ADR at the call site, so the next reader finds the reasoning rather than rediscovering it: report from above the gate, send inline on a path that exits, derive the session id rather than fix it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYQo5mXvPp3zT9tjq5mEaj * Generate the module map from the first SENTENCE, not the first line Drift Bot reported clawmetry/field_report.py as missing from MODULE_MAP.md. It was not missing: it was at line 166, cut mid-clause, because the generator took the first LINE of the docstring and that sentence wrapped. Thirteen other modules were already in that state, so the table has been quietly describing detectors.py as "research-backed, judge-free, CPU-cheap trajec" for as long as anyone has been reading it. Nothing failed, because the generator and its --check gate agreed with each other perfectly on a truncated string. _summary() now joins the first paragraph and cuts at the first sentence end, ignoring the two things that look like one and are not (a version or section number, an initial), bounded at 200 chars. Wrapping a line is a formatting choice and must not change generated documentation. 17 rows become whole sentences; no row loses information. tests/test_module_map_summaries_are_whole.py: an auto-discovering check that no row ends mid-sentence, plus the generator's behaviour pinned directly. Proven red against origin/main's generator (5 failures). Named in ci.yml. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYQo5mXvPp3zT9tjq5mEaj * chore: regenerate MODULE_MAP.md after field_report module addition Fixes the Syntax & Lint CI failure: docs/MODULE_MAP.md is out of date. Regenerate it with: python3 scripts/gen_module_map.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBmS1miHmw8rA5a8jny1Qr * Put the stage in the dedupe key, and regenerate the module map Merging main brought in #5790, whose reconcile step closes a field-failure issue when its signature stops being reported. Both branches touch the same workflow; the merge was textually clean and the autoclose tests still pass, but it exposed a collision this PR introduces. /api/desktop/_failures groups by stage x class x OS x Python. The issue title is the dedupe key and carried only class/OS/Python, so a daemon that stopped ingesting and an install that never started could share a title. The second to arrive would merely refresh the first, under a lede describing the wrong failure and pointing the fixer at the wrong file -- the exact confusion the new per-stage lede exists to prevent. Demonstrated: same class, OS and Python, two stages, one title. A stage other than bootstrap_failed now prefixes the title. bootstrap_failed keeps the historical shape byte for byte, because renaming it orphans every open issue, which the title contract forbids; a new stage takes a new prefix rather than reshaping the existing key. Pinned by three tests that compute the title with the workflow's OWN shell rather than a reimplementation, and proven red against the colliding version. Recorded on the Field Failure Reporting and Auto-Triage blueprint as a contract on the key. docs/MODULE_MAP.md was stale after the merge, which is what reddened Syntax & Lint AND produced Drift Bot's only finding (field_report.py 'missing' from the map). One root cause, both cleared by regenerating. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xb6A5G74JiMe3zHFs1JZEP * Record that the sink half is already live, and re-trigger drift The blueprint's status banner said 'until BOTH merge', but clawmetry-cloud #2364 merged 2026-09-09 and is deployed: /api/desktop/_failures returns a stage on every signature today, verified against the live endpoint. So the ordering this feature needs is already satisfied and holding this PR only leaves the cloud change unused. The banner now says which half is live and which is in review, and records the caveat found while merging: a drift finding that field_report.py is absent from MAIN is correct and expected until this merges, but a finding that it is missing from this PR's own MODULE_MAP.md has now been wrong twice -- the entry sits between extensions.py and flow_trace.py, alphabetically correct, while one finding proposed a position between efficiency.py and endpoints.py that it could not occupy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xb6A5G74JiMe3zHFs1JZEP --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

8090-software-factory Bot reviewed Sep 10, 2026

View reviewed changes

vivekchand merged commit eb3edb1 into main Sep 10, 2026
38 checks passed

vivekchand deleted the fix/field-failure-issues-close-when-resolved branch September 10, 2026 04:22

vivekchand mentioned this pull request Sep 10, 2026

[field-failure] no_distribution on Windows (py 3.11) #5739

Closed

vivekchand mentioned this pull request Sep 10, 2026

feat(daemon): the daemon reports the failures that stop it working #5752

Merged

Navigation Menu

Field-failure issues close themselves when the signature stops occurring - #5790

Field-failure issues close themselves when the signature stops occurring#5790
vivekchand merged 2 commits into
mainfrom
fix/field-failure-issues-close-when-resolved

vivekchand commented Sep 10, 2026 •

edited

Loading

Uh oh!

8090-software-factory Bot commented Sep 10, 2026

Uh oh!

8090-software-factory Bot Sep 10, 2026

Uh oh!

8090-software-factory Bot Sep 10, 2026

Uh oh!

8090-software-factory Bot commented Sep 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

vivekchand commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The bug is in the pipeline, not the failure

What changed

Verification

Uh oh!

8090-software-factory Bot commented Sep 10, 2026

⚠️ Drift Bot (ClawMetry): 3 potential drift finding(s)

1. Blueprint: Field Failure Reporting and Auto-Triage

2. Blueprint: Field Failure Reporting and Auto-Triage

3. Blueprint: Field Failure Reporting and Auto-Triage

Uh oh!

8090-software-factory Bot Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

8090-software-factory Bot Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

8090-software-factory Bot commented Sep 10, 2026

✅ Drift Bot (ClawMetry): no drift detected

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vivekchand commented Sep 10, 2026 •

edited

Loading