feat(daemon): the daemon reports the failures that stop it working - #5752
Conversation
We already run this loop end to end for ONE class of failure: the desktop shell classifies a bootstrap that never completed, the cloud sink counts signatures, field-failure-issues.yml files a deduped issue, and the hourly fixer routine picks it up. It is live and it works (a Windows machine on a blocked index surfaced as no_distribution/Windows/3.11 with nobody reporting it). That pipeline had one producer, and it was the installer. The daemon, which runs on every node and is the single point of failure for all customer data, reported nothing when it could not run. On 2026-09-08 a Pro node's daemon was locked out of its own pid lock and retried every 30s for twelve hours: it exits 0 so launchd recorded successes, install_daemon_error_event_handler (sync.py:23972) and start_update_check_thread (sync.py:24459) both initialise AFTER the gate at 23832 so neither ever ran, and with no daemon there was no heartbeat, so the node just went quiet. Quiet is not an event. The failure existed in one place: a log file on the customer's laptop. clawmetry/field_report.py makes the daemon the second producer: - daemon_lock_refused, reported from ABOVE the gate (inline, not a thread: the process exits on the next line and would take a background thread with it) and only when ingest is actually stale, because refusing the lock is normally correct and frequent. - daemon_ingest_stalled, from the watchdog thread that keeps running when the ingest loop does not. This is detectors.py's no_progress question, asked about ourselves for the first time. Same privacy contract as the shell's, asserted key by key over the serialised body: a closed enum, platform names, a version. Same opt-outs, and the CANONICAL egress gate: endpoints.egress_suppressed(), not the narrower is_custom_endpoint(), because an air-gapped node sets no endpoint and a self-hosted server IS the endpoint. Verified live: air-gapped sends nothing. Throttled by an on-disk stamp, since a restart loop is a new process every 30 seconds and in-memory throttling would throttle nothing. The session id is derived as <class>-<UTC date> because the sink is keyed UNIQUE (install_id, session_id, stage) with ON CONFLICT DO NOTHING: a constant would have recorded the first daemon failure an install ever had and muted every one after it. Verified end to end against a local stand-in for the sink, reproducing the customer's machine: before, the failure reached nobody; after, one classified aggregate arrives, and three more restarts into the same failure add none. Needs clawmetry-cloud PR (new stage + enum entries + the aggregate view) to reach the issue filer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYQo5mXvPp3zT9tjq5mEaj
|
| @@ -0,0 +1,265 @@ | |||
| """Field-failure reports from the sync daemon (Requirement: Field Failure | |||
There was a problem hiding this comment.
The requirement specifies field failure reporting for "desktop shell bootstrap failures" (AC-FFR-001.1), but this PR implements daemon-specific failure reporting (daemon_lock_refused, daemon_ingest_stalled) that extends the scope significantly beyond what is documented in the acceptance criteria.
| # A sync cycle is about a minute. An hour without one completing is not a slow | ||
| # machine, it is a daemon that is alive and not working: the failure mode that | ||
| # has no supervisor at all, because the process is up and every liveness probe | ||
| # in the stack says so. | ||
| _STALLED_INGEST_SECS = float(os.environ.get("CLAWMETRY_STALLED_INGEST_SECS", "") or 3600) | ||
|
|
||
|
|
||
| def _report_if_ingest_stalled() -> None: | ||
| """detectors.py ships ``no_progress`` to tell a customer their agent has | ||
| stopped getting anywhere. This is the same question asked about ourselves, | ||
| from the watchdog thread, which keeps running when the ingest loop does | ||
| not. Throttled to one report per six hours by the field-report stamp. | ||
| """ | ||
| try: | ||
| from clawmetry import field_report as _fr | ||
|
|
||
| age = _fr.last_sync_age_secs() | ||
| if age is not None and age > _STALLED_INGEST_SECS: | ||
| _fr.report_daemon_failure("daemon_ingest_stalled", | ||
| version=_get_version()) | ||
| except Exception as e: # noqa: BLE001 - the watchdog must never die | ||
| log.debug("stall check skipped: %s", e) | ||
|
|
||
|
|
There was a problem hiding this comment.
The requirement's delivery status indicates daemon failure reporting is "NOT BUILT" and scoped to the clawmetry-cloud repo, but this PR implements _report_if_ingest_stalled() in the daemon's sync.py with a configurable stall detection threshold (_STALLED_INGEST_SECS).
The lock fix stops this node breaking again; it does nothing about the fact that a node in this state reaches nobody. #5752 makes the daemon a producer on the field-failure pipeline. Also re-triggers the CodeQL analyses, which queue-priority cancelled when main moved and which cannot be rerun in place. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYQo5mXvPp3zT9tjq5mEaj
Drift Bot was right: this extends "Field Failure Reporting and Auto-Triage" from install-time to run-time failures, and that scope change existed only in the code. The parent requirement scopes AC-FFR-001 to the desktop shell's bootstrap and lists daemon reporting as NOT BUILT. Written as a LIVE child requirement rather than an edit to the parent, since a pending suggestion is not a record: "Daemon Field-Failure Reporting" (AC-FFR-005.1 through .9), carrying the reason the daemon reached nobody -- every self-repair mechanism it has initialises after the gate that failed, it exits 0 so the supervisor counts successes, and a dead daemon sends no heartbeat -- and the boundary that this bounds DETECTION latency and repairs nothing. Tests now name the criteria they cover. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYQo5mXvPp3zT9tjq5mEaj
|
| @@ -0,0 +1,268 @@ | |||
| """Field-failure reports from the sync daemon. | |||
There was a problem hiding this comment.
The blueprint is a template placeholder with no architecture, contracts, or specifications, while the implementation in clawmetry/field_report.py delivers a complete field-failure reporting system with specific failure classes (daemon_lock_refused, daemon_ingest_stalled), privacy contracts, session ID strategies, and throttling mechanisms that must be documented.
| @@ -0,0 +1,268 @@ | |||
| """Field-failure reports from the sync daemon. | |||
There was a problem hiding this comment.
The parent blueprint is a template placeholder with no documented architecture or contracts. The implementation includes two distinct failure classes (daemon_lock_refused from sync.py:23858 and daemon_ingest_stalled from sync.py:433) with different triggering mechanisms (inline vs. watchdog thread) that should be specified in the architecture documentation.
The map carries each module's first docstring line, so rewording one drifts it. Caught by CI's own guard, which is the guard working. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYQo5mXvPp3zT9tjq5mEaj
|
| @@ -0,0 +1,268 @@ | |||
| """Field-failure reports from the sync daemon. | |||
There was a problem hiding this comment.
The blueprint is a template placeholder with no architecture or specifications, while the implementation delivers a complete field-failure reporting system with specific failure classes, privacy contracts, session ID strategies, and throttling mechanisms that should be documented in the blueprint to maintain design-code alignment.
| @@ -0,0 +1,268 @@ | |||
| """Field-failure reports from the sync daemon. | |||
There was a problem hiding this comment.
The parent blueprint is a template placeholder with no documented architecture or contracts. The implementation includes two distinct failure classes with different triggering mechanisms (daemon_lock_refused inline vs. daemon_ingest_stalled from watchdog thread) that should be specified in the blueprint's architecture documentation.
Drift Bot's remaining objection was that both blueprints were still template placeholders while the code carried real architecture. Both are now written: the child documents this feature (five ADRs), and the parent documents the pipeline it joins, which had shipped and been live for weeks against an empty blueprint. The three choices here that look arbitrary and are not now name their ADR at the call site, so the next reader finds the reasoning rather than rediscovering it: report from above the gate, send inline on a path that exits, derive the session id rather than fix it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYQo5mXvPp3zT9tjq5mEaj
|
Drift Bot reported clawmetry/field_report.py as missing from MODULE_MAP.md. It was not missing: it was at line 166, cut mid-clause, because the generator took the first LINE of the docstring and that sentence wrapped. Thirteen other modules were already in that state, so the table has been quietly describing detectors.py as "research-backed, judge-free, CPU-cheap trajec" for as long as anyone has been reading it. Nothing failed, because the generator and its --check gate agreed with each other perfectly on a truncated string. _summary() now joins the first paragraph and cuts at the first sentence end, ignoring the two things that look like one and are not (a version or section number, an initial), bounded at 200 chars. Wrapping a line is a formatting choice and must not change generated documentation. 17 rows become whole sentences; no row loses information. tests/test_module_map_summaries_are_whole.py: an auto-discovering check that no row ends mid-sentence, plus the generator's behaviour pinned directly. Proven red against origin/main's generator (5 failures). Named in ci.yml. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYQo5mXvPp3zT9tjq5mEaj
|
The lock fix stops this node breaking again; it does nothing about the fact that a node in this state reaches nobody. #5752 makes the daemon a producer on the field-failure pipeline. Also re-triggers the CodeQL analyses, which queue-priority cancelled when main moved and which cannot be rerun in place. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYQo5mXvPp3zT9tjq5mEaj
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
|
✨ auto-fixed: merged latest main into branch to keep it up to date Generated by Claude Code |
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
|
✨ auto-fixed: merged latest main into branch (was BEHIND; no conflicts) Generated by Claude Code |
#5745) (#5751) * [RELEASE] daemon lock self-heal + honest sync status (carries #5745) Publishes the fix for the field failure reported 2026-09-09: a Pro node whose sync daemon had been locked out of its own pid lock since the previous evening, behind a status screen of green ticks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYQo5mXvPp3zT9tjq5mEaj * Note the follow-up in the CHANGELOG entry The lock fix stops this node breaking again; it does nothing about the fact that a node in this state reaches nobody. #5752 makes the daemon a producer on the field-failure pipeline. Also re-triggers the CodeQL analyses, which queue-priority cancelled when main moved and which cannot be rerun in place. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYQo5mXvPp3zT9tjq5mEaj --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Resolves conflict in docs/MODULE_MAP.md: take main's addition of openclaw_share.py adapter row alongside the existing openclaw.py entry. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UNcAS3GeEW3FkQyZ2vcsrF
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
Fixes the Syntax & Lint CI failure: docs/MODULE_MAP.md is out of date. Regenerate it with: python3 scripts/gen_module_map.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBmS1miHmw8rA5a8jny1Qr
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
|
Merging main brought in #5790, whose reconcile step closes a field-failure issue when its signature stops being reported. Both branches touch the same workflow; the merge was textually clean and the autoclose tests still pass, but it exposed a collision this PR introduces. /api/desktop/_failures groups by stage x class x OS x Python. The issue title is the dedupe key and carried only class/OS/Python, so a daemon that stopped ingesting and an install that never started could share a title. The second to arrive would merely refresh the first, under a lede describing the wrong failure and pointing the fixer at the wrong file -- the exact confusion the new per-stage lede exists to prevent. Demonstrated: same class, OS and Python, two stages, one title. A stage other than bootstrap_failed now prefixes the title. bootstrap_failed keeps the historical shape byte for byte, because renaming it orphans every open issue, which the title contract forbids; a new stage takes a new prefix rather than reshaping the existing key. Pinned by three tests that compute the title with the workflow's OWN shell rather than a reimplementation, and proven red against the colliding version. Recorded on the Field Failure Reporting and Auto-Triage blueprint as a contract on the key. docs/MODULE_MAP.md was stale after the merge, which is what reddened Syntax & Lint AND produced Drift Bot's only finding (field_report.py 'missing' from the map). One root cause, both cleared by regenerating. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xb6A5G74JiMe3zHFs1JZEP
|
The blueprint's status banner said 'until BOTH merge', but clawmetry-cloud #2364 merged 2026-09-09 and is deployed: /api/desktop/_failures returns a stage on every signature today, verified against the live endpoint. So the ordering this feature needs is already satisfied and holding this PR only leaves the cloud change unused. The banner now says which half is live and which is in review, and records the caveat found while merging: a drift finding that field_report.py is absent from MAIN is correct and expected until this merges, but a finding that it is missing from this PR's own MODULE_MAP.md has now been wrong twice -- the entry sits between extensions.py and flow_trace.py, alphabetically correct, while one finding proposed a position between efficiency.py and endpoints.py that it could not occupy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xb6A5G74JiMe3zHFs1JZEP
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
#5752) (#5795) Claude-Session: https://claude.ai/code/session_01Xb6A5G74JiMe3zHFs1JZEP Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
Live in PyPI Downloaded Both call sites ship in The payload, from the shipped code: {"arch": "arm64", "bootstrap_python": "3.14", "desktop_version": "0.12.855",
"failure_class": "daemon_lock_refused", "install_id": "763e5d9c-…",
"os": "Darwin", "os_version": "25.3.0",
"session_id": "daemon_lock_refused-20260910", "stage": "daemon_failed"}Privacy asserted over the serialised body, not key by key, so a nested value cannot smuggle anything past a key check:
Suppression is total:
The sink half was already live and stage-aware (clawmetry-cloud #2364, merged 2026-09-09): No further cloud promotion needed: the cloud change shipped a day ago, and this is the node-side half that reaches machines through the daemon's own PyPI self-update. |
Requirement: Daemon Field-Failure Reporting (AC-FFR-005), a live child of Field Failure Reporting and Auto-Triage.
Needs clawmetry-cloud PR (linked below) to reach the issue filer.
The gap
We already run this loop end to end, for exactly one class of failure:
It is live and it works. Right now it is showing
no_distribution / Windows / 3.11, a machine nobody reported.It has one producer, and it is the installer. The daemon, which runs on every node and is the single point of failure for all of a customer's data, produced nothing when it could not run.
On 2026-09-08 a Pro node's daemon was locked out of its own pid lock and retried every 30 seconds for twelve hours. It reached nobody, by construction:
0, soKeepAliverecorded ~1,400 successful runsinstall_daemon_error_event_handler()is atsync.py:23972; the gate is at23832start_update_check_thread(role="daemon")is atsync.py:24459, 627 lines past the gateclawmetry statuslaunchctl list's exit code, which is0for a merely registered job (fixed in #5745)Every self-repair mechanism we ship starts after the gate that failed. The failure existed in one place on Earth: a log file on the customer's laptop. He emailed us a screenshot.
What this adds
clawmetry/field_report.py, the daemon's producer for that same pipeline. Two classes:daemon_lock_refused— reported from ABOVE the gate, and only when ingest is actually stale. Refusing the lock is normally correct and frequent (a double start, a manualpython -m clawmetry.sync); refusing it while nothing is moving is the field failure. Sent inline rather than on a thread, because the process exits on the next line and would take a background thread with it, which would have made the report as silent as the failure.daemon_ingest_stalled— from the watchdog thread, which keeps running when the ingest loop does not. This isdetectors.py'sno_progressquestion, asked about ourselves for the first time. We ship stuck-detection for our customers' agents and had never once pointed it at our own daemon.Privacy and egress
Same contract as the shell's, asserted key by key over the serialised body so nothing smuggles a path through a nested value: a closed enum, platform names, a version. No paths, usernames, hostnames, node id, or log text.
The gate is
endpoints.egress_suppressed(), deliberately not the narroweris_custom_endpoint()I reached for first. That one misses both cases that matter: an air-gapped node sets no endpoint, and a self-hosted server IS the endpoint so it never sets one either. A report about a broken daemon is exactly the kind of well-meant call that gets shipped past an air gap. Verified live:CLAWMETRY_OFFLINE=1in the same broken state sends nothing.docs/EGRESS.mdcarries the new row.Two things that would have made it useless
UNIQUE (install_id, session_id, stage)withON CONFLICT DO NOTHING. A constant session id would have recorded the first daemon failure an install ever had and silently discarded every one after it, including a different failure class:last_seenfrozen forever. The id is derived as<class>-<UTC date>, which makes the server key "one row per class per install per day" and holds even when the local stamp is gone.Verified end to end, not just unit-tested
A local stand-in for the cloud sink, and the customer's machine reproduced (a lock held by a live daemon, ingest stale since 19:32):
Three further restarts into the same failure: zero additional posts. Air-gapped: zero.
tests/test_daemon_field_report.py, 24 tests, named inci.yml. The workflow body now says which stage a signature came from, because "bootstrap" over a runtime failure sends the fixer to the wrong file.🤖 Generated with Claude Code
https://claude.ai/code/session_01CYQo5mXvPp3zT9tjq5mEaj
Update after merging main: a collision this PR introduced
mainnow carries #5790, whose reconcile step closes a field-failure issue when its signature stops being reported. Both branches touchfield-failure-issues.yml. The merge was textually clean and the autoclose tests still pass — but a clean textual merge is not a working one, and checking turned up a real collision./api/desktop/_failuresgroups by stage × class × OS × Python. The issue title is the dedupe key, and it carried only class/OS/Python. So:Two distinct signatures, one key. Whichever arrived second would merely refresh the first, under a lede describing the wrong failure and pointing the fixer at the wrong file — precisely the confusion this PR's per-stage lede exists to prevent.
Fixed: a stage other than
bootstrap_failedprefixes the title.bootstrap_failedkeeps the historical shape byte for byte — that first line is #5739's actual title — because renaming it orphans every open issue, which the title contract explicitly forbids. A new stage takes a new prefix rather than reshaping the existing key.Three tests pin it, computing the title with the workflow's own shell rather than a reimplementation (a reimplementation would drift from the thing it claims to check). Proven red against the colliding version:
Recorded on the Field Failure Reporting and Auto-Triage blueprint as a contract on the key: the dedupe key must carry every dimension the aggregate groups by.
Also fixed
docs/MODULE_MAP.mdhad gone stale as main moved under the branch. That single cause produced both red signals:Syntax & Lintfailing ongen_module_map.py --check, and Drift Bot's only finding ("field_report.pyis missing from MODULE_MAP.md"). Regenerated —field_report.pyis now atdocs/MODULE_MAP.md:167and the map is in sync.837 tests pass locally across the field-report, autoclose, module-map, workflow-YAML and workflow-integrity suites.