fix(scheduler): tolerate backwards clock steps in worker liveness refresh

[b7r6]

What and why

A backwards step on the scheduler's wall clock (NTP step, VM time sync) makes refresh_lifetime error on the "stale" timestamp, and that error aborts processing of the very message that proved the worker alive. When that message is a finished ExecuteResult or ExecuteComplete, the result the worker already produced is discarded and the operation is stranded in Executing with nothing left to finish it.

Two changes:

• api_worker_scheduler: a liveness refresh with an older timestamp keeps the newer one and logs a warning instead of erroring — the worker is no less alive because the scheduler's clock stepped.
• worker_api_server: the liveness refresh on result/accept/decline/complete messages is best-effort (touch_liveness_best_effort), so a result the worker already produced always reaches update_action even if the refresh fails for some other reason (e.g. the worker was just evicted).

How was this verified?

• New nativelink-scheduler/tests/liveness_clock_step_test.rs: steps the mock clock backwards between two messages from the same worker and asserts the refresh succeeds, the newer timestamp is kept, and the action completes normally.
• New case in worker_api_server_test.rs (execution_response_lands_when_clock_steps_backwards_test): the scheduler's now_fn steps back 2s before the worker's ExecuteResult arrives; asserts the result still reaches update_action and the worker keeps its registration. Both exercise the exact path that errored before the change.
• Observed in deployment: a self-hosted fleet hit the backwards-step on its scheduler host; workers' finished results were dropped and operations hung in Executing until timeout.

Risk

Low and contained to the worker-liveness path. The behavior change: a backwards clock step (previously an error that could cost a worker its message) is tolerated; the keepalive-gap metric uses saturating_sub, so a step contributes 0 rather than garbage. The scheduler-side execution timeout still runs off the retained (newer) timestamp, so no deadline gets extended.

AI assistance

An agent (Claude) drafted the fix and tests from a deployment incident I triaged; I reviewed the diff and the test assertions.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
nativelink	Ready	Preview	Oct 6, 2026 8:29pm UTC
nativelink-aidm	Ready	Preview	Oct 6, 2026 8:29pm UTC

[b7r6]

Red-team pass on this PR (adversarial review of the best-effort liveness change, specifically whether it widens what a stale/evicted worker can do):

Stale-worker verdict: the W1 → eviction → reassignment → late-W1 attack is fenced, and none of the fences involve liveness. The chain:

1. ApiWorkerScheduler::update_action requires the worker to exist in the worker map (api_worker_scheduler.rs, first lookup) — an evicted W1 fails here.
2. It then requires running_action_infos.contains_key(operation_id) — a live W1 whose operation was requeued fails here ("should not be running on worker", already covered by update_action_with_wrong_worker_id_errors_test).
3. SimpleSchedulerStateManager::inner_update_operation independently rejects a worker-id mismatch against the awaited action's assigned worker (Code::Aborted).
4. Worker IDs are per-connection UUIDv6, so a reconnect can never alias an old identity (no ABA on worker identity).

touch_liveness never gated any of these — it only refreshed a timestamp — so making it best-effort swallows the refresh error without widening write authority. The one write that moved ahead of the fence, record_action_resource_usage, is itself gated per-operation on worker-in-map + operation-in-running-set, and the origin event on running_action_info; a stale worker contributes at most global usage histograms for work it genuinely executed.

Clock verdict: the fix handles backward steps (equal timestamps were already accepted; max() latches the newest; the keepalive-gap metric uses saturating_sub so a step records 0, and the eviction threshold uses saturating_sub so there is no underflow). A backward step of N seconds delays that worker's timeout eviction by at most N — bounded and benign. Forward jumps are unchanged by this PR: liveness is wall-clock-based, so a large forward step can still make workers look stale (pre-existing behavior, out of scope here).

Test gap found and closed: with the fix in place, the original clock-step test no longer exercises a failing touch_liveness — so it didn't establish that swallowing a non-clock failure is safe. Added two deterministic tests (mock clocks, no sleeps/races):

• late_result_from_evicted_worker_cannot_touch_reassigned_attempt — full W1→evict→W2 interleaving against a real SimpleScheduler; W1's late result (distinct exit code) is refused and the client observes only W2's result.
• evicted_worker_late_result_never_reaches_state_manager_test — forces the swallowed-error path itself (evicted worker ⇒ refresh fails ⇒ warning) and proves the stale ExecuteResult never reaches the state manager.

Both pass locally alongside the existing suites (2/2 and 16/16).

[MarcusSorealheis]

I expected a panic and didn't get one. The agent is smarter. I think this looks good pending @rmlarsen's test.

[MarcusSorealheis]

/build-image nativelink

[MarcusSorealheis]

/build-image nativelink-worker-init

[github-actions]

Image built and pushed!

ghcr.io/TraceMachina/nativelink:2026-10-06-b6f1b9b520bdbb20ffb9118acbb2fff26e8fded4

[github-actions]

Image built and pushed!

ghcr.io/TraceMachina/nativelink-worker-init:2026-10-06-b6f1b9b520bdbb20ffb9118acbb2fff26e8fded4