feat(adk): tolerate transient heartbeat failures

[shentongmartin]

Problem

A background task currently loses its execution context after the first heartbeat storage error, even when the last confirmed lease is still valid. Retrying immediately would increase storage pressure, while continuing without fencing new side effects could let work run after ownership becomes uncertain.

Solution

Add opt-in transient-heartbeat tolerance to backgroundtask.Config:

&backgroundtask.Config{
    HeartbeatInterval:                 30 * time.Second,
    LeaseDuration:                     90 * time.Second,
    TolerateTransientHeartbeatErrors: true,
}

With the option enabled, heartbeat attempts remain single-flight and are scheduled one full HeartbeatInterval after the previous attempt finishes. A transient error keeps the attempt alive but closes an execution gate: new tool calls and version-dependent writes wait until ownership is confirmed, canceled, or timed out. Operations already admitted may finish.

The Manager tracks a conservative local lease deadline from the start of the last confirmed renewal and cancels before expiry with a safety margin of half the heartbeat interval. This watchdog remains effective even if the storage request ignores context cancellation.

On a version conflict after an uncertain response, the Manager reads authoritative task state. It accepts only the same running Attempt with exactly one version advancement as a lost successful heartbeat response. Cancellation, takeover, terminal state, expired lease, and other explicit fencing errors terminate immediately; the Manager never calls Start during reconciliation.

The zero-value behavior is unchanged: transient-heartbeat tolerance is disabled, with the existing 10-second heartbeat and 30-second lease defaults.

Key Insight

A failed heartbeat response does not prove that renewal failed. Safety requires separating confirmed lease time from response status: pause new side effects while uncertain, reconcile the same attempt on the next normal cadence, and never extend the local deadline until the renewal is proven.

Validation

• golangci-lint run --new-from-rev=alpha/10 ./...
• go test -race ./...
• go test -race -count=30 ./adk/backgroundtask -run 'TestHeartbeat|TestManagerExecutionStops|TestManagerConfigures|TestManagerRejects'
• GOTOOLCHAIN=go1.18.10 go test ./adk/backgroundtask/... ./compose ./internal/core
• go test -coverprofile=/tmp/eino2-heartbeat-tolerance.coverage.out ./... (83.0% total)

---

问题

后台任务当前遇到第一次 heartbeat 存储错误就会取消执行，即使最后一次确认的租约仍然有效。立即重试会增加存储压力；如果不限制新的副作用，则可能在所有权不确定后继续执行任务。

解决方案

在 backgroundtask.Config 中增加显式开启的临时 heartbeat 错误容忍能力：

&backgroundtask.Config{
    HeartbeatInterval:                 30 * time.Second,
    LeaseDuration:                     90 * time.Second,
    TolerateTransientHeartbeatErrors: true,
}

开启后，heartbeat 保持单飞，并在上一次请求结束一个完整 HeartbeatInterval 后再发起下一次请求。临时错误不会立即取消 attempt，但会关闭 execution gate：新的工具调用和依赖版本号的写入等待租约确认、取消或超时；已经准入的操作可以正常返回。

Manager 从最后一次确认续租的请求开始时间计算保守的本地租约截止时间，并在租约过期前预留半个 heartbeat interval 的安全余量后取消执行。即使存储请求不响应 context 取消，本地 watchdog 仍会生效。

不确定状态下再次续租遇到版本冲突时，Manager 会读取权威任务状态。只有仍处于 running、Attempt 相同且版本恰好推进一次时，才认定上次续租已提交但响应丢失。明确取消、接管、终态、租约失效及其他 fencing 错误仍立即终止；对账过程不会调用 Start 或复活任务。

默认行为保持不变：未显式开启容错时，第一次 heartbeat 错误仍会取消执行，heartbeat 和租约默认值仍分别为 10 秒和 30 秒。

关键认知

heartbeat 响应失败不等于续租提交失败。安全做法是将“最后确认的租约时间”与“本次响应状态”分开：不确定期间暂停新副作用，按正常周期确认同一 attempt，并且在续租被证明成功前绝不延长本地截止时间。

验证

• golangci-lint run --new-from-rev=alpha/10 ./...
• go test -race ./...
• go test -race -count=30 ./adk/backgroundtask -run 'TestHeartbeat|TestManagerExecutionStops|TestManagerConfigures|TestManagerRejects'
• GOTOOLCHAIN=go1.18.10 go test ./adk/backgroundtask/... ./compose ./internal/core
• go test -coverprofile=/tmp/eino2-heartbeat-tolerance.coverage.out ./...（总覆盖率 83.0%）

[codecov]

Codecov Report

❌ Patch coverage is 74.94647% with 117 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (alpha/10@11ed2ab). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
adk/backgroundtask/executor.go	73.20%	88 Missing and 24 partials ⚠️
compose/tool_node.go	37.50%	3 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             alpha/10    #1273   +/-   ##
===========================================
  Coverage            ?   82.08%           
===========================================
  Files               ?      220           
  Lines               ?    37830           
  Branches            ?        0           
===========================================
  Hits                ?    31053           
  Misses              ?     4551           
  Partials            ?     2226           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.