[P2] Transient atelet unavailability defaults to codes.Internal — outside the documented retryable set; strands actor in SUSPENDING and surfaces as http/500

[aditya-shantanu]

Dedup note: verified against all open+closed issues 2026-07-30 — nothing
files the Internal-misclassification of transient infra conditions or the
SUSPENDING-strand scenario. Closest relatives: #605 (inverse
classification bug: terminal ENOSPC treated as retriable), #50 (same
SUSPENDING symptom, different root cause — runsc exit 128, retry does NOT
converge there), #554 (reentrancy umbrella), #600 (precedent for the
"Terminating pod has no IP" window, worker-pod side).

Summary

When the atelet DaemonSet pod on an actor's node is being replaced (routine
during node upgrades), SuspendActor fails with:

rpc error: code = Internal desc = internal server error: workflow failed at
step CallAteletSuspend: while getting atelet conn for worker pod: selected
atelet "ate-system/atelet-w78v7" has no assigned IPs

Mechanism (verified on main):

• cmd/ateapi/internal/controlapi/dialer.go:112 produces the error as a
  plain fmt.Errorf — no gRPC code attached.
• workflow_suspend.go:140 wraps it with %w — still no code.
• internal/ateinterceptors/ateinterceptors.go:72 then defaults every
  non-status error to codes.Internal "internal server error".

Why the code matters — this is not a style nit; the system documents
contracts around specific codes:

1. docs/request-parking.md: the router parks/retries only
   FailedPrecondition, Aborted, Unavailable; everything else is
   fail-fast. A self-healing ~10 s condition lands outside the retryable
   set.
2. Router HTTP mapping (PR fix(router): map gRPC errors to proper HTTP codes (#31) #68): Unavailable/FailedPrecondition → 503,
   "anything else" → 500. So Internal-classified churn errors surface
   to HTTP callers as 500s — which matches the 1,840 unexplained
   http/500 responses in the 2026-07-30 overnight soak (during
   ateapi/atelet chaos kills).

Live reproduction (GKE substrate-poc, 2026-07-30)

kubectl delete pod -n ate-system <atelet-on-actor-node> --grace-period=0,
then SuspendActor on the RUNNING actor hosted there ~0.5 s later:

• SuspendActor returned the Internal error above.
• Actor left in STATUS_SUSPENDING. In-cluster HTTP probe went from steady
  200s to hanging until its 5 s client timeout — consistent with requests
  being parked on the resume path's FailedPrecondition (SUSPENDING is not
  resumable) with nothing repairing the state, rather than a fast 503.
• A retried SuspendActor after the replacement atelet got its IP converged
  cleanly (SUSPENDED → auto-resume on next request). Forward recovery
  works; the defect is purely that the error classification tells clients
  not to retry.

Suggested fix

• In the dialer / workflow step, classify "no atelet conn / atelet has no
  assigned IPs / atelet not found" as codes.Unavailable.
• Audit other plain fmt.Errorf paths in workflow steps that hit the
  ateinterceptors.go:72 Internal default for infrastructure-transient
  conditions (the [P1] ENOSPC not classified as a terminal filesystem error — disk-full causes infinite retry loop #605 ENOSPC bug is the same audit from the opposite
  direction: both directions of misclassification exist).
• Router: ensure upstream Unavailable during wake maps to 503 +
  Retry-After, never a park-until-timeout hang.