fix(backend): recover panics in gRPC handlers instead of crashing the daemon

[ericvicenti]

Follow-up to #909 / #910, as suggested in juligasa's review. Those PRs fixed one panic; this closes the class.

The problem

grpc-go does not recover handler panics, and neither of our servers installs an interceptor that does — daemon.go chains only otel + Prometheus, hmnet.go only the metrics interceptors. So any panic reachable from request-handling code kills the whole process instead of failing one call.

That matters here because both servers parse data that originated somewhere else:

• the user-facing API serves documents fetched from the network,
• the p2p server applies blobs pushed by arbitrary peers.

So a panic("BUG: this can never happen") deep in that code is not an assertion about our own state — it's an assertion about someone else's bytes, and they get to decide whether it holds.

We already paid for this. The op-ID index overflow fixed in #909/#910 was exactly that shape: a real imported document (the Spanish civil code, a 6,411-block move op) crossed a limit our own writer could never reach, and daemon_gateway panicked 7 times in 72 hours through plain GetDocument calls. A public URL was, in effect, a restart button.

The fix

A small backend/util/grpcrecovery package wrapping go-grpc-middleware/v2/interceptors/recovery, installed on both servers.

Ordering — recovery goes innermost (last in each chain). grpc.ChainUnaryInterceptor appends, so the last interceptor added is the one closest to the handler. Recovering there means the outer metrics/tracing interceptors see an ordinary Internal error and record the failed RPC; recovering outermost would let the panic unwind through them and the call would go unaccounted for.

What goes where. The panic value, the stack, and the method name go to the log. The caller gets a bare codes.Internal — notably not the library's default handler, which returns the entire stack trace to the caller as the status message.

Wiring point. On the daemon server it's inside initGRPC rather than at the cmd/seed-daemon call site, so every embedder is covered — desktop app and tests included, not just the CLI binary.

Tests

• End-to-end over bufconn: a panicking handler yields Internal, the panic value and stack do not appear in the client-visible error, the log entry carries the method name, and — the actual point — a second call on the same connection still succeeds.
• The stack assertion is deliberate: debug.Stack() is called from inside the deferred recover, and the test pins that it still walks back past the runtime's panic( frame into the handler. Without that it would only show the interceptor and the panic site would be lost.
• Direct unit test for the stream interceptor.

Notes

• go-grpc-middleware/v2 was already in go.mod as an indirect dep; this only promotes it to direct. No version change, go.sum untouched.
• This is a net, not a cure. A recovered panic still means a broken request and a real bug to fix at the source — a document that panics is still unreadable, just non-fatally. The goal is only that finding the next one costs an RPC rather than the node.
• Verified locally: daemon, hmnet, and the new package pass (incl. -race), and golangci-lint --new-from-rev=origin/main ./backend/... reports 0 issues.

🤖 Generated with Claude Code