fix: retry backend init with backoff so indexer connection failures self-heal

[pcfreak30]

Summary

If the indexer connection fails during startup, the backend initialization (siastorage SDK CheckAppAuth / initSDK / OptimalDataSize inside factory.Init) failed permanently, setting backend status to error and never retrying — so the operator had to restart the server to recover. This PR makes async backend init self-heal with exponential backoff.

Changes

• internal/backend/manager.go
  • Retry transient init failures with exponential backoff (5s base → 5min cap) until success, a terminal error, or context cancel.
  • isTerminalInitError classifies config/precondition errors (no app key set, no access keys, failed to open database) as non-retryable — retrying can't fix them.
  • New attemptInit runs one init attempt and closes the store on failure (no handle leak) without recording failInit; the loop decides retry vs. fail.
  • Context cancellation stops retrying without firing onFailure, so a clean shutdown doesn't reset onboarding state.
  • retryDelay is an injectable Manager field (defaults to retryBackoff) for fast, deterministic tests.
• internal/backend/manager_test.go
  • Updated async-failure tests to the new retry semantics.
  • Added SelfHeals (2 failures then success via backoff) and RetryBackoffBounds tests.

During retries, status stays starting (surfaced via SSE), so the dashboard shows startup in progress rather than a terminal error.

Testing

• go build ./... ✅
• go vet ./... ✅
• gofmt -l clean on changed files ✅
• go test ./internal/backend/... ✅ (all pass, incl. new tests)

Pre-existing failures in internal/config and internal/store (permission-simulation tests that can't fail when run as root) are unrelated to this change and verified to fail on develop too.

---

Summary

This pull request implements automatic retry with exponential backoff for backend initialization failures, allowing the system to self-heal from transient indexer/connection failures at startup without requiring manual intervention.

Key Changes

Retry Mechanism with Backoff

• Added retry logic: Backend initialization failures are now retried automatically with exponential backoff
• Backoff schedule: Starts at 5 seconds and doubles with each retry attempt, capped at 5 minutes maximum
• Self-healing: Transient failures (e.g., unreachable indexer, connection refused) are automatically retried until successful, eliminating the need for manual restarts

Error Classification

• Terminal errors: Certain configuration/precondition errors are identified as non-retryable (e.g., "no app key set", "no access keys", "failed to open database", "backend init cancelled")
• Retryable errors: All other errors (network issues, connection failures, etc.) are treated as transient and retried automatically

Lifecycle Improvements

• Context cancellation handling: Cleanly stops retry attempts on shutdown without incorrectly triggering failure callbacks
• Status preservation: Status remains "starting" during retry attempts rather than transitioning to an error state
• Improved state management: Refactored initialization into a reusable attemptInit function that handles store lifecycle and cleanup consistently

Test Coverage

• Added tests verifying self-healing behavior with retryable failures
• Updated existing tests to reflect new error classification logic
• Added tests for backoff bounds validation (base delay, exponential growth, and maximum cap)

[kody-ai]

Found critical issues please review the requested changes

• InitAfterOnboardingAsync retry loop reuses a store closed by the first failed attempt, causing an unbounded retry that can never succeed.

[kody-ai]

Found critical issues please review the requested changes

• Panic inside swapBackend leaves manager reporting Running with a closed store.
• runAsyncInit's non-deferred restartMu.Unlock is skipped when attemptInit panics, permanently deadlocking restartMu and all subsequent init/restart calls.

[kody-ai]

Found critical issues please review the requested changes

• Async init retry loop fires stale onFailure (wiping onboarding/keys) after a concurrent successful Restart because the terminal/budget branches lack the stillStarting guard the panic branch has.

[pcfreak30]

Thanks @kody. Addressed the stale-onFailure finding in d0b407a.

The terminal-error and retry-budget-exhausted branches in runAsyncInit now check m.stillStarting() (reading m.starting under m.mu) before calling failInit(..., onFailure), matching the guard the panic branch already had. If a concurrent Restart() already brought the backend to Running (clearing starting), the loop exits without firing onFailure, so onboarding access keys / ResetToAppKeySet() are never wiped by a stale failure.

Added regression test TestManager_InitFromConfigAsync_StaleTerminal_DoesNotFireOnFailure: first attempt fails transiently and parks in backoff (gated via retryDelay), a concurrent swapBackend brings the backend to Running, then the loop's next terminal attempt must not fire onFailure.

Note: this same push (d0b407a) also includes the earlier isTerminalInitError fix treating SQLite lock/busy as transient (from eae3a86). Full backend suite passes with -race; build and vet clean.

@kody start-review

[kody-ai]

The new code (lines 593-655 in the diff) does not introduce new GORM queries, but the retry logic around the database operations (OpenDatabase, initCore) could benefit from explicit context timeout handling. However, the specific rule requires GORM queries needing .WithContext(ctx) — this PR doesn't add any GORM queries in the changed lines, and the existing GORM queries are in other files outside this diff. Therefore, no violation of rule 1 is introduced by this PR.

Kody rule violation: Disallow GORM queries without timeout

[kody-ai]

Found critical issues please review the requested changes

• failInitIfStarting fires the destructive onFailure callback after releasing the lock, leaving a TOCTOU window where a concurrent Restart can bring the backend to Running and get its access keys wiped.
• failInitIfStarting fires the destructive onboarding onFailure outside the lock after releasing it, allowing a concurrent successful Restart to be wiped by a stale failure callback.

[kody-ai]

The retry loop in runAsyncInit passes initCtx (a context.WithCancel from main.go with no timeout) through attemptInit/initCore to factory.Init, so GORM database operations lack a timeout and can run indefinitely. Derive a per-attempt timed context with context.WithTimeout or context.WithDeadline before calling attemptInit/factory.Init, since GORM queries require a timeout unless inside a db.RetryableComponentLock call.

Kody rule violation: Disallow GORM queries without timeout

[kody-ai]

Found critical issues please review the requested changes

• recordInitFailure invokes the destructive onboarding reset outside the lock, leaving a window where a concurrent successful Restart can still have access keys wiped.

[kody-ai]

Kody Review Complete

Great news! 🎉
No issues were found that match your current review configurations.

Keep up the excellent work! 🚀

Kody Guide: Usage and Configuration

Interacting with Kody

• Request a Review: Ask Kody to review your PR manually by adding a comment with the @kody start-review command at the root of your PR.

• Validate Business Logic: Ask Kody to validate your code against business rules by adding a comment with the @kody -v business-logic command.

• Provide Feedback: Help Kody learn and improve by reacting to its comments with a 👍 for helpful suggestions or a 👎 if improvements are needed.

Current Kody Configuration

Review Options

The following review options are enabled or disabled:

Options	Enabled
Bug	✅
Performance	✅
Security	✅
Business Logic	✅

Access your configuration settings here.

​

[kody-ai]

New GORM database operations in internal/backend/manager.go — validateAccessKeysExist and ListAccessKeys within the syncInit/attemptInit retry paths — lack .WithContext(ctx) timeouts, allowing queries to run indefinitely when the store reopens. Wrap all new GORM calls with .WithContext(ctx) to enforce timeout boundaries, except inside db.RetryableComponentLock.

Kody rule violation: Disallow GORM queries without timeout

[github-actions]

Docker image for testing

Pull the image for this PR:

docker pull ghcr.io/lumeweb/s3-server:sha-ab005c8

Or use in docker-compose:

services:
  s3-server:
    image: ghcr.io/lumeweb/s3-server:sha-ab005c8

The image is rebuilt on every push to this PR's branch.