Skip to content

ops: an unconfigured republisher emits no walk signal, so a skipped walk is invisible until names expire #1229

Description

@FSM1

An unconfigured republisher is invisible to every operational signal it owns. The
boot report added in #1228 makes the misconfiguration visible at startup, but the
walk still reports nothing, and the walk is the thing whose failure costs vault
availability.

Mechanism

RepublisherTask.runOnce returns before it can emit anything:

async runOnce(): Promise<void> {
  // No routing endpoint (BYO-only deploy) → the walk can neither resolve nor
  // re-PUT; skip it rather than fire a resolve-failure alert for every name.
  if (!this.transport.configured) {
    return;
  }
  ...
  this.alerter.walkComplete(names.length, republished);
}

apps/api/src/republisher/republisher.task.ts:76-94. The early return precedes
alerter.walkComplete, alerter.staleRepublish, and walkAll, so an unconfigured
republisher produces no error, no warning, and no counter movement.

walkComplete is the only path that writes the walk gauges —
observeRepublisherWalk sets republisherLastWalkNames and
republisherLastWalkRepublished (apps/api/src/ops/metrics.service.ts:67-70).
Unset, those gauges never move, so a dashboard cannot distinguish "the republisher
is not configured" from "the republisher has not walked yet".

The failure is silent until IPNS records reach their 90-day EOL and names stop
resolving, at which point it is a user-visible availability loss with no prior
signal.

Why the boot report is not sufficient

RoutingV1RecordTransport now logs ROUTING_V1_URL is unset; the republisher walk will not run at construction (apps/api/src/republisher/record-transport.ts). That
catches a misconfigured deploy at startup, which is the common case. It does not
catch the walk being absent for any other reason, and a single boot line is not
something an operator can alert on the way they can alert on a gauge.

Options considered, and why each was not built

Emit walkComplete(0, 0) on the unconfigured path. Contained in apps/api, but
it makes the gauges report a completed walk of zero names — indistinguishable from a
configured republisher with zero registered names. That manufactures a healthy
signal for an unhealthy state, which is worse than the current silence.

Throw on the unconfigured path. WorkerScheduler catches task rejections and
logs sweep failed (apps/api/src/common/worker-scheduler.ts:120-124), so this
would surface. But an absent routing endpoint is an explicitly supported deployment
shape — record-transport.ts documents "unit deploys, BYO-only" and the early
return exists precisely to avoid alert-storming those deploys. This turns a
supported configuration into a recurring error, the same mistake avoided for
KUBO_API_URL in #1228.

A distinct signal. The correct fix: a dedicated indicator an operator can alert
on separately from a zero-count walk — a republisher_configured gauge, or a
walk-outcome enum distinguishing completed / skipped-unconfigured. This is an
addition to the RepublisherAlerter contract
(apps/api/src/republisher/republisher.alerter.ts) and to MetricsService, and it
changes what staging dashboards and any alert rules receive. That is an operational
contract decision, so it is filed here rather than guessed at inside a config-wiring
PR.

Wanted

  • A signal that distinguishes "republisher not configured / walk skipped" from
    "walk completed with nothing to do", exposed where an operator can alert on it.
  • A decision on whether the skip should also be visible in /metrics as a gauge or
    as a walk-outcome label, since that determines the alert rule.
  • Coverage in apps/api/src/republisher/republisher.task.test.ts pinning that the
    unconfigured path emits the new signal and still does not alert per name.

Metadata

Metadata

Assignees

No one assigned

    Labels

    comp:apiapps/api — NestJS residual surface, registry, mailbox, republisherv2-buildv2 rewrite build slice

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions