feat(bulk): release export/submit leases on graceful shutdown (#1531)

[smunini]

Closes #1531.

Summary

On a graceful shutdown (SIGINT/SIGTERM), the bulk export and submit workers now stop claiming, stop their current job at a safe boundary, and release the lease. Another instance can claim the job at once instead of waiting out HFS_BULK_{EXPORT,SUBMIT}_LEASE_DURATION.

Export release (SQLite, PostgreSQL)

One fenced transaction (worker_id + fencing_token, status in-progress):

• the job goes back to accepted;
• the attempt is refunded (attempts - 1, floored at 0), so rolling restarts no longer use up max_attempts;
• this run's progress and file rows are wiped, so the next attempt cannot resume from a cursor whose parts are gone (bulk-export: a job whose lease expires mid-run restarts from scratch forever, with no cap and no error #1041). As on a reclaim, artifacts are left for the TTL sweep.

release now returns bool. A zombie release after a reclaim is a no-op.

Submit release (SQLite, PostgreSQL, MongoDB, S3, composite wrappers)

Audited as the issue asked. A manifest released to pending ends up in the same state as one whose lease lapsed, and that path is already safe:

• there is no attempt counter to refund;
• the re-walk upserts idempotently and each line is charged once (bulk-submit ingest: a mid-body stream error completes the manifest with rows missing; the lease dies behind a WAL checkpoint and forces a full re-walk; Elasticsearch idles through the whole import #1127);
• byte progress is MAX-monotonic;
• artifacts are published only with the terminal state;
• the sqlite-es: the automatic rebuild after an import fails on oversized _bulk requests, leaves all 11,704 Provenance resources unsearchable (1 % cut), spends 1,551 s on its first attempt alone before retrying all 24 types, and writes a SQLite index nothing reads #1125 index-pending marker is set only after publication.

So release only gains a bool return. The S3 version no longer swallows storage errors.

Workers

• DefaultExportWorker::with_shutdown: stops between batches, or mid-read/_typeFilter search where no part is open, then releases.
• DefaultSubmitWorker::with_shutdown: a bridge task trips the existing cooperative cancel (bulk-submit: Abort cannot stop an in-flight manifest, and a failed abort is invisible #968). The wind-down paths release when the stop came from shutdown and the lease is still held. A concurrent abort wins: the release is fenced on processing.

hfs wiring

• New crates/hfs/src/worker_shutdown.rs tracks the worker loops (TaskTracker + CancellationToken).
• The shutdown signal cancels the token. After the HTTP drain, serve() waits up to HFS_WORKER_SHUTDOWN_TIMEOUT (default 20 s), then records the audit shutdown and flushes audit/OTLP. Past the deadline, leases lapse as before.
• Claims are never raced against shutdown, so a claim cannot commit and then be dropped.

Shutdown signal

Hooked into helios_observability::shutdown::signal() (SIGINT and SIGTERM, #907), so a rolling restart's SIGTERM releases leases too. The worker drain runs after the HTTP drain and before the audit/OTLP flush, including on a serve error.

Tests

• SQLite and Postgres export: claim, partial run, release, reclaim under a cap of 1 gives the same attempt count and no leftover rows. Zombie release and release-after-finish are no-ops.
• Shared tests/bulk_submit/release_contract.rs, run on SQLite, Postgres, MongoDB and S3: release re-queues; zombie release, double release and release-after-abort are no-ops.
• Worker-level shutdown tests for export and submit (inline and independent file scheduling): the next worker completes with every resource.
• worker_shutdown unit tests.

Verified locally: persistence lib tests (sqlite, s3), sqlite_tests and the SQLite bulk-submit suites, the Postgres bulk/export and Mongo bulk_submit filtered suites, hfs tests, and clippy with the CI flags.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 93.54839% with 30 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/persistence/src/core/bulk_export_worker.rs	91.37%	10 Missing ⚠️
crates/hfs/src/worker_shutdown.rs	88.88%	7 Missing ⚠️
...s/persistence/src/backends/postgres/bulk_export.rs	16.66%	5 Missing ⚠️
crates/persistence/src/core/bulk_submit_worker.rs	95.96%	5 Missing ⚠️
...tes/persistence/src/backends/sqlite/bulk_submit.rs	50.00%	1 Missing ⚠️
crates/persistence/src/composite/bulk_submit.rs	0.00%	1 Missing ⚠️
.../persistence/src/composite/indexing_submit_jobs.rs	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!