workflow-world-celld

A Workflow SDK world backed by celld — self-hosted, distributed Cloudflare Durable Objects. It makes eve / Workflow SDK apps durable on your own infrastructure: every namespace's workflow state lives in one celld cell (a private SQLite database, continuously replicated to an S3-compatible bucket, owned by exactly one node at a time).

Passes the full @workflow/world-testing conformance suite (12/12).

Architecture

┌─────────────────────────────┐            ┌───────────────────────────────┐
│ App / eve (Node, Nitro)     │            │ celld fleet                   │
│                             │  RPC over  │                               │
│ @wirkflow/workflow-world-   │  /cell/    │ WorkflowCell:<ns> (DO)        │
│ celld  (thin client)        ├───────────►│  • world-local storage engine │
│                             │            │    on a SQLite virtual fs     │
│ /.well-known/workflow/v1/   │◄───────────┤  • stream chunk store         │
│ flow  (queue handler)       │  alarm-    │  • alarm-driven queue         │
└─────────────────────────────┘  driven    └───────────────┬───────────────┘
                                 delivery                  │ replication
                                                    S3-compatible bucket

Requires celld v0.2+ (the v0.1 ingress and bucket format are gone; see Upgrading).

• packages/world — @wirkflow/workflow-world-celld, the Node-side createWorld() client. Proxies the World interface (runs, steps, events, hooks, streams) to the cell over /cell/<name>?op=rpc on any node's public listener — the bundle's stateless worker forwards it to the Durable Object binding, and that dispatch redirects to the owning node — and implements createQueueHandler for the deliveries the cell makes back into the app (world-local wire protocol: x-vqs-* headers).
• packages/cell — the Worker bundle deployed to celld. The Durable Object runs @workflow/world-local's storage + streamer engine unchanged on top of a virtual node:fs backed by the cell's SQLite (ctx.storage.sql), plus a durable queue that delivers due messages from the DO alarm via outbound fetch.
• probe/ — a tiny worker that verifies the celld runtime assumptions (re-entrant requests during an awaited outbound fetch, storage.sql, alarms, outbound fetch from alarms).

Design notes

• One cell per namespace. The namespace defaults to WORKFLOW_CELLD_NAMESPACE, falling back to WORKFLOW_LOCAL_DATA_DIR (which @workflow/world-testing sets uniquely per test server).
• The RPC protocol selects operations with ?op=rpc + a JSON body, never sub-paths, so the cell-facing URL shape stays flat.
• World RPCs carry the caller's W3C traceparent (a no-op unless the app registers an OpenTelemetry SDK), and the queue handler resumes the trace context the cell's delivery carries — one workflow hop is one trace: app → cell → alarm → app. See Telemetry.
• Queue semantics mirror world-local: stable message ids across redeliveries (step-ownership lease), {timeoutSeconds} re-scheduling, 5s retry backoff, 256-delivery cap — but the scheduler is the DO alarm, so pending work survives process and node restarts and lands on whichever node owns the cell.
• streams.get live-follows via getChunks polling instead of holding a streaming response through the ingress.

Local development

Requirements: bun, docker (for minio), a built celld binary — v0.2.0 or later (cargo build --release in the celld repo; set CELLD_BIN, default /tmp/celld/target/release/celld), and esbuild (installed as a dev dep).

bun install
bun scripts/fleet.ts up      # minio + bucket + build + deploy + celld node
bun run test                 # @workflow/world-testing conformance suite
bun scripts/fleet.ts deploy  # rebuild + redeploy after cell changes
bun scripts/fleet.ts down

The dev node listens on 127.0.0.1:8420 (public: worker routes + health) and 127.0.0.1:8425 (internal: operator API + peer traffic). deploy restarts the node through POST /shutdown?handoff=preserve, so the replacement resumes the local cell databases without a bucket restore. Telemetry is on (CELLD_OTEL=1); traces and logs land in the minio bucket under telemetry/.

Using with eve

// agent/agent.ts
export default defineAgent({
  experimental: {
    workflow: { world: "@wirkflow/workflow-world-celld" },
  },
});

Environment for the eve process:

Variable	Meaning
WORKFLOW_CELLD_NODE_URL	celld node base URL (default http://127.0.0.1:8420)
WORKFLOW_CELLD_NAMESPACE	namespace → cell identity
WORKFLOW_CELLD_BASE_URL	origin of the app for queue deliveries (e.g. http://localhost:3000)
WORKFLOW_CELLD_RECOVER_ACTIVE_RUNS	0 disables re-enqueueing active runs on start

Note: eve pins its vendored @workflow/* protocol line (currently 5.0.0-beta); keep this package's @workflow/world dependency on the same line as the eve release you deploy.

Telemetry

celld v0.2 writes OpenTelemetry traces and logs as Parquet files straight into the fleet bucket (CELLD_OTEL=1, on in the dev fleet) — no collector needed. Query them with duckdb:

INSTALL httpfs; LOAD httpfs;
CREATE SECRET minio (TYPE s3, KEY_ID 'minioadmin', SECRET 'minioadmin',
  ENDPOINT '127.0.0.1:9000', URL_STYLE 'path', USE_SSL false);
CREATE VIEW traces AS SELECT * FROM
  read_parquet('s3://celld-dev/telemetry/traces/*/*/*/*/*/*.parquet');
CREATE VIEW logs AS SELECT * FROM
  read_parquet('s3://celld-dev/telemetry/logs/*/*/*/*/*/*.parquet');
-- slowest cell events
SELECT name, duration_us, trace_id FROM traces
  ORDER BY duration_us DESC LIMIT 20;

Spans cover each cell event (fetch, alarm), each outbound fetch (the queue deliveries), queue waits and durability facts; cell console.log output becomes log records tagged with trace and span ids. When the app registers an OpenTelemetry SDK, the world client's traceparent makes all of that join the app's own traces. For an OTLP collector instead of the bucket, set CELLD_OTEL_SINK=otlp.

Upgrading celld

v0.2 cannot coexist with v0.1: stop every v0.1 node, then start the v0.2 nodes (rolling updates fail — v0.1 peers cannot follow v0.2 ownership records, and v0.1 readers cannot restore v0.2's compacted block objects). This adapter's v0.2 line also moved the RPC ingress from /do/ (now on the internal listener) to the worker route /cell/<name>, so deploy the current cell bundle together with the node upgrade.

Security

celld does not terminate TLS, and both listeners are unauthenticated: the public listener serves the worker routes (and with them every cell), and the internal listener carries the operator API (/state, /shutdown) and all peer traffic. Run the fleet and the app on a trusted private network (or WireGuard/Tailscale), never expose either listener publicly, and keep the internal listener reachable for the other nodes only. celld is alpha software — treat the bucket credentials as fleet-admin access.

Performance

Measured with bun scripts/load-test.ts on a local fleet (one celld node, minio as the bucket), Apple Silicon, against celld v0.2:

• Durable event writes: ~63 ops/s sequential per cell (one bucket round trip each, ~17 ms), ~750 ops/s at concurrency 20 on one cell — celld coalesces concurrent writes to a cell into shared uploads — and ~220 ops/s spread over 40 cells at concurrency 40.
• Reads (runs.get): ~16,000 ops/s, p50 ≈ 1 ms (resident cell, no bucket traffic).
• End-to-end workflow runs (single-step workflow through the real runtime, one namespace): ~33 runs/s at concurrency 10, ~55 runs/s at concurrency 25–50, p99 ≤ 1.3 s under sustained load.

Notes from load testing:

• Right after a node cold-starts, a bucket hiccup can hold a cell's responses for up to 30 s (celld's S3 retry timeout; the output gate releases every held response at once). Warm fleets did not show this.
• A busy cell's resident memory grows with its history (roughly 1 MB per workflow run kept); there is no retention/cleanup yet — same as world-local's on-disk store, but it surfaces as node RSS here. celld v0.2 cuts the fixed per-cell overhead (~3.4 MB → ~0.5 MB, shared isolates), so the history, not the runtime, dominates.

Durability and failure behavior

Verified by bun scripts/chaos-test.ts (failure injection: app crash, node kill, bucket outage, two-node failover, hostile input, oversized payloads, idempotency races):

• App-server crash: in-flight runs resume via durable queue redelivery when an instance comes back on the same flow URL (10/10 in the chaos suite).
• celld node kill: cells restore from the bucket, durable alarms re-fire, and in-flight runs complete (10/10) — even on a SIGKILL immediately after a write burst. celld v0.2 replicates the re-entrant writes flow handlers make during queue deliveries (the v0.1 loss window is fixed upstream, see docs/celld-findings.md), and a SIGTERM drain flushes pending state before exit. A restart on the same address serves again within seconds.
• Bucket outage: the node self-fences and halts (by design). Run celld under a supervisor; state from before the outage survives, new work is refused while the bucket is down (output gate) and proceeds after restart.
• Node failover: with WORKFLOW_CELLD_NODE_URL=nodeA,nodeB, the client rides through the loss of the owning node. A graceful stop hands cells to the peer immediately (zero client-visible errors in the chaos suite); a hard node loss waits for celld's dead-node detection first. Retries are time-budgeted (WORKFLOW_CELLD_RETRY_MS, default 30 s — raise it for multi-node fleets).
• Payloads: values above the ~2 MB SQLite binding limit are chunked transparently; 4 MB event inputs and stream chunks round-trip.

Known limitations

• streams.get polls (~30 ms interval); a hibernatable-WebSocket push channel would cut stream latency.
• The queue delivers from the cell to one app base URL captured at enqueue time. Multi-instance apps should put the flow endpoint behind a stable internal URL.
• Upstream: a concurrent cold-start burst against one cell can rarely execute on two activation lineages, which can duplicate an idempotent enqueue (~1 in 10 bursts in the chaos idem scenario; see docs/celld-findings.md). The queue's unique index absorbs every same-database race.

License

Apache-2.0. The deployed cell bundle embeds the @workflow/world-local storage engine, which is likewise Apache-2.0.