Research: run the agent loop on Apache Airflow as a pluggable runner

[mroops0111]

Motivation

Bootstrap runs are long, expensive, and rate-limited. Today the harness owns scheduling, retry and resume itself, and the semantics are not strong enough:

• a provider rate limit mid-run does not stop the batch, it cascades into a run of repeated failures
• resume granularity is unclear. It is not obvious whether resuming continues a half-finished session or restarts it
• there is no durable, inspectable record of what ran, what failed, and what was retried

These are solved problems in a workflow engine. Worth evaluating whether to delegate them rather than keep hand-rolling.

Proposal

Evaluate Apache Airflow as an alternative execution backend, exposed as a pluggable runner alongside the existing in-process one.

Non-goal

Making Airflow a hard dependency. composeFsApp must keep working with zero external infrastructure. Airflow is opt-in, in the same spirit as the existing ontology and source-loader plugins.

Cross-language is not the blocker

Airflow orchestrates processes, not in-process calls. We already ship a Dockerfile and a compose.yaml, so DockerOperator / KubernetesPodOperator running the braid CLI, or plain HTTP against the server, covers the boundary. TypeScript vs Python is not the hard part.

Open questions, the actual research

1. Task granularity. What maps to a single Airflow task: a source, a module, one agent call? Too coarse and a rate limit still destroys the batch, which is the problem being solved. Too fine and scheduler overhead dominates the work.
2. State ownership. braid owns the graph and the Git-versioned history. Airflow must not become a second source of truth. XCom should carry ids and pointers, never payloads.
3. Idempotency. Is extract safe to re-run? If a retried task can emit duplicate proposals, Airflow retries cannot be trusted and the whole premise falls apart.
4. Human-in-the-loop gates. A run pauses until a person decides, sometimes for days. Deferrable operators and triggers look like the right primitive because they release the worker while waiting, but this needs verification. Weakest area, and the most likely to need work upstream.
5. Rate limits. Does Airflow's retry honour Retry-After on a 429, or only its own backoff policy? If only the latter, a provider-aware retry policy is needed.

Next step

Spike: run one PRD bootstrap end to end against a local Airflow, one task per module, and record where it hurts. Decide from the friction, not from the design doc.