feat: hflow serve — an HTTP API over a workspace, plus corpus-identity fixes

.gitignore

@@ -13,6 +13,15 @@ __pycache__/
 # never source).
 data/
 
+# Frontend build outputs: the source of truth is ui/; built assets are copied
+# into the hflow-server wheel at packaging time, never committed.
+ui/node_modules/
+ui/dist/
+packages/hflow-server/src/hflow_server/static/
+
 # Local maintainer tooling (agent skills, settings); not part of the public repo.
 .claude/
 .agents/
+
+# Transient: `pnpm gen:api` dumps the schema here on its way to src/apiSchema.ts.
+.openapi.json

CONTRIBUTING.md

@@ -114,6 +114,26 @@ repository root, and name the observable result. Keep examples on public APIs;
 tests belong to business logic and boundary behavior, not to checking that a
 documentation snippet copied a third-party SDK correctly.
 
+## Changing how episodes are processed
+
+Identities in HFlow are content hashes, and one of them -- `pipeline_version`
+-- is stamped inside the canonical bytes that `episode_id` hashes. A release
+number deliberately does **not** feed any of them: a CLI fix or a docs bump
+must never invalidate somebody's corpus. What does feed them is
+`TRANSFORM_BEHAVIOR_VERSION` in [`src/hflow/behavior.py`](./src/hflow/behavior.py).
+
+**Bump it in the same commit whenever your change makes the transform write
+different bytes for the same input** -- encoder settings or defaults,
+chunking and grouping, timestamp handling, the provenance record's shape, or
+a bugfix to any of those. Bumping re-versions every existing corpus exactly
+once, which is the honest cost; not bumping when behavior changed silently
+mixes two behaviors under one version, which is worse. When in doubt, bump,
+and say so in the pull request.
+
+Changes to checks, enrichments, or anything a step merely calls do not need a
+bump: a step's own content hash already covers its source and captured
+configuration. `tests/test_identity_stability.py` pins these rules.
+
 ## Quality checks
 
 Run the Python quality gate and fix every reported issue:

docs/ARCHITECTURE.md

@@ -44,7 +44,7 @@ that is actually scheduled.
 ## Design tenets
 
 1. **Evidence, not verdicts.** Quality checks record measurements, intervals, and tags. Pass/fail policy belongs to the consumer at curation time, never hardcoded into the corpus.
-2. **Standard formats at every boundary; no new UIs.** Episodes are standard MCAP (Foxglove/Rerun open them), runs are standard Airflow DAGs (Airflow's UI shows them), the catalog and manifests are Parquet (DuckDB/pandas/anything reads them). We ship no UI and hide nothing; the system is extensible without touching our code.
+2. **Standard formats at every boundary; no captive UIs.** Episodes are standard MCAP (Foxglove/Rerun open them), runs are standard Airflow DAGs (Airflow's UI shows them), the catalog and manifests are Parquet (DuckDB/pandas/anything reads them). The optional workspace UI (`hflow serve`, shipped separately as `hflow-server`) is a strict client of these same open surfaces through a documented JSON API -- it hides nothing, gates nothing, and everything it shows stays reachable without it. The system is extensible without touching our code.
 3. **Your code stays your code.** Transformations, checks, and enrichments are plain Python functions in the user's own environment. Existing processing code plugs in through small adapters rather than being rebuilt inside a framework.
 4. **Ship code only where it earns its place.** Either the canonical format forces bridging (video lives in-band; nothing can read it without our accessors) or the code encodes a painfully-rediscoverable pitfall. We ship no client wrappers around things users already know (`openai`, `subprocess.run(["ffmpeg", ...])`); the examples are the documentation.
 5. **Coarse-grained steps.** One task processes one episode or one batch and runs for seconds to minutes. Hot loops live inside a task, never across tasks.

docs/FORMAT.md

@@ -16,7 +16,7 @@ The single overriding rule: **a canonical episode is spec-conforming MCAP.** Eve
 
 - MCAP magic, `Header`, data section, `DataEnd`, summary section, `Footer`, closing magic, per the [MCAP spec](https://mcap.dev/spec).
 - `Header.profile` is the empty string `""` (the file mixes protobuf video channels with pass-through channels of arbitrary encoding, so no single profile applies).
-- `Header.library` is informational only (e.g. `hflow/0.2.0 episode-format/1`). No reader may key behavior off it (see [Identifier rules](#identifier-rules)).
+- `Header.library` is informational only (e.g. `hflow episode-format/1 transform-behavior/1`). It deliberately carries no release number: the header is inside the bytes the content episode id hashes, so a release would otherwise give a byte-identical input a new identity. No reader may key behavior off it (see [Identifier rules](#identifier-rules)).
 - Chunks are compressed with **zstd** by default (`"none"` is permitted). Each `Chunk` record carries `uncompressed_crc`; the `Footer` carries a summary CRC.
 - The summary section repeats all `Schema` and `Channel` records and contains `Statistics`, all `ChunkIndex` records, `AttachmentIndex`/`MetadataIndex` records, and `SummaryOffset` records. A canonical episode always has a complete summary; unindexed files are not canonical.
 

docs/HOSTING.md

@@ -154,18 +154,32 @@ deployment against facts:
   store.
 - **No tenant-facing log or metrics API.** Observability is Airflow's own
   UI and task logs on the workspace.
+- **The workspace UI (`hflow serve`) authenticates nobody.** It is a local
+  developer tool bound to `127.0.0.1`, deliberately without a login; it is
+  not a tenant-facing surface, and serving it to anyone but the workspace's
+  own operator means putting an authenticating proxy in front of it. Signing
+  people in and scoping them to a workspace is the control plane's job --
+  per-user identity and revocable sessions, which no shared launch secret
+  could stand in for.
 - **Task processes share the runtime's environment**, including the
   workspace's storage credentials, and the venv build runs as root at
   provision time -- isolation between principals must come from your
   infrastructure (one workspace per trust domain), not from the engine.
 - **A workspace's Airflow stack idles at several GB of RAM** across five
   long-running services (the compose file defines seven; two are one-shot
   init containers).
-- **Engine upgrades re-version steps.** Step versions content-hash captured
-  globals, including referenced modules with their versions, so a step that
-  touches `hflow.*` gets a new version on every hflow release: `hflow
-  stale` will list its episodes, and curation pins keep working because the
-  corpus is designed to be permanently mixed-version.
+- **Engine upgrades re-version a corpus only when processing changed.** An
+  hflow release no longer moves any identity by itself: `pipeline_version`
+  folds in `hflow.behavior.TRANSFORM_BEHAVIOR_VERSION` (bumped deliberately,
+  only when the transform would write different bytes) instead of the release
+  number, the canonical file's header carries no release number, and step
+  versions record the modules they reference by name rather than by version.
+  A byte-identical input therefore keeps its `episode_id` across upgrades, so
+  content-addressed dedupe holds. The flip side is a real one: an engine
+  change that alters processing without a behavior bump is invisible to
+  `hflow stale`, so operators upgrading across a behavior bump should expect
+  exactly one corpus-wide re-version and plan reprocessing then. The corpus is
+  designed to be permanently mixed-version, so curation pins keep working.
 - **ffmpeg licensing**: the pinned build is BtbN's **GPL** variant (it
   carries the H.264 encoder the canonical transform needs). GPL source
   obligations attach to **redistribution** -- shipping worker images or

docs/README.md

@@ -44,6 +44,7 @@ the whole workflow before adapting it.
 Use these when you already know the outcome you need.
 
 - [Port existing processing and quality-check code](./PORTING.md)
+- [Serve a workspace over HTTP](./SERVE.md)
 - [Call an OpenAI vision endpoint from a step](./how-to/call-openai-vision.md)
 - [Run and operate the local Airflow runtime](./RUNTIME.md)
 - [Deploy into an existing Airflow environment](./RUNTIME.md#bring-your-own-airflow-hflow-deploy)

docs/SERVE.md

@@ -0,0 +1,139 @@
+# Serve a workspace over HTTP: `hflow serve`
+
+`hflow serve` is a local read-mostly server over one data root: it answers
+questions about episodes and their quality evidence, compiles curation SQL,
+pins manifests, monitors and triggers ingest runs, and describes the
+registered pipeline. The server never rewrites or deletes an episode -- the
+only files it writes are the manifests you pin and its own small state file.
+It can trigger an ingest run, though, which the runtime then writes through
+the normal pipeline; `--read-only` refuses that along with the other writes.
+
+**The JSON API is the product surface, not an implementation detail of some
+frontend.** Every fact a browser could show is reachable from `/api/v1`, and
+the OpenAPI schema at `/api/openapi.json` describes all of it -- so a
+workspace UI is a *client*, and you can build or swap one without touching
+this package. The server ships no frontend of its own; point
+`HFLOW_UI_ASSETS` at a directory containing an `index.html` to serve one, or
+install a wheel that packages assets under `hflow_server/static/`.
+
+It ships as a separate package, `hflow-server`, on purpose: pipeline workers
+install the `hflow` wheel into every task venv, and they should never carry a
+web server. **It is not published to PyPI yet** -- until the first release,
+run it from a clone:
+
+```bash
+git clone https://github.com/Hebbian-Robotics/hflow.git
+cd hflow
+uv sync --all-extras                 # installs hflow and hflow-server
+uv run hflow serve                      # browses $HFLOW_DATA_ROOT, else ./data
+uv run hflow serve --data-root ./data --no-browser
+```
+
+Starting the server prints its URL (`http://127.0.0.1:4356/`) and opens your
+browser. With no assets installed, that URL serves a page pointing at the
+API. There is no login: see [Trust posture](#trust-posture) for what that
+means and when it stops being appropriate.
+
+## What it shows
+
+- **Episodes** -- the corpus as a faceted, sortable table over the catalog's
+  wide `episodes` view (task, operator, status, quality measurements as
+  columns). Every filter you click compiles to DuckDB SQL server-side, and
+  the exact SQL is always visible and copyable at the bottom of the screen --
+  ready to paste into `hflow curate`.
+- **Episode** -- one recording's dossier: status and quarantine tags, contact
+  sheets, every check run with its content-hash version, measurements with
+  their producing step, intervals, tags, append history, and a canonical-MCAP
+  download.
+- **Curate** -- a SQL studio over the catalog views: schema sidebar with
+  per-table profiles, editor with run-selection, result preview with
+  per-column statistics, and the coverage report (which checks ran over how
+  much of the corpus) before you pin. **Pin manifest** freezes a query's
+  result as an immutable Parquet manifest under `<data_root>/manifests/`,
+  recorded with its SQL, row count, and coverage in the Manifests registry.
+- **Runs** -- the ingest runtime's health, recent runs with their trigger
+  configuration, per-stage activity, and a trigger form (`hflow ingest`'s
+  wire shape, as a form). It addresses a rendered local bundle or a remote
+  runtime (`HFLOW_AIRFLOW_URL` and friends); when neither is reachable the
+  page says which it looked for and why it failed, rather than disappearing.
+- **Pipeline** -- the generated DAG plus the registered steps by stage, with
+  content-hash versions, critical flags, and endpoint aliases, and the
+  versions actually observed in the catalog. Each stage's steps are drawn
+  *inside* its `process_batch` node, which is where they run: they have no
+  dependency edges on each other, so the graph nests them instead of
+  inventing a chain between them. Requires `--pipeline path/to/pipeline.py[:app]`,
+  which imports (executes) the pipeline file exactly like `hflow manifest` does.
+
+## Flags
+
+| flag | meaning |
+|---|---|
+| `--data-root` | workspace to browse (default `$HFLOW_DATA_ROOT`, else `./data`) |
+| `--host` | bind address (default `127.0.0.1`; widening past loopback exposes your corpus) |
+| `--port` | default `4356`, auto-retries upward when taken |
+| `--no-browser` | do not open a browser (headless machines, tunnels) |
+| `--read-only` | viewer mode: hides and refuses manifest pinning, saved-query edits, and run triggering |
+| `--pipeline` | pipeline file for the Pipeline page (imported once at startup) |
+
+## Nothing is UI-only
+
+The UI is a strict client of a documented JSON API (`/api/v1/...`; a running
+server publishes its OpenAPI schema at `/api/openapi.json`, ready for a client
+generator or any local OpenAPI viewer). Curation, the runs monitor and the
+pipeline page are thin calls into the same library functions the CLI uses; the
+episode listing, facets, stats and timeline endpoints compile their own
+presentation-shaped SQL over the same [catalog views](./CATALOG.md) that
+`hflow curate` reads. Either way, everything the UI can show or do is
+reachable with `curl`, scriptable, and buildable-upon. If you want a different
+frontend over your workspace, the API is the contract; the shipped UI is the
+reference client.
+
+## Trust posture
+
+**The server is unauthenticated.** There is no login, no token, and no
+session: anyone who can reach the bound address can read your whole workspace
+and trigger ingest runs. What protects it is the address it binds --
+`127.0.0.1` by default, reachable only from your own machine. This is the
+posture of every local developer tool that browses a working directory
+(`mlflow ui`, TensorBoard, `dagster dev`, the DuckDB UI): a credential in
+front of a single-user machine buys nothing but friction.
+
+Passing `--host` past loopback is therefore a deliberate exposure, and it is
+the only flag that changes who can reach the data. If you need the UI from
+another machine, forward the port over SSH (`ssh -L 4356:127.0.0.1:4356
+host`) rather than binding a network interface; if you must bind one, put a
+reverse proxy that authenticates in front of it and firewall the port itself.
+`--read-only` narrows what a reacher can *do* (no pins, no saved-query edits,
+no triggering) but not what they can *read* -- it is a safety catch, not
+access control.
+
+Hosted, multi-user HFlow is a different problem and is solved elsewhere: the
+control plane authenticates people and scopes them to workspaces
+([HOSTING.md](./HOSTING.md)). That needs per-user identity and revocable
+sessions, which one shared launch secret could never provide -- which is why
+this server does not pretend to have a piece of it.
+
+The rest of the posture is real and holds regardless. The UI runs fully
+local: all assets ship in the wheel (no CDN, no fonts, no outbound requests),
+and your data never leaves your machine. That is why the server publishes the
+schema JSON and no interactive Swagger page -- FastAPI's built-in one fetches
+its JavaScript and CSS from a public CDN, which would break the promise and
+run third-party script same-origin with your workspace's API. The browser
+never sees filesystem paths of its choosing (media is addressed by episode and
+artifact name, and the server refuses anything outside the data root), Airflow
+credentials stay server-side behind a proxy, and curation SQL runs on a
+[constrained DuckDB connection](./CATALOG.md) that cannot reach the catalog's
+files or the network. What this server writes: `<data_root>/curation/state.json`
+(saved queries and the manifest registry) and your pinned manifests -- nothing
+else. Episodes, media and catalog rows are written by the ingest runtime, on
+runs you trigger from the Runs page.
+
+## See also
+
+- [Catalog tables and curation API](./CATALOG.md) -- the views and SQL idioms
+  the Episodes and Curate screens are built on
+- [Runtime guide](./RUNTIME.md) -- the Airflow runtime the Runs screen fronts
+- [Hosting HFlow](./HOSTING.md) -- the data-plane contract for operating
+  workspaces for other people, whose seams (bucket data roots, scoped
+  credentials, constrained SQL, remote runtime addressing) are the ones this
+  UI reads through

packages/hflow-server/README.md

@@ -0,0 +1,41 @@
+# hflow-server
+
+The HFlow workspace UI: a local web app over one HFlow data root — browse
+episodes, quality evidence, and the Parquet catalog in a browser. It writes
+nothing but your pinned manifests and its own small state file.
+
+```bash
+hflow serve --data-root ./data
+```
+
+It binds `127.0.0.1` and authenticates nobody, like other local developer
+tools that browse a working directory: anyone who can reach the bound address
+can read the workspace and trigger runs, so binding past loopback is a
+deliberate exposure. `docs/SERVE.md` ("Trust posture") has the details.
+
+This package is not on PyPI yet. Until the first release, run it from a clone
+of the [repository](https://github.com/Hebbian-Robotics/hflow); `docs/SERVE.md`
+there has the exact steps, including the frontend build.
+
+The UI is a strict client of the same surfaces the `hflow` CLI uses (the
+DuckDB-queryable catalog, episode files, and manifests): everything it shows
+is reachable with `curl` against its documented JSON API, and nothing is
+UI-only. It runs fully offline — all assets ship in this wheel, and your data
+never leaves your machine. There is deliberately no Swagger page: FastAPI's
+built-in one would load its JavaScript and CSS from a CDN.
+
+Every endpoint publishes a typed response schema, so `/api/openapi.json` — the
+schema the running server serves — is a usable contract to generate a client
+from rather than a list of paths returning "object". One module —
+`hflow_server/_contract.py` — owns those payload shapes; the routes construct its
+models instead of hand-building dicts.
+
+This package is deliberately separate from the `hflow` SDK wheel so that
+pipeline worker environments (which install `hflow` into every task venv)
+never carry a web server.
+
+It ships no frontend. A UI is a client of the schema above: point
+`HFLOW_UI_ASSETS` at a directory containing an `index.html` to serve one, or
+package assets under `hflow_server/static/` in a wheel and they are picked up
+automatically. Nothing here is reachable only from a browser, so more than
+one UI can exist against the same server without forking it.

packages/hflow-server/pyproject.toml

@@ -0,0 +1,32 @@
+[project]
+name = "hflow-server"
+version = "0.1.0"
+description = "Local web UI for HFlow: browse episodes, quality evidence, and the catalog"
+readme = "README.md"
+license = "Apache-2.0"
+authors = [{ name = "Hebbian Robotics" }]
+requires-python = ">=3.11"
+dependencies = [
+    # The floor is a real one, not a formality: this package imports
+    # hflow.workspace, hflow.import_pipeline_application,
+    # hflow.runtime.ingest_dag_topology and hflow.app's artifact-key constant
+    # at module scope, and none of them exist in hflow 0.2.0. Without the
+    # floor, `pip install hflow-server` beside an older hflow resolves happily and
+    # then dies with ImportError on `import hflow_server`. [tool.uv.sources] below
+    # only steers resolution inside this repo -- it is stripped from the built
+    # wheel's metadata -- and uv ignores this specifier for the workspace
+    # member, so in-repo development is unaffected by the number.
+    "hflow>=0.3.0",
+    "fastapi>=0.115",
+    "uvicorn>=0.32",
+]
+
+[project.urls]
+Repository = "https://github.com/Hebbian-Robotics/hflow"
+
+[tool.uv.sources]
+hflow = { workspace = true }
+
+[build-system]
+requires = ["uv_build>=0.11.33,<0.12"]
+build-backend = "uv_build"

packages/hflow-server/src/hflow_server/__init__.py

@@ -0,0 +1,13 @@
+"""HFlow workspace server: a read-mostly HTTP API over one data root.
+
+The public surface is deliberately tiny: :class:`ServerSettings` (parsed launch
+configuration) and :func:`serve` (runs the server). The CLI's ``hflow serve``
+subcommand is a thin caller of exactly these two names.
+"""
+
+from hflow_server._settings import ServerSettings
+from hflow_server.server import create_app, serve
+
+__version__ = "0.1.0"
+
+__all__ = ["ServerSettings", "__version__", "create_app", "serve"]