From 197e667dc952ea67b1b95ecd1124b8311c37a245 Mon Sep 17 00:00:00 2001 From: Set Paing Date: Wed, 9 Sep 2026 11:23:48 -0700 Subject: [PATCH 1/8] Organize v2 test layout per #513 review Move apps/interactive_drive/interactive_drive/tests/ to apps/interactive_drive/tests/, matching every other v2 app's layout (tests beside the package, not inside it). Document the v2 test ownership rule in AGENTS.md and CONTRIBUTING.md: engine tests in flashdreams/test_v2/, app tests in apps//tests/, architecture/adapter tests in integrations_v2//tests/. Fix stale references to the v1 integrations/ layout, fully removed by #581 but still described as current or mid-migration in AGENTS.md, CONTRIBUTING.md (two spots), flashdreams/test_v2/README.md (test_cli.py moved to apps/t2v/tests/, not deleted), and pyproject.toml's workspace globs. Verified: pytest collection matches before/after the move (84/84, apps/interactive_drive + apps/t2v). Full-repo collection inside tests/run_tests_docker.sh on real GPU hardware: 2350/3188 collected, 0 errors. Actual execution of the moved interactive_drive tests: 21/21 passed. Co-Authored-By: Claude Sonnet 5 Signed-off-by: Set Paing --- AGENTS.md | 19 +++++++++++++++--- CONTRIBUTING.md | 10 +++++---- .../tests/output/synthetic_scene_fixture.usdz | Bin .../tests/test_application.py | 0 .../tests/test_rasterizer_projection.py | 0 flashdreams/test_v2/README.md | 11 +++++----- pyproject.toml | 10 ++++----- 7 files changed, 32 insertions(+), 18 deletions(-) rename apps/interactive_drive/{interactive_drive => }/tests/output/synthetic_scene_fixture.usdz (100%) rename apps/interactive_drive/{interactive_drive => }/tests/test_application.py (100%) rename apps/interactive_drive/{interactive_drive => }/tests/test_rasterizer_projection.py (100%) diff --git a/AGENTS.md b/AGENTS.md index ad8ce090d..f33fe8931 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -46,9 +46,12 @@ Start here, then use the narrower docs for the task in front of you: - `flashdreams/flashdreams/infra/`: framework contracts and orchestration for configs, pipelines, encoders, decoders, diffusion models, schedulers, runners, profiling, and CUDA graph wrapping. - `flashdreams/flashdreams/recipes/`: built-in reusable recipe code such as WAN, Cosmos, TAEHV, and template wiring. - `flashdreams/flashdreams/configs/`, `plugins/`, and `scripts/`: runner registry, plugin discovery, and CLI entry points. -- `integrations//`: workspace-member model/plugin packages with their own configs, runners, tests, README files, and `pyproject.toml` entry points. +- `integrations_v2//`: v2 architecture + checkpoint wiring (`config.py`, `impl/`), adapters (`apps//adapter.py`), tests (`tests/`). Replaces the now-fully-removed v1 `integrations/`. +- `apps//`: v2 interactive apps. Package at `apps///`; tests beside it at `apps//tests/`, not inside the package. +- `flashdreams/test_v2/`: v2 engine tests. Merges into `flashdreams/tests/` once v1→v2 concludes. - `docs/source/`: Sphinx sources for quickstart, models, developer guides, API, and community docs. -- `tests/`: root test helpers plus package/integration tests. Ignore `.claude/worktrees/` when scanning the source tree; those are nested worktree artifacts, not the repo's current source. +- `flashdreams/tests/`: v1 core/engine tests, not yet migrated. +- `tests/` (repo root): test-runner scripts plus repo-meta checks, not package/integration tests. Ignore `.claude/worktrees/` when scanning the source tree; those are nested worktree artifacts, not the repo's current source. ## Skill Map @@ -91,9 +94,19 @@ Use `--no-instantiate` before GPU work to inspect the resolved runner config wit Every pytest test must carry exactly one of `ci_cpu`, `ci_gpu`, or `manual`; `CONTRIBUTING.md` has the exact rules. Use module-level `pytestmark = pytest.mark.ci_cpu` for pure Python/metadata tests. Keep GPU, `libGL`/`cv2`, large-checkpoint, credential, and download-heavy checks out of `ci_cpu`. +**v2 test ownership** — put a new test next to the thing it validates, not in a mirror of old package names: + +| You are testing… | Test lives in… | +| --- | --- | +| Engine loop (window, threads, presentation) | `flashdreams/test_v2/` | +| An app (flags, WASD, physics) | `apps//tests/` | +| An architecture or its adapter | `integrations_v2//tests/` | + +App tests must not need a real checkpoint or neural renderer — a stub network is enough; apps must not import `integrations_v2//impl/` directly. + ## Boundaries -Keep dependency direction strict: `core` -> `infra` -> recipes/integrations. `core` and `infra` must not import from `integrations/`; expose a generic config slot or override hook instead of adding model-specific branches. Built-in reusable model pieces belong in `flashdreams/flashdreams/recipes/`; standalone plugin packages belong in `integrations//`. +Keep dependency direction strict: `core` -> `infra` -> recipes/integrations. `core` and `infra` must not import from `integrations_v2/`; expose a generic config slot or override hook instead of adding model-specific branches. Built-in reusable model pieces belong in `flashdreams/flashdreams/recipes/`; standalone plugin packages belong in `integrations_v2//`. ## Known Pitfalls diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2895bb895..e53670bcf 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -232,9 +232,11 @@ short ping comment. locally is the easiest way to avoid surprises. - Prefer small, well-named functions over long functions with comments explaining each block. Comments should explain *why*, not *what*. -- Tests live in `flashdreams/tests/`, `integrations/*/tests/`, and - `integrations_v2/*/tests/`. Use - `pytest` and prefer existing fixtures over hand-rolled setup. See +- Tests live next to the thing they validate: engine tests in + `flashdreams/test_v2/`, app tests in `apps//tests/`, architecture/adapter + tests in `integrations_v2//tests/`. v1 tests remain in + `flashdreams/tests/` until migrated. Use `pytest` and prefer existing + fixtures over hand-rolled setup. See [Testing](#testing) for marker requirements. - Every source file added by a contribution must include the SPDX header used elsewhere in the project: @@ -321,7 +323,7 @@ the declared minimums. This means: ## Working with a single integration package -The workspace contains many integration packages under `integrations/`. +The workspace contains many integration packages under `integrations_v2/`. A full `uv sync` installs dependencies for *all* of them. If you only need one (e.g. you're working on `omnidreams`), use the distribution package name with `--package` to sync only that package's dependencies: diff --git a/apps/interactive_drive/interactive_drive/tests/output/synthetic_scene_fixture.usdz b/apps/interactive_drive/tests/output/synthetic_scene_fixture.usdz similarity index 100% rename from apps/interactive_drive/interactive_drive/tests/output/synthetic_scene_fixture.usdz rename to apps/interactive_drive/tests/output/synthetic_scene_fixture.usdz diff --git a/apps/interactive_drive/interactive_drive/tests/test_application.py b/apps/interactive_drive/tests/test_application.py similarity index 100% rename from apps/interactive_drive/interactive_drive/tests/test_application.py rename to apps/interactive_drive/tests/test_application.py diff --git a/apps/interactive_drive/interactive_drive/tests/test_rasterizer_projection.py b/apps/interactive_drive/tests/test_rasterizer_projection.py similarity index 100% rename from apps/interactive_drive/interactive_drive/tests/test_rasterizer_projection.py rename to apps/interactive_drive/tests/test_rasterizer_projection.py diff --git a/flashdreams/test_v2/README.md b/flashdreams/test_v2/README.md index e18bcf326..748b33769 100644 --- a/flashdreams/test_v2/README.md +++ b/flashdreams/test_v2/README.md @@ -26,11 +26,12 @@ CPU-only tests for the v2 protocols themselves: skipped when the serving packages are missing, which is also why a run writing a file does not import them. - [`apps/t2v/tests`](../../apps/t2v/tests) covers the reusable text-to-video - application, session, model loop, and stand-in model checks. -- `test_cli.py` covers `flashdreams-run-v2`: finding an application, splitting - the command line at `--`, choosing a window, describing the session to ask for, - and running one into a real MP4 with a stand-in for a model. An application - that describes no session of its own is run there too, since running more than + application, session, model loop, and stand-in model checks. Its + [`test_cli.py`](../../apps/t2v/tests/test_cli.py) also covers + `flashdreams-run-v2` itself: finding an application, splitting the command + line at `--`, choosing a window, describing the session to ask for, and + running one into a real MP4 with a stand-in for a model. An application that + describes no session of its own is run there too, since running more than text-to-video is the point of the command. - `test_metrics_output_sink.py` covers the sink that records what a run measured, which is a file another tool reads: what a benchmark expects of it diff --git a/pyproject.toml b/pyproject.toml index 7798b5be8..44a12b8a6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,13 +1,12 @@ [tool.uv.workspace] members = [ "flashdreams", - # Each populated subdir under ``integrations/`` or ``integrations_v2/`` - # is a standalone external integration repo (e.g. ``causal_forcing`` / - # ``self_forcing`` + # Each populated subdir under ``integrations_v2/`` is a standalone + # external integration repo (e.g. ``causal_forcing`` / ``self_forcing`` # plugins, ``omnidreams`` / ``lingbot`` integrations); empty stub # folders alongside them are intentional placeholders for future - # extractions and don't ship a ``pyproject.toml`` yet. - "integrations/*", + # extractions and don't ship a ``pyproject.toml`` yet. The v1 + # ``integrations/`` layout this replaced has been fully removed (#581). "integrations_v2/*", "apps/*", # Nested sub-packages that the integration globs do not reach. @@ -16,7 +15,6 @@ members = [ exclude = [ "apps/__init__.py", "apps/README.md", - "integrations/omnidreams", "integrations_v2/README.md", ] From 3509bf5ef4da5539e597b4b13c4027f3a05ab3c7 Mon Sep 17 00:00:00 2001 From: Set Paing Date: Wed, 9 Sep 2026 12:44:17 -0700 Subject: [PATCH 2/8] Address review: drop v1/integrations/ history, add real file tree MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit integrations/ is fully removed, not just replaced — stop narrating its history in AGENTS.md, CONTRIBUTING.md, and pyproject.toml comments. Replace AGENTS.md's partial/inaccurate repo map (impl/ mislabeled as "checkpoint wiring" rather than model implementation) with a pointer to a new CONTRIBUTING.md "File Tree Of Flashdreams" section, so there is one accurate source instead of two partial ones. Co-Authored-By: Claude Sonnet 5 Signed-off-by: Set Paing --- AGENTS.md | 11 +--------- CONTRIBUTING.md | 54 ++++++++++++++++++++++++++++++++++++++----------- pyproject.toml | 3 +-- 3 files changed, 44 insertions(+), 24 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f33fe8931..693d3d997 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,16 +42,7 @@ Start here, then use the narrower docs for the task in front of you: ## Repo Map -- `flashdreams/flashdreams/core/`: reusable numerical primitives, checkpoint loading, distributed helpers, attention, and I/O. Keep it model-agnostic. -- `flashdreams/flashdreams/infra/`: framework contracts and orchestration for configs, pipelines, encoders, decoders, diffusion models, schedulers, runners, profiling, and CUDA graph wrapping. -- `flashdreams/flashdreams/recipes/`: built-in reusable recipe code such as WAN, Cosmos, TAEHV, and template wiring. -- `flashdreams/flashdreams/configs/`, `plugins/`, and `scripts/`: runner registry, plugin discovery, and CLI entry points. -- `integrations_v2//`: v2 architecture + checkpoint wiring (`config.py`, `impl/`), adapters (`apps//adapter.py`), tests (`tests/`). Replaces the now-fully-removed v1 `integrations/`. -- `apps//`: v2 interactive apps. Package at `apps///`; tests beside it at `apps//tests/`, not inside the package. -- `flashdreams/test_v2/`: v2 engine tests. Merges into `flashdreams/tests/` once v1→v2 concludes. -- `docs/source/`: Sphinx sources for quickstart, models, developer guides, API, and community docs. -- `flashdreams/tests/`: v1 core/engine tests, not yet migrated. -- `tests/` (repo root): test-runner scripts plus repo-meta checks, not package/integration tests. Ignore `.claude/worktrees/` when scanning the source tree; those are nested worktree artifacts, not the repo's current source. +File structure is in [CONTRIBUTING.md's File Tree Of Flashdreams](CONTRIBUTING.md#file-tree-of-flashdreams). Ignore `.claude/worktrees/` when scanning the source tree; those are nested worktree artifacts, not the repo's current source. ## Skill Map diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e53670bcf..1219abcaa 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -18,13 +18,14 @@ issue and we'll fix it. 3. [Developer Certificate of Origin (DCO)](#developer-certificate-of-origin-dco) 4. [Submitting a pull request](#submitting-a-pull-request) 5. [Code review and merge](#code-review-and-merge) -6. [Coding conventions](#coding-conventions) -7. [Testing](#testing) -8. [Dependency version bounds](#dependency-version-bounds) -9. [Working with a single integration package](#working-with-a-single-integration-package) -10. [Licensing of contributions](#licensing-of-contributions) -11. [Reporting issues](#reporting-issues) -12. [Code of Conduct](#code-of-conduct) +6. [File Tree Of Flashdreams](#file-tree-of-flashdreams) +7. [Coding conventions](#coding-conventions) +8. [Testing](#testing) +9. [Dependency version bounds](#dependency-version-bounds) +10. [Working with a single integration package](#working-with-a-single-integration-package) +11. [Licensing of contributions](#licensing-of-contributions) +12. [Reporting issues](#reporting-issues) +13. [Code of Conduct](#code-of-conduct) ## Ways to contribute @@ -223,6 +224,37 @@ We aim for an initial review on every PR within two business days. If your PR has been quiet longer than that, please feel free to leave a short ping comment. +## File Tree Of Flashdreams + +```text +apps// # v2 interactive app (Drive, T2V, Cam2V, ...) + / # app package: session, UI, controls + tests/ # app-level tests (stub net, no checkpoint) + pyproject.toml + README.md + +integrations_v2// # v2 model architecture + demo bindings + config.py # model's pipeline config + impl/ # all model-specific implementation + tests/ # model-level tests (stub net + optional real checkpoint) + apps//adapter.py # create_app() -> IApplication, binds model to an app + pyproject.toml + README.md + +flashdreams/flashdreams/ # the framework package + core/ # numerical primitives, checkpoint loading, attention, I/O + infra/ # framework contracts: configs, pipelines, encoders/decoders, schedulers, runners + recipes/ # built-in reusable recipe code (WAN, Cosmos, TAEHV, ...) + api_v2/ # protocols an application implements + runtime_v2/ # the two-thread loop that runs one + configs/, plugins/, scripts/ # runner registry, plugin discovery, CLI entry points + +flashdreams/test_v2/ # v2 engine tests (window, run_session, threads) +flashdreams/tests/ # framework tests not yet migrated to test_v2/ +tests/ # repo-wide test-runner scripts + meta checks, not package tests +docs/source/ # Sphinx sources +``` + ## Coding conventions - Python 3.10+. Type-annotate new code; the project type-checks with @@ -232,11 +264,9 @@ short ping comment. locally is the easiest way to avoid surprises. - Prefer small, well-named functions over long functions with comments explaining each block. Comments should explain *why*, not *what*. -- Tests live next to the thing they validate: engine tests in - `flashdreams/test_v2/`, app tests in `apps//tests/`, architecture/adapter - tests in `integrations_v2//tests/`. v1 tests remain in - `flashdreams/tests/` until migrated. Use `pytest` and prefer existing - fixtures over hand-rolled setup. See +- Tests live next to the thing they validate — see the File Tree Of + Flashdreams above. Use `pytest` and prefer existing fixtures over + hand-rolled setup. See [Testing](#testing) for marker requirements. - Every source file added by a contribution must include the SPDX header used elsewhere in the project: diff --git a/pyproject.toml b/pyproject.toml index 44a12b8a6..70fa6f291 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,8 +5,7 @@ members = [ # external integration repo (e.g. ``causal_forcing`` / ``self_forcing`` # plugins, ``omnidreams`` / ``lingbot`` integrations); empty stub # folders alongside them are intentional placeholders for future - # extractions and don't ship a ``pyproject.toml`` yet. The v1 - # ``integrations/`` layout this replaced has been fully removed (#581). + # extractions and don't ship a ``pyproject.toml`` yet. "integrations_v2/*", "apps/*", # Nested sub-packages that the integration globs do not reach. From aad7555e26ace118fd1875ef8136882c7cec1cdb Mon Sep 17 00:00:00 2001 From: Set Paing Date: Wed, 9 Sep 2026 14:03:57 -0700 Subject: [PATCH 3/8] Address second review round: terminology, Dependencies section, wording - Drop the undefined "old package names" phrase. - Apply suggested table wording (Flashdreams Runtime/Protocol, not "Engine loop") consistently across AGENTS.md and CONTRIBUTING.md. - Rename Boundaries -> Dependencies; state apps/integrations_v2 dependency direction explicitly, with the consequence for app tests spelled out and why. Note CI enforcement as a separate follow-up. - Apply suggested File Tree wording verbatim (impl/, config.py, adapter.py, tests/ descriptions). Verified against the actual tree: config.py holds multiple pipeline definitions (checked omnidreams and wan21), every adapter.py exports create_app, tests/ presence pattern holds, no leftover nested apps/ tests/ anywhere, docs/source/ and the flashdreams/flashdreams/* breakdown are accurate. Co-Authored-By: Claude Sonnet 5 Signed-off-by: Set Paing --- AGENTS.md | 12 +++++++----- CONTRIBUTING.md | 18 +++++++++--------- 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 693d3d997..d3914e64e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -85,19 +85,21 @@ Use `--no-instantiate` before GPU work to inspect the resolved runner config wit Every pytest test must carry exactly one of `ci_cpu`, `ci_gpu`, or `manual`; `CONTRIBUTING.md` has the exact rules. Use module-level `pytestmark = pytest.mark.ci_cpu` for pure Python/metadata tests. Keep GPU, `libGL`/`cv2`, large-checkpoint, credential, and download-heavy checks out of `ci_cpu`. -**v2 test ownership** — put a new test next to the thing it validates, not in a mirror of old package names: +**v2 test ownership** — put a new test next to the thing it validates: | You are testing… | Test lives in… | | --- | --- | -| Engine loop (window, threads, presentation) | `flashdreams/test_v2/` | +| Flashdreams Runtime/Protocol (window, threads, presentation) | `flashdreams/test_v2/` | | An app (flags, WASD, physics) | `apps//tests/` | | An architecture or its adapter | `integrations_v2//tests/` | -App tests must not need a real checkpoint or neural renderer — a stub network is enough; apps must not import `integrations_v2//impl/` directly. +## Dependencies -## Boundaries +- `core` -> `infra` -> `recipes`/`integrations_v2`. `core` and `infra` must not import from `integrations_v2/`; expose a generic config slot or override hook instead of adding model-specific branches. Built-in reusable model pieces belong in `flashdreams/flashdreams/recipes/`; standalone plugin packages belong in `integrations_v2//`. +- `apps//` depends only on `flashdreams` (`core`, `infra`, `api_v2`) — never on `integrations_v2/`. A demo app must run against a stub network; wiring in a real model is the adapter's job, not the app's. +- `integrations_v2//` depends on `flashdreams` and on the `apps//` it adapts for (via `apps//adapter.py`) — never the other way around. -Keep dependency direction strict: `core` -> `infra` -> recipes/integrations. `core` and `infra` must not import from `integrations_v2/`; expose a generic config slot or override hook instead of adding model-specific branches. Built-in reusable model pieces belong in `flashdreams/flashdreams/recipes/`; standalone plugin packages belong in `integrations_v2//`. +Because of this direction, tests in `apps//tests/` must not import from `integrations_v2/` (real checkpoints or model `impl/`) — that would break the dependency rule above. CI enforcement of this is a separate follow-up, not yet built. ## Known Pitfalls diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1219abcaa..5dff1cc49 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -227,17 +227,17 @@ short ping comment. ## File Tree Of Flashdreams ```text -apps// # v2 interactive app (Drive, T2V, Cam2V, ...) - / # app package: session, UI, controls - tests/ # app-level tests (stub net, no checkpoint) +apps// # apps (Drive, T2V, Cam2V, ...) + / # app implementation + tests/ # validate app implementation pyproject.toml README.md -integrations_v2// # v2 model architecture + demo bindings - config.py # model's pipeline config - impl/ # all model-specific implementation - tests/ # model-level tests (stub net + optional real checkpoint) - apps//adapter.py # create_app() -> IApplication, binds model to an app +integrations_v2// # model integrations + demo bindings + config.py # collection of pipeline definitions for a particular `` + impl/ # implementation details of a model + tests/ # validate model implementation + apps//adapter.py # contains all entry point definitions (ex: `create_app`) for a particular `` pyproject.toml README.md @@ -249,7 +249,7 @@ flashdreams/flashdreams/ # the framework package runtime_v2/ # the two-thread loop that runs one configs/, plugins/, scripts/ # runner registry, plugin discovery, CLI entry points -flashdreams/test_v2/ # v2 engine tests (window, run_session, threads) +flashdreams/test_v2/ # Flashdreams Runtime/Protocol tests (window, run_session, threads) flashdreams/tests/ # framework tests not yet migrated to test_v2/ tests/ # repo-wide test-runner scripts + meta checks, not package tests docs/source/ # Sphinx sources From 60bf8a6eb6a0ba937042162e201a61e69449ba4a Mon Sep 17 00:00:00 2001 From: Set Paing Date: Wed, 9 Sep 2026 17:55:37 -0700 Subject: [PATCH 4/8] Address third review round: enforce the dependency rule, complete the file tree Stop app tests from importing model packages. apps/crazy_robotaxi/tests/ now builds a stub pipeline config from base flashdreams config classes; the tests that genuinely needed OmniDreams internals moved to integrations_v2/omnidreams/tests/test_recipe_configs.py, beside the interactive_drive adapter tests already there. No app test imports a model package now. Nest apps/v2v/ to apps/v2v/v2v/ with a sibling tests/, replacing its package-dir remap with the same auto-discovery every other app uses. It was the last app not matching the documented shape. Fix claims in AGENTS.md that were not true: apps do import integrations_v2 in one live-feature path, and the submodule list omitted runtime_v2, the most-used one. Rename Boundaries to Dependencies and state every direction the same way. Generalize the .claude/worktrees note to gitignored AI-tool directories. Complete the CONTRIBUTING.md file tree with the subpackages it omitted (runtime/, serving/, demo/, accelerated/, quality/, _pytest_plugins/), one entry per line, and note that the five fixture packages do not follow the model shape. Drop remaining references to the deleted v1 integrations/ directory from README.md, docs/README.md, and docs/source/, including a link that 404s and two sentences that only existed to contrast against v1. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Set Paing --- AGENTS.md | 15 +- CONTRIBUTING.md | 60 ++++-- README.md | 2 +- apps/crazy_robotaxi/tests/test_application.py | 197 +++++------------- apps/v2v/pyproject.toml | 5 +- apps/v2v/{ => v2v}/__init__.py | 0 apps/v2v/{ => v2v}/v2v.py | 0 docs/README.md | 2 +- docs/source/api/integrations.rst | 17 +- docs/source/community/faq.rst | 6 +- .../developer_guides/new_integration.rst | 2 +- flashdreams/test_v2/README.md | 5 +- .../omnidreams/tests/test_recipe_configs.py | 128 +++++++++++- pyproject.toml | 4 +- 14 files changed, 247 insertions(+), 196 deletions(-) rename apps/v2v/{ => v2v}/__init__.py (100%) rename apps/v2v/{ => v2v}/v2v.py (100%) diff --git a/AGENTS.md b/AGENTS.md index d3914e64e..c833005b0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,7 +42,7 @@ Start here, then use the narrower docs for the task in front of you: ## Repo Map -File structure is in [CONTRIBUTING.md's File Tree Of Flashdreams](CONTRIBUTING.md#file-tree-of-flashdreams). Ignore `.claude/worktrees/` when scanning the source tree; those are nested worktree artifacts, not the repo's current source. +File structure is in [CONTRIBUTING.md's File Tree Of FlashDreams](CONTRIBUTING.md#file-tree-of-flashdreams). Ignore gitignored AI-tool directories (see `.gitignore`) when scanning the source tree; they hold tool state, not the repo's current source. ## Skill Map @@ -89,17 +89,18 @@ Every pytest test must carry exactly one of `ci_cpu`, `ci_gpu`, or `manual`; `CO | You are testing… | Test lives in… | | --- | --- | -| Flashdreams Runtime/Protocol (window, threads, presentation) | `flashdreams/test_v2/` | +| FlashDreams Runtime/Protocol (window, threads, presentation) | `flashdreams/test_v2/` | | An app (flags, WASD, physics) | `apps//tests/` | -| An architecture or its adapter | `integrations_v2//tests/` | +| A model or its adapter | `integrations_v2//tests/` | ## Dependencies -- `core` -> `infra` -> `recipes`/`integrations_v2`. `core` and `infra` must not import from `integrations_v2/`; expose a generic config slot or override hook instead of adding model-specific branches. Built-in reusable model pieces belong in `flashdreams/flashdreams/recipes/`; standalone plugin packages belong in `integrations_v2//`. -- `apps//` depends only on `flashdreams` (`core`, `infra`, `api_v2`) — never on `integrations_v2/`. A demo app must run against a stub network; wiring in a real model is the adapter's job, not the app's. -- `integrations_v2//` depends on `flashdreams` and on the `apps//` it adapts for (via `apps//adapter.py`) — never the other way around. +- `infra` depends on `core` — never the other way around. `core` stays model-agnostic. +- `recipes`/`integrations_v2` depend on `infra` and `core` — never the other way around. Expose a generic config slot or override hook in `core`/`infra` instead of adding model-specific branches. Built-in reusable model pieces belong in `flashdreams/flashdreams/recipes/`; standalone plugin packages belong in `integrations_v2//`. +- `apps//` depends on `flashdreams` — never the other way around. An app is written against the framework, not against any one model: it must run against a stub network, and binding a real model is the adapter's job. +- `integrations_v2//` depends on `flashdreams` and on the app it adapts for (via its own `integrations_v2//apps//adapter.py`) — never the other way around. -Because of this direction, tests in `apps//tests/` must not import from `integrations_v2/` (real checkpoints or model `impl/`) — that would break the dependency rule above. CI enforcement of this is a separate follow-up, not yet built. +Because of this direction, tests in `apps//tests/` must not import from `integrations_v2/` — an app's tests run against a stub, and model-specific checks belong in `integrations_v2//tests/`. CI enforcement of this is a separate follow-up, not yet built. ## Known Pitfalls diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5dff1cc49..0f71ab93f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -18,7 +18,7 @@ issue and we'll fix it. 3. [Developer Certificate of Origin (DCO)](#developer-certificate-of-origin-dco) 4. [Submitting a pull request](#submitting-a-pull-request) 5. [Code review and merge](#code-review-and-merge) -6. [File Tree Of Flashdreams](#file-tree-of-flashdreams) +6. [File Tree Of FlashDreams](#file-tree-of-flashdreams) 7. [Coding conventions](#coding-conventions) 8. [Testing](#testing) 9. [Dependency version bounds](#dependency-version-bounds) @@ -224,37 +224,51 @@ We aim for an initial review on every PR within two business days. If your PR has been quiet longer than that, please feel free to leave a short ping comment. -## File Tree Of Flashdreams +## File Tree Of FlashDreams ```text -apps// # apps (Drive, T2V, Cam2V, ...) - / # app implementation - tests/ # validate app implementation +apps// # apps (Drive, T2V, Cam2V, ...) + / # app implementation + tests/ # validate app implementation pyproject.toml README.md -integrations_v2// # model integrations + demo bindings - config.py # collection of pipeline definitions for a particular `` - impl/ # implementation details of a model - tests/ # validate model implementation - apps//adapter.py # contains all entry point definitions (ex: `create_app`) for a particular `` +integrations_v2// # model integrations + demo bindings + config.py # collection of pipeline definitions for a particular `` + impl/ # implementation details of a model + tests/ # validate model implementation + apps//adapter.py # contains all entry point definitions (ex: `create_app`) for a particular `` pyproject.toml README.md -flashdreams/flashdreams/ # the framework package - core/ # numerical primitives, checkpoint loading, attention, I/O - infra/ # framework contracts: configs, pipelines, encoders/decoders, schedulers, runners - recipes/ # built-in reusable recipe code (WAN, Cosmos, TAEHV, ...) - api_v2/ # protocols an application implements - runtime_v2/ # the two-thread loop that runs one - configs/, plugins/, scripts/ # runner registry, plugin discovery, CLI entry points - -flashdreams/test_v2/ # Flashdreams Runtime/Protocol tests (window, run_session, threads) -flashdreams/tests/ # framework tests not yet migrated to test_v2/ -tests/ # repo-wide test-runner scripts + meta checks, not package tests -docs/source/ # Sphinx sources +flashdreams/flashdreams/ # the framework package + core/ # numerical primitives, checkpoint loading, attention, I/O + infra/ # framework contracts: configs, pipelines, encoders/decoders, schedulers, runners + recipes/ # built-in reusable recipe code (WAN, Cosmos, TAEHV, ...) + api_v2/ # protocols an application implements + runtime_v2/ # the two-thread loop that runs an application + runtime/ # experimental inference runtime API envelope (v0, pre-v2) + serving/ # optional serving utilities (WebRTC, network, launch) + demo/ # transport-neutral application hosting and I/O API + accelerated/ # accelerated kernels (quantization, multi-head attention) + quality/ # output-quality regression utilities (video, CLIP compare) + configs/ # runner registry and CLI aggregator + plugins/ # external-runner plugin layer (RunnerConfig discovery) + scripts/ # console-script entry points (flashdreams-run) + _pytest_plugins/ # pytest plugins (e.g. CI-tier marker enforcement) + +flashdreams/test_v2/ # FlashDreams Runtime/Protocol tests (window, run_session, threads) +flashdreams/tests/ # framework tests not yet migrated to test_v2/ +tests/ # repo-wide test-runner scripts + meta checks, not package tests +docs/source/ # Sphinx sources ``` +The `integrations_v2//` shape above is the layout for a model +integration. The smaller demo and fixture packages (`color_fade`, +`red_screen`, `null_model`, `imgui_ui_demo`, `slangpy_ui_demo`) carry a flat +`/` package instead; see +[`integrations_v2/README.md`](https://github.com/NVIDIA/flashdreams/blob/main/integrations_v2/README.md) for what each one is. + ## Coding conventions - Python 3.10+. Type-annotate new code; the project type-checks with @@ -265,7 +279,7 @@ docs/source/ # Sphinx sources - Prefer small, well-named functions over long functions with comments explaining each block. Comments should explain *why*, not *what*. - Tests live next to the thing they validate — see the File Tree Of - Flashdreams above. Use `pytest` and prefer existing fixtures over + FlashDreams above. Use `pytest` and prefer existing fixtures over hand-rolled setup. See [Testing](#testing) for marker requirements. - Every source file added by a contribution must include the SPDX diff --git a/README.md b/README.md index 70e80b0d6..37207c2fb 100644 --- a/README.md +++ b/README.md @@ -110,7 +110,7 @@ Drive a world model in real time with the unified OmniDreams `local-window` or ## Supported models FlashDreams ships first-party integrations under -[`integrations/`](integrations/). Each model has a dedicated docs page with +[`integrations_v2/`](integrations_v2/). Each model has a dedicated docs page with runner slugs, multi-GPU commands, and (where available) profiling benchmarks. | Model | Family | diff --git a/apps/crazy_robotaxi/tests/test_application.py b/apps/crazy_robotaxi/tests/test_application.py index d16b7b1e4..4ac2c4def 100644 --- a/apps/crazy_robotaxi/tests/test_application.py +++ b/apps/crazy_robotaxi/tests/test_application.py @@ -3,7 +3,7 @@ """CPU tests for Crazy Robotaxi's application boundary against FlashDreams V2.""" -from dataclasses import replace +from dataclasses import dataclass, field, replace from pathlib import Path from types import SimpleNamespace from typing import Any, cast @@ -29,30 +29,6 @@ _taxi_driver_command, ) from crazy_robotaxi.ui import CrazyRobotaxiImGuiUILoop -from omnidreams.apps.crazy_robotaxi.adapter import ( - OMNIDREAMS_CRAZY_ROBOTAXI_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_RESPONSIVE_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_RESPONSIVE_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_RESPONSIVE_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_PERF_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_PERF_RESPONSIVE_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_RESPONSIVE_DEFAULTS, -) -from omnidreams.config import ( - OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG, - OMNIDREAMS_FAST_PERF_RESPONSIVE_PIPELINE_CONFIG, - OMNIDREAMS_OPTIMIZED_GB300_PIPELINE_CONFIG, - OMNIDREAMS_OPTIMIZED_GB300_RESPONSIVE_PIPELINE_CONFIG, - OMNIDREAMS_OPTIMIZED_RTX_PRO_6000_PIPELINE_CONFIG, - OMNIDREAMS_OPTIMIZED_RTX_PRO_6000_RESPONSIVE_PIPELINE_CONFIG, - OMNIDREAMS_PERF_PIPELINE_CONFIG, - OMNIDREAMS_PERF_RESPONSIVE_PIPELINE_CONFIG, - OMNIDREAMS_PIPELINE_CONFIG, - OMNIDREAMS_RESPONSIVE_PIPELINE_CONFIG, -) from omnidreams_game_engine.config import BevConfig, RasterConfig from omnidreams_game_engine.input import DriverInput from omnidreams_game_engine.renderer_settings import RendererSettings @@ -63,11 +39,12 @@ SceneDefinition, ) -from flashdreams.runtime_v2.native_window_client_window import ( - NativeWindowClientWindow, -) +from flashdreams.infra.diffusion.model import DiffusionModelConfig +from flashdreams.infra.diffusion.scheduler.base import SchedulerConfig +from flashdreams.infra.diffusion.transformer.base import TransformerConfig +from flashdreams.infra.encoder.base import EncoderConfig +from flashdreams.infra.pipeline import StreamInferencePipelineConfig from flashdreams.runtime_v2.session_desc import PresentationMode -from flashdreams.runtime_v2.step_result import StepResult from flashdreams.runtime_v2.user_input_event import ( GamepadUserInputEvent, KeyboardInputState, @@ -86,9 +63,58 @@ ) +@dataclass(kw_only=True) +class _StubTransformerConfig(TransformerConfig): + """Adds the fields CrazyRobotaxiApplication logs unconditionally. + + Placeholder values only; no test in this file inspects them (tests that + care about real acceleration/backend settings live under + integrations_v2/omnidreams/tests/, since apps/ must not import + integrations_v2/). + """ + + native_dit_acceleration: str | None = None + native_dit_backend: str | None = None + native_dit_attention_backend: str | None = None + skip_finalize_kv_cache: bool = False + compile_network: bool = False + + +@dataclass(kw_only=True) +class _StubEncoderConfig(EncoderConfig): + """Adds the fields CrazyRobotaxiApplication logs unconditionally.""" + + native_vae_acceleration: str | None = None + native_vae_backend: str | None = None + + +@dataclass(kw_only=True) +class _StubSchedulerConfig(SchedulerConfig): + """Adds the field CrazyRobotaxiApplication logs unconditionally.""" + + denoising_timesteps: list[int] = field(default_factory=list) + + +_STUB_PIPELINE_CONFIG = StreamInferencePipelineConfig( + name="crazy-robotaxi-test-stub", + diffusion_model=DiffusionModelConfig( + transformer=_StubTransformerConfig(), + scheduler=_StubSchedulerConfig(), + ), + encoder=_StubEncoderConfig(), +) +"""A pipeline config with no model behind it, built from base flashdreams +config classes only. CrazyRobotaxiApplication reads ``.name``, derives +``.enable_sync_and_profile``, and logs several transformer/encoder fields +unconditionally, so app-level tests need a real, structured pipeline config, +not a real *model*.""" + +_STUB_DEFAULTS = CrazyRobotaxiApplicationDefaults(pipeline_config=_STUB_PIPELINE_CONFIG) + + def _application( *, - defaults: CrazyRobotaxiApplicationDefaults = OMNIDREAMS_CRAZY_ROBOTAXI_DEFAULTS, + defaults: CrazyRobotaxiApplicationDefaults = _STUB_DEFAULTS, **kwargs: Any, ) -> CrazyRobotaxiApplication: return CrazyRobotaxiApplication(defaults=defaults, **kwargs) @@ -464,70 +490,6 @@ def test_pipeline_profiling_is_an_app_local_opt_in( assert configured[0].enable_sync_and_profile is expected assert app._config is not None assert app._config.pipeline_profiling is expected - assert OMNIDREAMS_PIPELINE_CONFIG.enable_sync_and_profile - - -def test_model_adapters_keep_their_packaged_pipeline_configs() -> None: - for defaults, pipeline_config in ( - (OMNIDREAMS_CRAZY_ROBOTAXI_DEFAULTS, OMNIDREAMS_PIPELINE_CONFIG), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_PERF_DEFAULTS, - OMNIDREAMS_PERF_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS, - OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_DEFAULTS, - OMNIDREAMS_OPTIMIZED_GB300_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_DEFAULTS, - OMNIDREAMS_OPTIMIZED_RTX_PRO_6000_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_RESPONSIVE_DEFAULTS, - OMNIDREAMS_RESPONSIVE_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_PERF_RESPONSIVE_DEFAULTS, - OMNIDREAMS_PERF_RESPONSIVE_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_RESPONSIVE_DEFAULTS, - OMNIDREAMS_FAST_PERF_RESPONSIVE_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_RESPONSIVE_DEFAULTS, - OMNIDREAMS_OPTIMIZED_GB300_RESPONSIVE_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_RESPONSIVE_DEFAULTS, - OMNIDREAMS_OPTIMIZED_RTX_PRO_6000_RESPONSIVE_PIPELINE_CONFIG, - ), - ): - assert defaults.pipeline_config is pipeline_config - - -def test_fast_perf_combines_native_dit_and_native_vae_paths() -> None: - pipeline: Any = OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG - perf_pipeline: Any = OMNIDREAMS_PERF_PIPELINE_CONFIG - assert pipeline.name == "omnidreams-fast-perf" - assert pipeline.diffusion_model.seed is None - assert pipeline.decoder.use_compile is perf_pipeline.decoder.use_compile - assert pipeline.decoder.use_cuda_graph is True - assert pipeline.image_encoder.native_vae_acceleration == "required" - assert pipeline.image_encoder.native_vae_backend == "fp8" - assert pipeline.image_encoder.native_vae_fp8_auto_export is True - assert pipeline.encoder.native_vae_acceleration == "required" - assert pipeline.encoder.native_vae_backend == "fp8" - assert pipeline.encoder.native_vae_fp8_auto_export is True - assert pipeline.diffusion_model.transformer.native_dit_acceleration == "required" - assert ( - pipeline.diffusion_model.transformer.native_dit_backend == "fp8_kvcache_cudnn" - ) - assert pipeline.diffusion_model.transformer.native_dit_attention_backend == "cudnn" @pytest.mark.parametrize("resolution_wh", [(1280, 704), (1168, 640)]) @@ -548,7 +510,7 @@ def load_test_scene(request: object, raster: RasterConfig) -> SceneDefinition: app = _application( defaults=replace( - OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS, + _STUB_DEFAULTS, width=resolution_wh[0], height=resolution_wh[1], ), @@ -584,51 +546,6 @@ def load_test_scene(request: object, raster: RasterConfig) -> SceneDefinition: ) -def test_fast_perf_honors_explicit_pipeline_overrides() -> None: - app = _application( - defaults=OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS, - ) - - app.init( - [ - "--seed", - "7", - "--no-compile", - "--profile-pipeline", - ] - ) - - pipeline = cast(Any, app._pipeline_config) - transformer = pipeline.diffusion_model.transformer - assert pipeline.diffusion_model.seed == 7 - assert transformer.compile_network is False - assert transformer.native_dit_acceleration == "required" - assert transformer.skip_finalize_kv_cache is True - assert pipeline.diffusion_model.scheduler.denoising_timesteps == [1000, 100] - assert pipeline.enable_sync_and_profile is True - - -def test_map_context_disables_only_native_dit_on_selected_preset() -> None: - app = _application(defaults=OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS) - - app.init(["--live-edit-map-context"]) - - pipeline = cast(Any, app._pipeline_config) - original: Any = OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG - transformer = pipeline.diffusion_model.transformer - assert app._config is not None - assert app._config.scene_request.use_prompt_context - assert pipeline.name == original.name - assert transformer.native_dit_acceleration == "disabled" - assert transformer.native_dit_backend == ( - original.diffusion_model.transformer.native_dit_backend - ) - assert transformer.skip_finalize_kv_cache is True - assert pipeline.diffusion_model.scheduler == original.diffusion_model.scheduler - assert pipeline.image_encoder.native_vae_acceleration == "required" - assert pipeline.encoder.native_vae_acceleration == "required" - - def test_bev_render_fit_preserves_authored_aspect_ratio_and_smaller_sources() -> None: raster = RasterConfig() wide = RendererSettings(raster=raster, bev=BevConfig(width=800, height=400)) diff --git a/apps/v2v/pyproject.toml b/apps/v2v/pyproject.toml index 994f81e1b..502b5c8c7 100644 --- a/apps/v2v/pyproject.toml +++ b/apps/v2v/pyproject.toml @@ -26,9 +26,8 @@ dev = [ [tool.uv.sources] flashdreams = { workspace = true } -[tool.setuptools] -packages = ["v2v"] -package-dir = { v2v = "." } +[tool.setuptools.packages.find] +include = ["v2v*"] [tool.uv] managed = true diff --git a/apps/v2v/__init__.py b/apps/v2v/v2v/__init__.py similarity index 100% rename from apps/v2v/__init__.py rename to apps/v2v/v2v/__init__.py diff --git a/apps/v2v/v2v.py b/apps/v2v/v2v/v2v.py similarity index 100% rename from apps/v2v/v2v.py rename to apps/v2v/v2v/v2v.py diff --git a/docs/README.md b/docs/README.md index 6aaa4b25b..dc7a3f7b9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -108,5 +108,5 @@ to be present. - **A new API category** — add `source/api/.rst`, then include it in the API toctree in `source/index.rst`. - **Plugin-first note** — most actively developed integrations live under - `integrations//`. Use `source/api/integrations.rst` to document + `integrations_v2//`. Use `source/api/integrations.rst` to document in-tree `flashdreams.recipes.*` API surface that remains public. diff --git a/docs/source/api/integrations.rst b/docs/source/api/integrations.rst index 68ad0ad60..8f3745d3f 100644 --- a/docs/source/api/integrations.rst +++ b/docs/source/api/integrations.rst @@ -14,7 +14,7 @@ .. limitations under the License. Pipelines, runners, and applications -=================================== +==================================== FlashDreams model integrations use these public layers: @@ -23,18 +23,17 @@ FlashDreams model integrations use these public layers: - **V2 applications** (``IApplication``) that bind reusable demo infrastructure directly to pipeline configs. -Most actively developed model implementations now live under ``integrations/*`` -as plugin-style standalone packages. This page keeps documenting the in-tree -pipeline modules that are still exposed from ``flashdreams.recipes``. +Most actively developed model implementations now live under +``integrations_v2//`` as plugin-style standalone packages. This page +keeps documenting the in-tree pipeline modules that are still exposed from +``flashdreams.recipes``. .. note:: Pipeline modules import the heavy GPU stack (transformer-engine, CUDA ops) at import time, so this page shows them by *automodule* with ``:no-undoc-members:`` to keep the rendered API focused on the names - that these in-tree modules actually expose. The unified ``flashdreams-run`` - CLI shows end-to-end usage; see :doc:`/models/index` for model launch - examples. + that these in-tree modules actually expose. Integration structure (current) ------------------------------- @@ -51,9 +50,7 @@ V2 demo ports follow ``integrations_v2//``: Apart from that unique config, the integration root contains no implementation modules. These packages do not add ``runner.py`` or -``flashdreams.runner_configs`` just to launch a v2 demo. Older plugin-style -integrations under ``integrations/`` may still expose runners through -``flashdreams-run``. +``flashdreams.runner_configs`` just to launch a v2 demo. The default application entry point uses ``-`` and ``create_app``. Additional compatible diff --git a/docs/source/community/faq.rst b/docs/source/community/faq.rst index 8d71d04a4..3e1257709 100644 --- a/docs/source/community/faq.rst +++ b/docs/source/community/faq.rst @@ -41,7 +41,7 @@ methods. Which model integrations ship in the box? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -First-party model integrations ship under ``integrations/`` in the +First-party model integrations ship under ``integrations_v2//`` in the repo. The :doc:`/models/index` page has the full list; the documented integrations are: @@ -69,7 +69,7 @@ Only the core ``flashdreams`` package is published as a pure-Python wheel on PyPI. Integration packages — ``flashdreams-self-forcing``, ``flashdreams-lingbot``, and the others listed in `DEV.md `__ — -are not published; they live under ``integrations/`` in the monorepo +are not published; they live under ``integrations_v2//`` in the monorepo and are designed to be consumed either as a workspace member or as git-installable packages. @@ -97,7 +97,7 @@ How do I plug in a new model integration? The :doc:`/developer_guides/new_integration` guide walks the full flow — what to subclass on the runner side, how the entry-point registration works, and what the per-integration directory layout -looks like. The in-tree integrations under ``integrations/`` +looks like. The in-tree integrations under ``integrations_v2/`` are the canonical references; pick the one closest in shape to your new integration and use it as a template. diff --git a/docs/source/developer_guides/new_integration.rst b/docs/source/developer_guides/new_integration.rst index 40147df04..cda2d937a 100644 --- a/docs/source/developer_guides/new_integration.rst +++ b/docs/source/developer_guides/new_integration.rst @@ -18,7 +18,7 @@ Add a new method Before you start adding a new method, we highly recommend reading the :doc:`/developer_guides/inference_pipeline_overview` and :doc:`/developer_guides/config_system` pages first to obtain an overview of the system architecture. -FlashDreams aims to offer researchers a codebase that they can utilize to extend and develop novel video and world models. Our vision is for users to establish a *standalone repository* that imports FlashDreams as a dependency and overrides pipeline components (such as encoders, transformers, or decoders) to cater to specific functionality requirements of the new approach. We encourage you to maintain your method externally rather than pushing changes directly into the `integrations/ `_ directory of this repository. +FlashDreams aims to offer researchers a codebase that they can utilize to extend and develop novel video and world models. Our vision is for users to establish a *standalone repository* that imports FlashDreams as a dependency and overrides pipeline components (such as encoders, transformers, or decoders) to cater to specific functionality requirements of the new approach. We encourage you to maintain your method externally rather than pushing changes directly into the `integrations_v2/ `_ directory of this repository. However, if any of your new features require modifications to the core FlashDreams infra or introduce generally useful components (such as the :mod:`TAEHV decoder `), we encourage you to submit a PR to enable others to benefit from them. diff --git a/flashdreams/test_v2/README.md b/flashdreams/test_v2/README.md index 748b33769..1e1988fd8 100644 --- a/flashdreams/test_v2/README.md +++ b/flashdreams/test_v2/README.md @@ -38,9 +38,8 @@ CPU-only tests for the v2 protocols themselves: is checked against the reader itself in `flashdreams/tests/test_benchmark_harness.py`. -Application behaviour is tested by the application that owns it — see -`integrations_v2/red_screen/red_screen/tests/` and -`integrations_v2/color_fade/color_fade/tests/`. +Reusable apps keep their tests beside the package, in `apps//tests/` +(see `apps/t2v/tests/` above, and `apps/interactive_drive/tests/`). Run commands from the repository root. diff --git a/integrations_v2/omnidreams/tests/test_recipe_configs.py b/integrations_v2/omnidreams/tests/test_recipe_configs.py index 264611afe..98c138d98 100644 --- a/integrations_v2/omnidreams/tests/test_recipe_configs.py +++ b/integrations_v2/omnidreams/tests/test_recipe_configs.py @@ -5,11 +5,24 @@ from collections.abc import Callable from pathlib import Path -from typing import cast +from typing import Any, cast import pytest import tomli as tomllib +from crazy_robotaxi.application import CrazyRobotaxiApplication from interactive_drive import InteractiveDriveApplication, InteractiveDriveConfig +from omnidreams.apps.crazy_robotaxi.adapter import ( + OMNIDREAMS_CRAZY_ROBOTAXI_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_RESPONSIVE_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_RESPONSIVE_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_RESPONSIVE_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_PERF_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_PERF_RESPONSIVE_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_RESPONSIVE_DEFAULTS, +) from omnidreams.apps.interactive_drive.adapter import ( OMNIDREAMS_INTERACTIVE_DRIVE_DEFAULTS, OMNIDREAMS_INTERACTIVE_DRIVE_FAST_PERF_DEFAULTS, @@ -200,10 +213,123 @@ def test_application_defaults_are_owned_by_each_adapter() -> None: OMNIDREAMS_INTERACTIVE_DRIVE_FAST_PERF_DEFAULTS, OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG, ), + (OMNIDREAMS_CRAZY_ROBOTAXI_DEFAULTS, OMNIDREAMS_PIPELINE_CONFIG), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_PERF_DEFAULTS, + OMNIDREAMS_PERF_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS, + OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_DEFAULTS, + OMNIDREAMS_OPTIMIZED_GB300_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_DEFAULTS, + OMNIDREAMS_OPTIMIZED_RTX_PRO_6000_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_RESPONSIVE_DEFAULTS, + OMNIDREAMS_RESPONSIVE_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_PERF_RESPONSIVE_DEFAULTS, + OMNIDREAMS_PERF_RESPONSIVE_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_RESPONSIVE_DEFAULTS, + OMNIDREAMS_FAST_PERF_RESPONSIVE_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_RESPONSIVE_DEFAULTS, + OMNIDREAMS_OPTIMIZED_GB300_RESPONSIVE_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_RESPONSIVE_DEFAULTS, + OMNIDREAMS_OPTIMIZED_RTX_PRO_6000_RESPONSIVE_PIPELINE_CONFIG, + ), ): assert defaults.pipeline_config is pipeline_config +def test_fast_perf_combines_native_dit_and_native_vae_paths() -> None: + """Moved from apps/crazy_robotaxi/tests/test_application.py: pure OmniDreams + pipeline-config assertions, no Crazy Robotaxi app involved.""" + pipeline: Any = OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG + perf_pipeline: Any = OMNIDREAMS_PERF_PIPELINE_CONFIG + assert pipeline.name == "omnidreams-fast-perf" + assert pipeline.diffusion_model.seed is None + assert pipeline.decoder.use_compile is perf_pipeline.decoder.use_compile + assert pipeline.decoder.use_cuda_graph is True + assert pipeline.image_encoder.native_vae_acceleration == "required" + assert pipeline.image_encoder.native_vae_backend == "fp8" + assert pipeline.image_encoder.native_vae_fp8_auto_export is True + assert pipeline.encoder.native_vae_acceleration == "required" + assert pipeline.encoder.native_vae_backend == "fp8" + assert pipeline.encoder.native_vae_fp8_auto_export is True + assert pipeline.diffusion_model.transformer.native_dit_acceleration == "required" + assert ( + pipeline.diffusion_model.transformer.native_dit_backend == "fp8_kvcache_cudnn" + ) + assert pipeline.diffusion_model.transformer.native_dit_attention_backend == "cudnn" + + +def test_crazy_robotaxi_fast_perf_honors_explicit_pipeline_overrides() -> None: + """Moved from apps/crazy_robotaxi/tests/test_application.py: tests that Crazy + Robotaxi's CLI parsing correctly mutates OmniDreams's pipeline config, which + is inherently an adapter-level (app x model) concern.""" + app = CrazyRobotaxiApplication( + defaults=OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS + ) + + app.init( + [ + "--seed", + "7", + "--no-compile", + "--profile-pipeline", + ] + ) + + pipeline = cast(Any, app._pipeline_config) + transformer = pipeline.diffusion_model.transformer + assert pipeline.diffusion_model.seed == 7 + assert transformer.compile_network is False + assert transformer.native_dit_acceleration == "required" + assert transformer.skip_finalize_kv_cache is True + assert pipeline.diffusion_model.scheduler.denoising_timesteps == [1000, 100] + assert pipeline.enable_sync_and_profile is True + + +def test_crazy_robotaxi_map_context_disables_only_native_dit_on_selected_preset() -> ( + None +): + """Moved from apps/crazy_robotaxi/tests/test_application.py; same reasoning + as test_crazy_robotaxi_fast_perf_honors_explicit_pipeline_overrides.""" + app = CrazyRobotaxiApplication( + defaults=OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS + ) + + app.init(["--live-edit-map-context"]) + + pipeline = cast(Any, app._pipeline_config) + original: Any = OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG + transformer = pipeline.diffusion_model.transformer + assert app._config is not None + assert app._config.scene_request.use_prompt_context + assert pipeline.name == original.name + assert transformer.native_dit_acceleration == "disabled" + assert transformer.native_dit_backend == ( + original.diffusion_model.transformer.native_dit_backend + ) + assert transformer.skip_finalize_kv_cache is True + assert pipeline.diffusion_model.scheduler == original.diffusion_model.scheduler + assert pipeline.image_encoder.native_vae_acceleration == "required" + assert pipeline.encoder.native_vae_acceleration == "required" + + @pytest.mark.parametrize( ("factory", "resolution_wh"), [ diff --git a/pyproject.toml b/pyproject.toml index 70fa6f291..9fced669c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -3,9 +3,7 @@ members = [ "flashdreams", # Each populated subdir under ``integrations_v2/`` is a standalone # external integration repo (e.g. ``causal_forcing`` / ``self_forcing`` - # plugins, ``omnidreams`` / ``lingbot`` integrations); empty stub - # folders alongside them are intentional placeholders for future - # extractions and don't ship a ``pyproject.toml`` yet. + # plugins, ``omnidreams`` / ``lingbot`` integrations). "integrations_v2/*", "apps/*", # Nested sub-packages that the integration globs do not reach. From b4a5164534b570e7b87f124b4dff950ed7d73a09 Mon Sep 17 00:00:00 2001 From: Set Paing Date: Thu, 10 Sep 2026 11:48:37 -0700 Subject: [PATCH 5/8] Document app and integration layouts next to those directories. CONTRIBUTING.md keeps the repo map; nested plugin shape lives in apps/README.md and integrations_v2/README.md so it changes with the code. Signed-off-by: Set Paing --- CONTRIBUTING.md | 27 ++++++++------------------ apps/README.md | 41 +++++++++++++++++++++++++++++++++++++++ integrations_v2/README.md | 1 + 3 files changed, 50 insertions(+), 19 deletions(-) create mode 100644 apps/README.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0f71ab93f..89042f5a4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -226,21 +226,16 @@ short ping comment. ## File Tree Of FlashDreams +Where a new app or model goes. Each documents its own layout: + ```text -apps// # apps (Drive, T2V, Cam2V, ...) - / # app implementation - tests/ # validate app implementation - pyproject.toml - README.md - -integrations_v2// # model integrations + demo bindings - config.py # collection of pipeline definitions for a particular `` - impl/ # implementation details of a model - tests/ # validate model implementation - apps//adapter.py # contains all entry point definitions (ex: `create_app`) for a particular `` - pyproject.toml - README.md +apps/ # reusable apps; layout in apps/README.md +integrations_v2/ # model packages; layout in integrations_v2/README.md +``` + +The framework package, and the test and doc trees: +```text flashdreams/flashdreams/ # the framework package core/ # numerical primitives, checkpoint loading, attention, I/O infra/ # framework contracts: configs, pipelines, encoders/decoders, schedulers, runners @@ -263,12 +258,6 @@ tests/ # repo-wide test-runner scripts + meta checks, n docs/source/ # Sphinx sources ``` -The `integrations_v2//` shape above is the layout for a model -integration. The smaller demo and fixture packages (`color_fade`, -`red_screen`, `null_model`, `imgui_ui_demo`, `slangpy_ui_demo`) carry a flat -`/` package instead; see -[`integrations_v2/README.md`](https://github.com/NVIDIA/flashdreams/blob/main/integrations_v2/README.md) for what each one is. - ## Coding conventions - Python 3.10+. Type-annotate new code; the project type-checks with diff --git a/apps/README.md b/apps/README.md new file mode 100644 index 000000000..eb7abe002 --- /dev/null +++ b/apps/README.md @@ -0,0 +1,41 @@ + + +Reusable applications built on the v2 API. Each directory is a standalone +package that depends on `flashdreams` and stays model-agnostic: an app runs +against a stub network, and binding a real model is the adapter's job, in +`integrations_v2//apps//adapter.py`. + +## The layout + +```text +apps// + / # the app implementation + tests/ # validate the app against a stub, not a real model + pyproject.toml + README.md # purpose, controls, example command line, options +``` + +Tests live in `apps//tests/`, beside the package rather than inside +it, and must not import from `integrations_v2/`. A check that needs a real +model or its adapter belongs in `integrations_v2//tests/`. + +## What is here + +- `t2v` — text-to-video: prompt in, frames out. The reference app to copy. +- `cam2v` — interactive camera-to-video. +- `action2v` — world models driven by direct action input. +- `v2v` — video-to-video, used for super-resolution. +- `interactive_drive` — the interactive driving demo, keyboard and wheel input. +- `crazy_robotaxi` — the driving game built on `omnidreams_game_engine`. +- `omnidreams_game_engine` — reusable simulation, authored-map, physics, and + conditioning components. It owns no runtime loop, so it is a library the + other apps build on rather than something you launch. + +`omnidreams_game_engine` is not the OmniDreams model. That is +`integrations_v2/omnidreams`. The names are close; the packages are unrelated. + +Each app's own README covers how to launch it and what the controls are. To +bind a model to one of these, read `integrations_v2/README.md`. diff --git a/integrations_v2/README.md b/integrations_v2/README.md index 049e11003..97f197c51 100644 --- a/integrations_v2/README.md +++ b/integrations_v2/README.md @@ -18,6 +18,7 @@ follows is already done for you. - `red_screen` — the smallest interactive one, streaming to a browser. - `slangpy_ui_demo` — three applications that draw widgets over model output, and the reference for writing a UI loop. +- `imgui_ui_demo` — an editable ImGui text field rendered over model output. - `lingbot` — the Lingbot World model and its `cam2v-lingbot` binding to the shared interactive camera-to-video application. - `waypoint` — the Waypoint model binding to the shared `apps/action2v` From db89d5b21cce21b6ca2bf703afa02a225f959757 Mon Sep 17 00:00:00 2001 From: Set Paing Date: Thu, 10 Sep 2026 12:10:32 -0700 Subject: [PATCH 6/8] Match integrations_v2 layout captions to the review wording. The File Tree now points at this README, so the nested comments need to use the same phrasing that landed in CONTRIBUTING.md yesterday. Signed-off-by: Set Paing --- integrations_v2/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/integrations_v2/README.md b/integrations_v2/README.md index 97f197c51..6774d6c95 100644 --- a/integrations_v2/README.md +++ b/integrations_v2/README.md @@ -44,13 +44,13 @@ integrations_v2// pyproject.toml README.md __init__.py - config.py # model's unique pipeline config or config wrapper - impl/ # all model-specific implementation - tests/ # model-specific tests, when needed + config.py # collection of pipeline definitions for a particular `` + impl/ # implementation details of a model + tests/ # validate model implementation apps/ / __init__.py - adapter.py # create_app() -> IApplication + adapter.py # all entry point definitions for `` (ex: `create_app`) README.md # launch instructions only ``` From fc031bfb4ed65ab9a49775cde38095123da899a0 Mon Sep 17 00:00:00 2001 From: Set Paing Date: Thu, 10 Sep 2026 12:43:24 -0700 Subject: [PATCH 7/8] Keep Interactive Drive tests out of workspace ty after the move. ty already excluded apps/interactive_drive/interactive_drive/**, which covered the old nested tests path. Sibling tests/ is the same files. Signed-off-by: Set Paing --- pyproject.toml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pyproject.toml b/pyproject.toml index 9fced669c..1535140cd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -121,6 +121,8 @@ extra-paths = [ exclude = [ "**/protos/*pb2*", "apps/interactive_drive/interactive_drive/**", + # Same files as above; tests live beside the package, not inside it. + "apps/interactive_drive/tests/**", # These map/archive parsers and numerical PhysX adapters intentionally # accept dynamically shaped YAML, Arrow, and NumPy values. Keep the new # V2 application, session, engine contracts, rollout, input, and renderer From 27087cd4f3f5e0ac328954bb9de7b310e19c87d9 Mon Sep 17 00:00:00 2001 From: Set Paing Date: Thu, 10 Sep 2026 15:55:39 -0700 Subject: [PATCH 8/8] Document pytest discovery and split the new-integration tutorial. #513 asked for the collection convention in CONTRIBUTING. The new-integration page still teaches runner.py; point at integrations_v2/README.md instead of rewriting it here. Signed-off-by: Set Paing --- AGENTS.md | 2 +- CONTRIBUTING.md | 22 ++++++++++++++++++- .../developer_guides/new_integration.rst | 7 +++++- tests/README.md | 4 +++- 4 files changed, 31 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index c833005b0..8a3141dd9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -83,7 +83,7 @@ Use `--no-instantiate` before GPU work to inspect the resolved runner config wit ## Testing Guidance -Every pytest test must carry exactly one of `ci_cpu`, `ci_gpu`, or `manual`; `CONTRIBUTING.md` has the exact rules. Use module-level `pytestmark = pytest.mark.ci_cpu` for pure Python/metadata tests. Keep GPU, `libGL`/`cv2`, large-checkpoint, credential, and download-heavy checks out of `ci_cpu`. +Every pytest test must carry exactly one of `ci_cpu`, `ci_gpu`, or `manual`; `CONTRIBUTING.md` has the exact rules, including how pytest discovers files and functions (`test_*.py`, `test_*`). Use module-level `pytestmark = pytest.mark.ci_cpu` for pure Python/metadata tests. Keep GPU, `libGL`/`cv2`, large-checkpoint, credential, and download-heavy checks out of `ci_cpu`. **v2 test ownership** — put a new test next to the thing it validates: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 89042f5a4..141e7f18f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -270,7 +270,7 @@ docs/source/ # Sphinx sources - Tests live next to the thing they validate — see the File Tree Of FlashDreams above. Use `pytest` and prefer existing fixtures over hand-rolled setup. See - [Testing](#testing) for marker requirements. + [Testing](#testing) for discovery and marker requirements. - Every source file added by a contribution must include the SPDX header used elsewhere in the project: @@ -297,6 +297,26 @@ docs/source/ # Sphinx sources ## Testing +Pytest collection is configured in the root `pyproject.toml`. There is +no `testpaths` setting, so `pytest` from the repo root walks the tree: + +- Files named `test_*.py`. Pytest's default also collects `*_test.py`; + don't use that name here. +- Classes named `Test*` (pytest default). Some modules use classes, + some only module-level functions. +- Functions and methods named `test_*` (pytest default), including + `async def test_*`. + +Same test filename in different folders is fine (several +`test_application.py` files exist). Root `pyproject.toml` already sets +`--import-mode=importlib` so pytest keeps them separate. Do not remove +that flag. + +A `test_*.py` next to the code it validates is collected with no CI +change. The root `pyproject.toml` skips `parity_check`, +`parity_check_v2`, `baseline_fastvideo`, and `baseline_lightx2v` via +`norecursedirs`. + Every test function must be marked with exactly one **CI tier marker**. A pytest plugin (`flashdreams._pytest_plugins.marker_enforcement`) enforces this at collection time -- tests without a marker are diff --git a/docs/source/developer_guides/new_integration.rst b/docs/source/developer_guides/new_integration.rst index cda2d937a..91b500e7e 100644 --- a/docs/source/developer_guides/new_integration.rst +++ b/docs/source/developer_guides/new_integration.rst @@ -25,6 +25,11 @@ However, if any of your new features require modifications to the core FlashDrea File structure -------------- +This page still describes a standalone package that registers a +``flashdreams.runner_configs`` runner (``runner.py`` below). In-tree +packages under ``integrations_v2/`` do not add ``runner.py``; they follow +the layout in `integrations_v2/README.md `_. + We recommend the following file structure for your new method: .. code-block:: text @@ -104,7 +109,7 @@ The runner handles CLI-facing I/O and runtime loops. ) -You can use the existing integrations under the `integrations_v2/ `_ directory as a minimal guide. These folders are simple examples of what mini standalone repositories that depend on FlashDreams look like. Examples are often the best way to learn; take a look at the `OmniDreams `_ and `Self-Forcing `_ integrations for good references on how to extend and use FlashDreams in your own projects. +The in-tree packages under `integrations_v2/ `_ are the current examples, but they follow the v2 layout (``config.py``, ``impl/``, ``apps//adapter.py``), not the ``runner.py`` tree on this page. `OmniDreams `_ and `Self-Forcing `_ are the usual starting points. Rewriting this tutorial onto that layout is a follow-up. Registering your method ----------------------- diff --git a/tests/README.md b/tests/README.md index 6d45cf93b..d762841c4 100644 --- a/tests/README.md +++ b/tests/README.md @@ -25,7 +25,9 @@ Both scripts resolve paths relative to their own location and can be invoked fro ## What gets run -When no `TEST_TARGET` is given, each script performs global discovery of `**/test_*.py`: +When no `TEST_TARGET` is given, each script runs `pytest` from the repo +root. See `CONTRIBUTING.md` Testing for how files and functions are +named. Pytest is invoked with `-m "not manual"` so any test marked `@pytest.mark.manual` is skipped.