[Test] Organize v2 test layout, fix stale integrations/ references (#513) by spaingnv · Pull Request #602 · NVIDIA/flashdreams

spaingnv · 2026-09-09T19:05:20Z

Closes #513.

What changed

Test layout

apps/interactive_drive/interactive_drive/tests/ moved to
apps/interactive_drive/tests/, so tests sit beside the package rather than
inside it.
apps/crazy_robotaxi/tests/ no longer imports omnidreams. It builds a stub
pipeline config from base flashdreams config classes, and the tests that
genuinely needed OmniDreams internals moved to
integrations_v2/omnidreams/tests/test_recipe_configs.py, beside the
interactive_drive adapter tests already there.
apps/v2v/ is nested to apps/v2v/v2v/ with a sibling tests/, replacing a
package-dir remap with the same auto-discovery every other app uses. All
seven apps now match one shape.

Documentation

AGENTS.md: the repo map points at CONTRIBUTING.md instead of duplicating
it; a table says which of the three test locations a new test belongs in;
Boundaries is renamed Dependencies and every bullet states a direction the
same way.
CONTRIBUTING.md: new File Tree Of FlashDreams section covering each
flashdreams/flashdreams/ subpackage, with a note that the five fixture
packages do not follow the model-integration shape.
flashdreams/test_v2/README.md: test_cli.py was not deleted by [Demos] Refactor+Move 6 T2V demos & Underlying integrations to v2 API #558, it
moved to apps/t2v/tests/test_cli.py. Fixed the reference.

Stale integrations/ references. That directory was removed by #581 but was
still described as current in AGENTS.md, CONTRIBUTING.md, root
pyproject.toml (a dead workspace glob and exclude), README.md,
docs/README.md, docs/source/api/integrations.rst,
docs/source/community/faq.rst, and
docs/source/developer_guides/new_integration.rst, which had a link that 404s.

Why

Most of the layout was already right after #558. What was missing was the rule
written down. Once it was written, the things contradicting it had to be fixed
too, or it is not a rule.

Not in this PR

test_v2/runtime/ grouping. This issue's original text asks for it and
[API] Organize test_v2 into runtime/ and t2v/ #569 did exactly that. The review landed on engine tests staying flat under
flashdreams/test_v2/ until v2 concludes and then merging into tests/, which
makes a subfolder now churn that gets undone. I raised this as an open question
in the design comment and nothing came back, so I am recording it here as a
decision rather than a silent drop. Say the word if you would rather have it.
apps/crazy_robotaxi/crazy_robotaxi/live_edit/style_ability.py lazy-imports
omnidreams.impl._edit_lora and omnidreams.impl._drift_corrector (lines 247,
309, 590, 690). That is app code reaching into a model's private impl/, and
omnidreams is not in that app's pyproject.toml at all, so the path only
resolves because the workspace installs everything. Unpicking it changes a live
feature rather than test layout, so I left it and worded Dependencies to
describe the intended direction without claiming the violation is absent.
Happy to fix it here or file it separately.
Flattening apps/<slug>/<slug>/. The nesting is what lets
[tool.setuptools.packages.find] auto-discover submodules; flattening trades
that for a hand-maintained package list, for a shorter path and no functional
gain.
skills/*.md still describe the removed integrations/<name>/ layout in
several worked examples. Same bug class as the sweep above, but it is workflow
guidance that needs re-verifying against current conventions rather than a path
rename, so it belongs in its own change.
CI enforcement of test location, and test-quality/coverage cleanup, are
separate follow-ups per the review.

Validation

pytest apps/ integrations_v2/omnidreams/tests/test_recipe_configs.py -m ci_cpu
gives 297 passed, 3 skipped. No app test imports a model package, checked
including lazy imports inside function bodies.
All seven app packages import cleanly and match the documented shape.
ruff check and ruff format --check clean on the touched files.
sphinx-build -E completes with no warnings.

Move apps/interactive_drive/interactive_drive/tests/ to apps/interactive_drive/tests/, matching every other v2 app's layout (tests beside the package, not inside it). Document the v2 test ownership rule in AGENTS.md and CONTRIBUTING.md: engine tests in flashdreams/test_v2/, app tests in apps/<name>/tests/, architecture/adapter tests in integrations_v2/<model>/tests/. Fix stale references to the v1 integrations/ layout, fully removed by NVIDIA#581 but still described as current or mid-migration in AGENTS.md, CONTRIBUTING.md (two spots), flashdreams/test_v2/README.md (test_cli.py moved to apps/t2v/tests/, not deleted), and pyproject.toml's workspace globs. Verified: pytest collection matches before/after the move (84/84, apps/interactive_drive + apps/t2v). Full-repo collection inside tests/run_tests_docker.sh on real GPU hardware: 2350/3188 collected, 0 errors. Actual execution of the moved interactive_drive tests: 21/21 passed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Set Paing <spaing@nvidia.com>

copy-pr-bot · 2026-09-09T19:05:24Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

jmccaffrey-nv · 2026-09-09T19:16:07Z

/ok to test 197e667

ArielG-NV

3 Comments

ArielG-NV · 2026-09-09T19:13:12Z

 - `flashdreams/flashdreams/recipes/`: built-in reusable recipe code such as WAN, Cosmos, TAEHV, and template wiring.
 - `flashdreams/flashdreams/configs/`, `plugins/`, and `scripts/`: runner registry, plugin discovery, and CLI entry points.
- `integrations/<name>/`: workspace-member model/plugin packages with their own configs, runners, tests, README files, and `pyproject.toml` entry points.
+- `integrations_v2/<model>/`: v2 architecture + checkpoint wiring (`config.py`, `impl/`), adapters (`apps/<demo>/adapter.py`), tests (`tests/`). Replaces the now-fully-removed v1 `integrations/`.


Informing the agent of v1 is unnecessary.

The description of the files are wrong. Example: impl/ is model implementation, not "v2 architecture + checkpoint wiring".

We could simplify the file by just stating: "File structure is located in CONTRIBUTING.md".

Then we can have a ## File Tree Of Flashdreams section that has a basic file-tree with explanation

apps/ # Contains all model-antagonistic demo templates apps/<app_slug>/<app_slug>/ # Implementation of app & entrypoint apps/<app_slug>/pyproject.toml # ... apps/<app_slug>/README.md # Specifies in order: program purpose, controls, example cmd-line use, command-line-options table flashdreams/ #.. ...

ArielG-NV · 2026-09-09T19:22:45Z

+- Tests live next to the thing they validate: engine tests in
+  `flashdreams/test_v2/`, app tests in `apps/<name>/tests/`, architecture/adapter
+  tests in `integrations_v2/<model>/tests/`. v1 tests remain in
+  `flashdreams/tests/` until migrated. Use `pytest` and prefer existing
+  fixtures over hand-rolled setup. See


This is not a complete explanation of our new file-tree.

ArielG-NV · 2026-09-09T19:23:03Z

-    # extractions and don't ship a ``pyproject.toml`` yet.
-    "integrations/*",
+    # extractions and don't ship a ``pyproject.toml`` yet. The v1
+    # ``integrations/`` layout this replaced has been fully removed (#581).


V1 does not need to be referenced.

integrations/ is fully removed, not just replaced — stop narrating its history in AGENTS.md, CONTRIBUTING.md, and pyproject.toml comments. Replace AGENTS.md's partial/inaccurate repo map (impl/ mislabeled as "checkpoint wiring" rather than model implementation) with a pointer to a new CONTRIBUTING.md "File Tree Of Flashdreams" section, so there is one accurate source instead of two partial ones. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Set Paing <spaing@nvidia.com>

spaingnv · 2026-09-09T19:44:25Z

Addressed all three: dropped every v1/integrations/ reference (it's fully removed, no reason to narrate history), and added a ## File Tree Of Flashdreams section to CONTRIBUTING.md with the corrected impl//config.py description — AGENTS.md now just points at it instead of duplicating.

ArielG-NV

suggestions of terminology/correctness

ArielG-NV · 2026-09-09T20:03:41Z


 Every pytest test must carry exactly one of `ci_cpu`, `ci_gpu`, or `manual`; `CONTRIBUTING.md` has the exact rules. Use module-level `pytestmark = pytest.mark.ci_cpu` for pure Python/metadata tests. Keep GPU, `libGL`/`cv2`, large-checkpoint, credential, and download-heavy checks out of `ci_cpu`.

+**v2 test ownership** — put a new test next to the thing it validates, not in a mirror of old package names:


old package names:

What is an "old package name"? we should not be defining new terms

ArielG-NV · 2026-09-09T20:04:36Z

+
+| You are testing… | Test lives in… |
+| --- | --- |
+| Engine loop (window, threads, presentation) | `flashdreams/test_v2/` |


ArielG-NV · 2026-09-09T20:06:26Z

+| An app (flags, WASD, physics) | `apps/<name>/tests/` |
+| An architecture or its adapter | `integrations_v2/<model>/tests/` |
+
+App tests must not need a real checkpoint or neural renderer — a stub network is enough; apps must not import `integrations_v2/<model>/impl/` directly.


Wrong location for this, we have a Boundaries section to draw "Boundaries"/depdencies

Additionally, intention on why we have this rule should be clear.

ArielG-NV · 2026-09-09T20:10:58Z

 ## Boundaries

-Keep dependency direction strict: `core` -> `infra` -> recipes/integrations. `core` and `infra` must not import from `integrations/`; expose a generic config slot or override hook instead of adding model-specific branches. Built-in reusable model pieces belong in `flashdreams/flashdreams/recipes/`; standalone plugin packages belong in `integrations/<name>/`.
+Keep dependency direction strict: `core` -> `infra` -> recipes/integrations. `core` and `infra` must not import from `integrations_v2/`; expose a generic config slot or override hook instead of adding model-specific branches. Built-in reusable model pieces belong in `flashdreams/flashdreams/recipes/`; standalone plugin packages belong in `integrations_v2/<name>/`.


Lets rename this section Dependencies and fix it so it maps correctly.

* ... * `apps` depends on... * `integrations_v2` depends on `apps/` and ... Due to these rules, tests in `apps/` should not import models/entry-points from `integrations_v2` since this would break the dependency rules we set here.

We likely want CI to validate boundaries as well if we are going to draw them... but this is a seperate task

ArielG-NV · 2026-09-09T20:12:53Z

+## File Tree Of Flashdreams
+
+```text
+apps/<slug>/                   # v2 interactive app (Drive, T2V, Cam2V, ...)


Suggested change

apps/<slug>/ # v2 interactive app (Drive, T2V, Cam2V, ...)

apps/<app_slug>/ # apps (Drive, T2V, Cam2V, ...)

Not all apps are interactive

ArielG-NV · 2026-09-09T20:16:18Z

+  README.md
+
+integrations_v2/<model>/       # v2 model architecture + demo bindings
+  config.py                    # model's pipeline config


Suggested change

config.py # model's pipeline config

config.py # collection of pipeline definitions for a particular `<model>`

this file can have multiple configs

ArielG-NV · 2026-09-09T20:16:40Z

+
+integrations_v2/<model>/       # v2 model architecture + demo bindings
+  config.py                    # model's pipeline config
+  impl/                        # all model-specific implementation


Suggested change

impl/ # all model-specific implementation

impl/ # implementation details of a model

ArielG-NV · 2026-09-09T20:18:19Z

+integrations_v2/<model>/       # v2 model architecture + demo bindings
+  config.py                    # model's pipeline config
+  impl/                        # all model-specific implementation
+  tests/                       # model-level tests (stub net + optional real checkpoint)


Suggested change

tests/ # model-level tests (stub net + optional real checkpoint)

tests/ # validate model implementation

ArielG-NV · 2026-09-09T20:19:25Z

+  config.py                    # model's pipeline config
+  impl/                        # all model-specific implementation
+  tests/                       # model-level tests (stub net + optional real checkpoint)
+  apps/<demo>/adapter.py        # create_app() -> IApplication, binds model to an app


Suggested change

apps/<demo>/adapter.py # create_app() -> IApplication, binds model to an app

apps/<demo>/adapter.py # contains all entry point definitions (ex: `create_app`) for a particular `<demo>`

ArielG-NV · 2026-09-09T20:20:14Z

+flashdreams/flashdreams/       # the framework package
+  core/                        # numerical primitives, checkpoint loading, attention, I/O
+  infra/                       # framework contracts: configs, pipelines, encoders/decoders, schedulers, runners
+  recipes/                     # built-in reusable recipe code (WAN, Cosmos, TAEHV, ...)
+  api_v2/                      # protocols an application implements
+  runtime_v2/                  # the two-thread loop that runs one
+  configs/, plugins/, scripts/ # runner registry, plugin discovery, CLI entry points
+
+flashdreams/test_v2/           # v2 engine tests (window, run_session, threads)
+flashdreams/tests/             # framework tests not yet migrated to test_v2/
+tests/                         # repo-wide test-runner scripts + meta checks, not package tests
+docs/source/                   # Sphinx sources


Are these correct?

- Drop the undefined "old package names" phrase. - Apply suggested table wording (Flashdreams Runtime/Protocol, not "Engine loop") consistently across AGENTS.md and CONTRIBUTING.md. - Rename Boundaries -> Dependencies; state apps/integrations_v2 dependency direction explicitly, with the consequence for app tests spelled out and why. Note CI enforcement as a separate follow-up. - Apply suggested File Tree wording verbatim (impl/, config.py, adapter.py, tests/ descriptions). Verified against the actual tree: config.py holds multiple pipeline definitions (checked omnidreams and wan21), every adapter.py exports create_app, tests/ presence pattern holds, no leftover nested apps/ tests/ anywhere, docs/source/ and the flashdreams/flashdreams/* breakdown are accurate. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Set Paing <spaing@nvidia.com>

spaingnv · 2026-09-09T21:04:19Z

Addressed all of these — applied your suggested wording throughout, renamed Boundaries → Dependencies with the apps/integrations_v2 relationship spelled out (CI enforcement noted as a follow-up), dropped the undefined "old package names" phrase. Re-checked docs/source/ against the tree — accurate as written.

One thing I found while verifying the new dependency rule: apps/crazy_robotaxi/tests/test_application.py currently imports omnidreams directly, which the rule says shouldn't happen. Out of scope for this PR (would be a real code fix, not a relocation), flagging it rather than fixing it here.

ArielG-NV · 2026-09-09T21:15:05Z

One thing I found while verifying the new dependency rule: apps/crazy_robotaxi/tests/test_application.py currently imports omnidreams directly, which the rule says shouldn't happen. Out of scope for this PR (would be a real code fix, not a relocation), flagging it rather than fixing it here.

This should be fixed if we want to enforce rules as presented.

ArielG-NV

questions + suggestions + 1 not resolved comment from previous review

ArielG-NV · 2026-09-09T21:16:14Z


 ## Repo Map

- `flashdreams/flashdreams/core/`: reusable numerical primitives, checkpoint loading, distributed helpers, attention, and I/O. Keep it model-agnostic.


Why are we referring to "Ignore .claude/worktrees/ when scanning the source tree; those are nested worktree artifacts, not the repo's current source." and not other AI specific hidden directories?

No good reason. I missed that. It points at .gitignore now instead of naming one tool. Thanks for pointing it out

ArielG-NV · 2026-09-09T21:17:23Z

+- `core` -> `infra` -> `recipes`/`integrations_v2`. `core` and `infra` must not import from `integrations_v2/`; expose a generic config slot or override hook instead of adding model-specific branches. Built-in reusable model pieces belong in `flashdreams/flashdreams/recipes/`; standalone plugin packages belong in `integrations_v2/<name>/`.
+- `apps/<name>/` depends only on `flashdreams` (`core`, `infra`, `api_v2`) — never on `integrations_v2/`. A demo app must run against a stub network; wiring in a real model is the adapter's job, not the app's.
+- `integrations_v2/<model>/` depends on `flashdreams` and on the `apps/<demo>/` it adapts for (via `apps/<demo>/adapter.py`) — never the other way around.


We are specifying dependency chains in 3 different ways. Please keep things consistent.

ArielG-NV · 2026-09-09T21:18:09Z

+
+flashdreams/test_v2/           # Flashdreams Runtime/Protocol tests (window, run_session, threads)
+flashdreams/tests/             # framework tests not yet migrated to test_v2/
+tests/                         # repo-wide test-runner scripts + meta checks, not package tests


https://github.com/NVIDIA/flashdreams/pull/602/changes#r3972653616
was not addressed

Checked against the repo. It's accurate. Also filled in the subpackages that were missing: runtime/, serving/, demo/, accelerated/, quality/, _pytest_plugins/. Sorry, I answered this in a top-level comment last time instead of here, which is why it looked untouched.

… file tree Stop app tests from importing model packages. apps/crazy_robotaxi/tests/ now builds a stub pipeline config from base flashdreams config classes; the tests that genuinely needed OmniDreams internals moved to integrations_v2/omnidreams/tests/test_recipe_configs.py, beside the interactive_drive adapter tests already there. No app test imports a model package now. Nest apps/v2v/ to apps/v2v/v2v/ with a sibling tests/, replacing its package-dir remap with the same auto-discovery every other app uses. It was the last app not matching the documented shape. Fix claims in AGENTS.md that were not true: apps do import integrations_v2 in one live-feature path, and the submodule list omitted runtime_v2, the most-used one. Rename Boundaries to Dependencies and state every direction the same way. Generalize the .claude/worktrees note to gitignored AI-tool directories. Complete the CONTRIBUTING.md file tree with the subpackages it omitted (runtime/, serving/, demo/, accelerated/, quality/, _pytest_plugins/), one entry per line, and note that the five fixture packages do not follow the model shape. Drop remaining references to the deleted v1 integrations/ directory from README.md, docs/README.md, and docs/source/, including a link that 404s and two sentences that only existed to contrast against v1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Set Paing <spaing@nvidia.com>

spaingnv · 2026-09-10T00:58:16Z

Addressed the rest. apps/crazy_robotaxi/tests/ builds a stub pipeline config instead of importing omnidreams, and the tests that needed OmniDreams internals moved to integrations_v2/omnidreams/tests/test_recipe_configs.py, so no app test imports a model package now. apps/v2v/ is nested to apps/v2v/v2v/ as well, so all seven apps match the documented shape.
297 tests pass across apps/ plus the OmniDreams recipe-config suite, ruff clean, docs build warning-free.

ArielG-NV reviewed Sep 9, 2026

View reviewed changes


		Every pytest test must carry exactly one of `ci_cpu`, `ci_gpu`, or `manual`; `CONTRIBUTING.md` has the exact rules. Use module-level `pytestmark = pytest.mark.ci_cpu` for pure Python/metadata tests. Keep GPU, `libGL`/`cv2`, large-checkpoint, credential, and download-heavy checks out of `ci_cpu`.

		v2 test ownership — put a new test next to the thing it validates, not in a mirror of old package names:

	\| Engine loop (window, threads, presentation) \| `flashdreams/test_v2/` \|
	\| Flashdreams Runtime/Protocol (window, threads, presentation) \| `flashdreams/test_v2/` \|

	apps/<slug>/ # v2 interactive app (Drive, T2V, Cam2V, ...)
	apps/<app_slug>/ # apps (Drive, T2V, Cam2V, ...)

	config.py # model's pipeline config
	config.py # collection of pipeline definitions for a particular `<model>`

	impl/ # all model-specific implementation
	impl/ # implementation details of a model

	tests/ # model-level tests (stub net + optional real checkpoint)
	tests/ # validate model implementation

	apps/<demo>/adapter.py # create_app() -> IApplication, binds model to an app
	apps/<demo>/adapter.py # contains all entry point definitions (ex: `create_app`) for a particular `<demo>`


		## Repo Map

		- `flashdreams/flashdreams/core/`: reusable numerical primitives, checkpoint loading, distributed helpers, attention, and I/O. Keep it model-agnostic.

Uh oh!

Conversation

spaingnv commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Why

Not in this PR

Validation

Uh oh!

copy-pr-bot Bot commented Sep 9, 2026

Uh oh!

jmccaffrey-nv commented Sep 9, 2026

Uh oh!

ArielG-NV left a comment

Choose a reason for hiding this comment

Uh oh!

ArielG-NV Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

spaingnv commented Sep 9, 2026

Uh oh!

ArielG-NV left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

spaingnv commented Sep 9, 2026

Uh oh!

ArielG-NV commented Sep 9, 2026

Uh oh!

ArielG-NV left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

spaingnv commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

spaingnv commented Sep 9, 2026 •

edited

Loading

ArielG-NV Sep 9, 2026 •

edited

Loading

ArielG-NV left a comment •

edited

Loading

ArielG-NV left a comment •

edited

Loading

spaingnv commented Sep 10, 2026 •

edited

Loading