diff --git a/AGENTS.md b/AGENTS.md index ad8ce090d..c833005b0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,13 +42,7 @@ Start here, then use the narrower docs for the task in front of you: ## Repo Map -- `flashdreams/flashdreams/core/`: reusable numerical primitives, checkpoint loading, distributed helpers, attention, and I/O. Keep it model-agnostic. -- `flashdreams/flashdreams/infra/`: framework contracts and orchestration for configs, pipelines, encoders, decoders, diffusion models, schedulers, runners, profiling, and CUDA graph wrapping. -- `flashdreams/flashdreams/recipes/`: built-in reusable recipe code such as WAN, Cosmos, TAEHV, and template wiring. -- `flashdreams/flashdreams/configs/`, `plugins/`, and `scripts/`: runner registry, plugin discovery, and CLI entry points. -- `integrations//`: workspace-member model/plugin packages with their own configs, runners, tests, README files, and `pyproject.toml` entry points. -- `docs/source/`: Sphinx sources for quickstart, models, developer guides, API, and community docs. -- `tests/`: root test helpers plus package/integration tests. Ignore `.claude/worktrees/` when scanning the source tree; those are nested worktree artifacts, not the repo's current source. +File structure is in [CONTRIBUTING.md's File Tree Of FlashDreams](CONTRIBUTING.md#file-tree-of-flashdreams). Ignore gitignored AI-tool directories (see `.gitignore`) when scanning the source tree; they hold tool state, not the repo's current source. ## Skill Map @@ -91,9 +85,22 @@ Use `--no-instantiate` before GPU work to inspect the resolved runner config wit Every pytest test must carry exactly one of `ci_cpu`, `ci_gpu`, or `manual`; `CONTRIBUTING.md` has the exact rules. Use module-level `pytestmark = pytest.mark.ci_cpu` for pure Python/metadata tests. Keep GPU, `libGL`/`cv2`, large-checkpoint, credential, and download-heavy checks out of `ci_cpu`. -## Boundaries +**v2 test ownership** — put a new test next to the thing it validates: -Keep dependency direction strict: `core` -> `infra` -> recipes/integrations. `core` and `infra` must not import from `integrations/`; expose a generic config slot or override hook instead of adding model-specific branches. Built-in reusable model pieces belong in `flashdreams/flashdreams/recipes/`; standalone plugin packages belong in `integrations//`. +| You are testing… | Test lives in… | +| --- | --- | +| FlashDreams Runtime/Protocol (window, threads, presentation) | `flashdreams/test_v2/` | +| An app (flags, WASD, physics) | `apps//tests/` | +| A model or its adapter | `integrations_v2//tests/` | + +## Dependencies + +- `infra` depends on `core` — never the other way around. `core` stays model-agnostic. +- `recipes`/`integrations_v2` depend on `infra` and `core` — never the other way around. Expose a generic config slot or override hook in `core`/`infra` instead of adding model-specific branches. Built-in reusable model pieces belong in `flashdreams/flashdreams/recipes/`; standalone plugin packages belong in `integrations_v2//`. +- `apps//` depends on `flashdreams` — never the other way around. An app is written against the framework, not against any one model: it must run against a stub network, and binding a real model is the adapter's job. +- `integrations_v2//` depends on `flashdreams` and on the app it adapts for (via its own `integrations_v2//apps//adapter.py`) — never the other way around. + +Because of this direction, tests in `apps//tests/` must not import from `integrations_v2/` — an app's tests run against a stub, and model-specific checks belong in `integrations_v2//tests/`. CI enforcement of this is a separate follow-up, not yet built. ## Known Pitfalls diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2895bb895..89042f5a4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -18,13 +18,14 @@ issue and we'll fix it. 3. [Developer Certificate of Origin (DCO)](#developer-certificate-of-origin-dco) 4. [Submitting a pull request](#submitting-a-pull-request) 5. [Code review and merge](#code-review-and-merge) -6. [Coding conventions](#coding-conventions) -7. [Testing](#testing) -8. [Dependency version bounds](#dependency-version-bounds) -9. [Working with a single integration package](#working-with-a-single-integration-package) -10. [Licensing of contributions](#licensing-of-contributions) -11. [Reporting issues](#reporting-issues) -12. [Code of Conduct](#code-of-conduct) +6. [File Tree Of FlashDreams](#file-tree-of-flashdreams) +7. [Coding conventions](#coding-conventions) +8. [Testing](#testing) +9. [Dependency version bounds](#dependency-version-bounds) +10. [Working with a single integration package](#working-with-a-single-integration-package) +11. [Licensing of contributions](#licensing-of-contributions) +12. [Reporting issues](#reporting-issues) +13. [Code of Conduct](#code-of-conduct) ## Ways to contribute @@ -223,6 +224,40 @@ We aim for an initial review on every PR within two business days. If your PR has been quiet longer than that, please feel free to leave a short ping comment. +## File Tree Of FlashDreams + +Where a new app or model goes. Each documents its own layout: + +```text +apps/ # reusable apps; layout in apps/README.md +integrations_v2/ # model packages; layout in integrations_v2/README.md +``` + +The framework package, and the test and doc trees: + +```text +flashdreams/flashdreams/ # the framework package + core/ # numerical primitives, checkpoint loading, attention, I/O + infra/ # framework contracts: configs, pipelines, encoders/decoders, schedulers, runners + recipes/ # built-in reusable recipe code (WAN, Cosmos, TAEHV, ...) + api_v2/ # protocols an application implements + runtime_v2/ # the two-thread loop that runs an application + runtime/ # experimental inference runtime API envelope (v0, pre-v2) + serving/ # optional serving utilities (WebRTC, network, launch) + demo/ # transport-neutral application hosting and I/O API + accelerated/ # accelerated kernels (quantization, multi-head attention) + quality/ # output-quality regression utilities (video, CLIP compare) + configs/ # runner registry and CLI aggregator + plugins/ # external-runner plugin layer (RunnerConfig discovery) + scripts/ # console-script entry points (flashdreams-run) + _pytest_plugins/ # pytest plugins (e.g. CI-tier marker enforcement) + +flashdreams/test_v2/ # FlashDreams Runtime/Protocol tests (window, run_session, threads) +flashdreams/tests/ # framework tests not yet migrated to test_v2/ +tests/ # repo-wide test-runner scripts + meta checks, not package tests +docs/source/ # Sphinx sources +``` + ## Coding conventions - Python 3.10+. Type-annotate new code; the project type-checks with @@ -232,9 +267,9 @@ short ping comment. locally is the easiest way to avoid surprises. - Prefer small, well-named functions over long functions with comments explaining each block. Comments should explain *why*, not *what*. -- Tests live in `flashdreams/tests/`, `integrations/*/tests/`, and - `integrations_v2/*/tests/`. Use - `pytest` and prefer existing fixtures over hand-rolled setup. See +- Tests live next to the thing they validate — see the File Tree Of + FlashDreams above. Use `pytest` and prefer existing fixtures over + hand-rolled setup. See [Testing](#testing) for marker requirements. - Every source file added by a contribution must include the SPDX header used elsewhere in the project: @@ -321,7 +356,7 @@ the declared minimums. This means: ## Working with a single integration package -The workspace contains many integration packages under `integrations/`. +The workspace contains many integration packages under `integrations_v2/`. A full `uv sync` installs dependencies for *all* of them. If you only need one (e.g. you're working on `omnidreams`), use the distribution package name with `--package` to sync only that package's dependencies: diff --git a/README.md b/README.md index 70e80b0d6..37207c2fb 100644 --- a/README.md +++ b/README.md @@ -110,7 +110,7 @@ Drive a world model in real time with the unified OmniDreams `local-window` or ## Supported models FlashDreams ships first-party integrations under -[`integrations/`](integrations/). Each model has a dedicated docs page with +[`integrations_v2/`](integrations_v2/). Each model has a dedicated docs page with runner slugs, multi-GPU commands, and (where available) profiling benchmarks. | Model | Family | diff --git a/apps/README.md b/apps/README.md new file mode 100644 index 000000000..eb7abe002 --- /dev/null +++ b/apps/README.md @@ -0,0 +1,41 @@ + + +Reusable applications built on the v2 API. Each directory is a standalone +package that depends on `flashdreams` and stays model-agnostic: an app runs +against a stub network, and binding a real model is the adapter's job, in +`integrations_v2//apps//adapter.py`. + +## The layout + +```text +apps// + / # the app implementation + tests/ # validate the app against a stub, not a real model + pyproject.toml + README.md # purpose, controls, example command line, options +``` + +Tests live in `apps//tests/`, beside the package rather than inside +it, and must not import from `integrations_v2/`. A check that needs a real +model or its adapter belongs in `integrations_v2//tests/`. + +## What is here + +- `t2v` — text-to-video: prompt in, frames out. The reference app to copy. +- `cam2v` — interactive camera-to-video. +- `action2v` — world models driven by direct action input. +- `v2v` — video-to-video, used for super-resolution. +- `interactive_drive` — the interactive driving demo, keyboard and wheel input. +- `crazy_robotaxi` — the driving game built on `omnidreams_game_engine`. +- `omnidreams_game_engine` — reusable simulation, authored-map, physics, and + conditioning components. It owns no runtime loop, so it is a library the + other apps build on rather than something you launch. + +`omnidreams_game_engine` is not the OmniDreams model. That is +`integrations_v2/omnidreams`. The names are close; the packages are unrelated. + +Each app's own README covers how to launch it and what the controls are. To +bind a model to one of these, read `integrations_v2/README.md`. diff --git a/apps/crazy_robotaxi/tests/test_application.py b/apps/crazy_robotaxi/tests/test_application.py index d16b7b1e4..4ac2c4def 100644 --- a/apps/crazy_robotaxi/tests/test_application.py +++ b/apps/crazy_robotaxi/tests/test_application.py @@ -3,7 +3,7 @@ """CPU tests for Crazy Robotaxi's application boundary against FlashDreams V2.""" -from dataclasses import replace +from dataclasses import dataclass, field, replace from pathlib import Path from types import SimpleNamespace from typing import Any, cast @@ -29,30 +29,6 @@ _taxi_driver_command, ) from crazy_robotaxi.ui import CrazyRobotaxiImGuiUILoop -from omnidreams.apps.crazy_robotaxi.adapter import ( - OMNIDREAMS_CRAZY_ROBOTAXI_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_RESPONSIVE_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_RESPONSIVE_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_RESPONSIVE_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_PERF_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_PERF_RESPONSIVE_DEFAULTS, - OMNIDREAMS_CRAZY_ROBOTAXI_RESPONSIVE_DEFAULTS, -) -from omnidreams.config import ( - OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG, - OMNIDREAMS_FAST_PERF_RESPONSIVE_PIPELINE_CONFIG, - OMNIDREAMS_OPTIMIZED_GB300_PIPELINE_CONFIG, - OMNIDREAMS_OPTIMIZED_GB300_RESPONSIVE_PIPELINE_CONFIG, - OMNIDREAMS_OPTIMIZED_RTX_PRO_6000_PIPELINE_CONFIG, - OMNIDREAMS_OPTIMIZED_RTX_PRO_6000_RESPONSIVE_PIPELINE_CONFIG, - OMNIDREAMS_PERF_PIPELINE_CONFIG, - OMNIDREAMS_PERF_RESPONSIVE_PIPELINE_CONFIG, - OMNIDREAMS_PIPELINE_CONFIG, - OMNIDREAMS_RESPONSIVE_PIPELINE_CONFIG, -) from omnidreams_game_engine.config import BevConfig, RasterConfig from omnidreams_game_engine.input import DriverInput from omnidreams_game_engine.renderer_settings import RendererSettings @@ -63,11 +39,12 @@ SceneDefinition, ) -from flashdreams.runtime_v2.native_window_client_window import ( - NativeWindowClientWindow, -) +from flashdreams.infra.diffusion.model import DiffusionModelConfig +from flashdreams.infra.diffusion.scheduler.base import SchedulerConfig +from flashdreams.infra.diffusion.transformer.base import TransformerConfig +from flashdreams.infra.encoder.base import EncoderConfig +from flashdreams.infra.pipeline import StreamInferencePipelineConfig from flashdreams.runtime_v2.session_desc import PresentationMode -from flashdreams.runtime_v2.step_result import StepResult from flashdreams.runtime_v2.user_input_event import ( GamepadUserInputEvent, KeyboardInputState, @@ -86,9 +63,58 @@ ) +@dataclass(kw_only=True) +class _StubTransformerConfig(TransformerConfig): + """Adds the fields CrazyRobotaxiApplication logs unconditionally. + + Placeholder values only; no test in this file inspects them (tests that + care about real acceleration/backend settings live under + integrations_v2/omnidreams/tests/, since apps/ must not import + integrations_v2/). + """ + + native_dit_acceleration: str | None = None + native_dit_backend: str | None = None + native_dit_attention_backend: str | None = None + skip_finalize_kv_cache: bool = False + compile_network: bool = False + + +@dataclass(kw_only=True) +class _StubEncoderConfig(EncoderConfig): + """Adds the fields CrazyRobotaxiApplication logs unconditionally.""" + + native_vae_acceleration: str | None = None + native_vae_backend: str | None = None + + +@dataclass(kw_only=True) +class _StubSchedulerConfig(SchedulerConfig): + """Adds the field CrazyRobotaxiApplication logs unconditionally.""" + + denoising_timesteps: list[int] = field(default_factory=list) + + +_STUB_PIPELINE_CONFIG = StreamInferencePipelineConfig( + name="crazy-robotaxi-test-stub", + diffusion_model=DiffusionModelConfig( + transformer=_StubTransformerConfig(), + scheduler=_StubSchedulerConfig(), + ), + encoder=_StubEncoderConfig(), +) +"""A pipeline config with no model behind it, built from base flashdreams +config classes only. CrazyRobotaxiApplication reads ``.name``, derives +``.enable_sync_and_profile``, and logs several transformer/encoder fields +unconditionally, so app-level tests need a real, structured pipeline config, +not a real *model*.""" + +_STUB_DEFAULTS = CrazyRobotaxiApplicationDefaults(pipeline_config=_STUB_PIPELINE_CONFIG) + + def _application( *, - defaults: CrazyRobotaxiApplicationDefaults = OMNIDREAMS_CRAZY_ROBOTAXI_DEFAULTS, + defaults: CrazyRobotaxiApplicationDefaults = _STUB_DEFAULTS, **kwargs: Any, ) -> CrazyRobotaxiApplication: return CrazyRobotaxiApplication(defaults=defaults, **kwargs) @@ -464,70 +490,6 @@ def test_pipeline_profiling_is_an_app_local_opt_in( assert configured[0].enable_sync_and_profile is expected assert app._config is not None assert app._config.pipeline_profiling is expected - assert OMNIDREAMS_PIPELINE_CONFIG.enable_sync_and_profile - - -def test_model_adapters_keep_their_packaged_pipeline_configs() -> None: - for defaults, pipeline_config in ( - (OMNIDREAMS_CRAZY_ROBOTAXI_DEFAULTS, OMNIDREAMS_PIPELINE_CONFIG), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_PERF_DEFAULTS, - OMNIDREAMS_PERF_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS, - OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_DEFAULTS, - OMNIDREAMS_OPTIMIZED_GB300_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_DEFAULTS, - OMNIDREAMS_OPTIMIZED_RTX_PRO_6000_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_RESPONSIVE_DEFAULTS, - OMNIDREAMS_RESPONSIVE_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_PERF_RESPONSIVE_DEFAULTS, - OMNIDREAMS_PERF_RESPONSIVE_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_RESPONSIVE_DEFAULTS, - OMNIDREAMS_FAST_PERF_RESPONSIVE_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_RESPONSIVE_DEFAULTS, - OMNIDREAMS_OPTIMIZED_GB300_RESPONSIVE_PIPELINE_CONFIG, - ), - ( - OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_RESPONSIVE_DEFAULTS, - OMNIDREAMS_OPTIMIZED_RTX_PRO_6000_RESPONSIVE_PIPELINE_CONFIG, - ), - ): - assert defaults.pipeline_config is pipeline_config - - -def test_fast_perf_combines_native_dit_and_native_vae_paths() -> None: - pipeline: Any = OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG - perf_pipeline: Any = OMNIDREAMS_PERF_PIPELINE_CONFIG - assert pipeline.name == "omnidreams-fast-perf" - assert pipeline.diffusion_model.seed is None - assert pipeline.decoder.use_compile is perf_pipeline.decoder.use_compile - assert pipeline.decoder.use_cuda_graph is True - assert pipeline.image_encoder.native_vae_acceleration == "required" - assert pipeline.image_encoder.native_vae_backend == "fp8" - assert pipeline.image_encoder.native_vae_fp8_auto_export is True - assert pipeline.encoder.native_vae_acceleration == "required" - assert pipeline.encoder.native_vae_backend == "fp8" - assert pipeline.encoder.native_vae_fp8_auto_export is True - assert pipeline.diffusion_model.transformer.native_dit_acceleration == "required" - assert ( - pipeline.diffusion_model.transformer.native_dit_backend == "fp8_kvcache_cudnn" - ) - assert pipeline.diffusion_model.transformer.native_dit_attention_backend == "cudnn" @pytest.mark.parametrize("resolution_wh", [(1280, 704), (1168, 640)]) @@ -548,7 +510,7 @@ def load_test_scene(request: object, raster: RasterConfig) -> SceneDefinition: app = _application( defaults=replace( - OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS, + _STUB_DEFAULTS, width=resolution_wh[0], height=resolution_wh[1], ), @@ -584,51 +546,6 @@ def load_test_scene(request: object, raster: RasterConfig) -> SceneDefinition: ) -def test_fast_perf_honors_explicit_pipeline_overrides() -> None: - app = _application( - defaults=OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS, - ) - - app.init( - [ - "--seed", - "7", - "--no-compile", - "--profile-pipeline", - ] - ) - - pipeline = cast(Any, app._pipeline_config) - transformer = pipeline.diffusion_model.transformer - assert pipeline.diffusion_model.seed == 7 - assert transformer.compile_network is False - assert transformer.native_dit_acceleration == "required" - assert transformer.skip_finalize_kv_cache is True - assert pipeline.diffusion_model.scheduler.denoising_timesteps == [1000, 100] - assert pipeline.enable_sync_and_profile is True - - -def test_map_context_disables_only_native_dit_on_selected_preset() -> None: - app = _application(defaults=OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS) - - app.init(["--live-edit-map-context"]) - - pipeline = cast(Any, app._pipeline_config) - original: Any = OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG - transformer = pipeline.diffusion_model.transformer - assert app._config is not None - assert app._config.scene_request.use_prompt_context - assert pipeline.name == original.name - assert transformer.native_dit_acceleration == "disabled" - assert transformer.native_dit_backend == ( - original.diffusion_model.transformer.native_dit_backend - ) - assert transformer.skip_finalize_kv_cache is True - assert pipeline.diffusion_model.scheduler == original.diffusion_model.scheduler - assert pipeline.image_encoder.native_vae_acceleration == "required" - assert pipeline.encoder.native_vae_acceleration == "required" - - def test_bev_render_fit_preserves_authored_aspect_ratio_and_smaller_sources() -> None: raster = RasterConfig() wide = RendererSettings(raster=raster, bev=BevConfig(width=800, height=400)) diff --git a/apps/interactive_drive/interactive_drive/tests/output/synthetic_scene_fixture.usdz b/apps/interactive_drive/tests/output/synthetic_scene_fixture.usdz similarity index 100% rename from apps/interactive_drive/interactive_drive/tests/output/synthetic_scene_fixture.usdz rename to apps/interactive_drive/tests/output/synthetic_scene_fixture.usdz diff --git a/apps/interactive_drive/interactive_drive/tests/test_application.py b/apps/interactive_drive/tests/test_application.py similarity index 100% rename from apps/interactive_drive/interactive_drive/tests/test_application.py rename to apps/interactive_drive/tests/test_application.py diff --git a/apps/interactive_drive/interactive_drive/tests/test_rasterizer_projection.py b/apps/interactive_drive/tests/test_rasterizer_projection.py similarity index 100% rename from apps/interactive_drive/interactive_drive/tests/test_rasterizer_projection.py rename to apps/interactive_drive/tests/test_rasterizer_projection.py diff --git a/apps/v2v/pyproject.toml b/apps/v2v/pyproject.toml index 994f81e1b..502b5c8c7 100644 --- a/apps/v2v/pyproject.toml +++ b/apps/v2v/pyproject.toml @@ -26,9 +26,8 @@ dev = [ [tool.uv.sources] flashdreams = { workspace = true } -[tool.setuptools] -packages = ["v2v"] -package-dir = { v2v = "." } +[tool.setuptools.packages.find] +include = ["v2v*"] [tool.uv] managed = true diff --git a/apps/v2v/__init__.py b/apps/v2v/v2v/__init__.py similarity index 100% rename from apps/v2v/__init__.py rename to apps/v2v/v2v/__init__.py diff --git a/apps/v2v/v2v.py b/apps/v2v/v2v/v2v.py similarity index 100% rename from apps/v2v/v2v.py rename to apps/v2v/v2v/v2v.py diff --git a/docs/README.md b/docs/README.md index 6aaa4b25b..dc7a3f7b9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -108,5 +108,5 @@ to be present. - **A new API category** — add `source/api/.rst`, then include it in the API toctree in `source/index.rst`. - **Plugin-first note** — most actively developed integrations live under - `integrations//`. Use `source/api/integrations.rst` to document + `integrations_v2//`. Use `source/api/integrations.rst` to document in-tree `flashdreams.recipes.*` API surface that remains public. diff --git a/docs/source/api/integrations.rst b/docs/source/api/integrations.rst index 68ad0ad60..8f3745d3f 100644 --- a/docs/source/api/integrations.rst +++ b/docs/source/api/integrations.rst @@ -14,7 +14,7 @@ .. limitations under the License. Pipelines, runners, and applications -=================================== +==================================== FlashDreams model integrations use these public layers: @@ -23,18 +23,17 @@ FlashDreams model integrations use these public layers: - **V2 applications** (``IApplication``) that bind reusable demo infrastructure directly to pipeline configs. -Most actively developed model implementations now live under ``integrations/*`` -as plugin-style standalone packages. This page keeps documenting the in-tree -pipeline modules that are still exposed from ``flashdreams.recipes``. +Most actively developed model implementations now live under +``integrations_v2//`` as plugin-style standalone packages. This page +keeps documenting the in-tree pipeline modules that are still exposed from +``flashdreams.recipes``. .. note:: Pipeline modules import the heavy GPU stack (transformer-engine, CUDA ops) at import time, so this page shows them by *automodule* with ``:no-undoc-members:`` to keep the rendered API focused on the names - that these in-tree modules actually expose. The unified ``flashdreams-run`` - CLI shows end-to-end usage; see :doc:`/models/index` for model launch - examples. + that these in-tree modules actually expose. Integration structure (current) ------------------------------- @@ -51,9 +50,7 @@ V2 demo ports follow ``integrations_v2//``: Apart from that unique config, the integration root contains no implementation modules. These packages do not add ``runner.py`` or -``flashdreams.runner_configs`` just to launch a v2 demo. Older plugin-style -integrations under ``integrations/`` may still expose runners through -``flashdreams-run``. +``flashdreams.runner_configs`` just to launch a v2 demo. The default application entry point uses ``-`` and ``create_app``. Additional compatible diff --git a/docs/source/community/faq.rst b/docs/source/community/faq.rst index 8d71d04a4..3e1257709 100644 --- a/docs/source/community/faq.rst +++ b/docs/source/community/faq.rst @@ -41,7 +41,7 @@ methods. Which model integrations ship in the box? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -First-party model integrations ship under ``integrations/`` in the +First-party model integrations ship under ``integrations_v2//`` in the repo. The :doc:`/models/index` page has the full list; the documented integrations are: @@ -69,7 +69,7 @@ Only the core ``flashdreams`` package is published as a pure-Python wheel on PyPI. Integration packages — ``flashdreams-self-forcing``, ``flashdreams-lingbot``, and the others listed in `DEV.md `__ — -are not published; they live under ``integrations/`` in the monorepo +are not published; they live under ``integrations_v2//`` in the monorepo and are designed to be consumed either as a workspace member or as git-installable packages. @@ -97,7 +97,7 @@ How do I plug in a new model integration? The :doc:`/developer_guides/new_integration` guide walks the full flow — what to subclass on the runner side, how the entry-point registration works, and what the per-integration directory layout -looks like. The in-tree integrations under ``integrations/`` +looks like. The in-tree integrations under ``integrations_v2/`` are the canonical references; pick the one closest in shape to your new integration and use it as a template. diff --git a/docs/source/developer_guides/new_integration.rst b/docs/source/developer_guides/new_integration.rst index 40147df04..cda2d937a 100644 --- a/docs/source/developer_guides/new_integration.rst +++ b/docs/source/developer_guides/new_integration.rst @@ -18,7 +18,7 @@ Add a new method Before you start adding a new method, we highly recommend reading the :doc:`/developer_guides/inference_pipeline_overview` and :doc:`/developer_guides/config_system` pages first to obtain an overview of the system architecture. -FlashDreams aims to offer researchers a codebase that they can utilize to extend and develop novel video and world models. Our vision is for users to establish a *standalone repository* that imports FlashDreams as a dependency and overrides pipeline components (such as encoders, transformers, or decoders) to cater to specific functionality requirements of the new approach. We encourage you to maintain your method externally rather than pushing changes directly into the `integrations/ `_ directory of this repository. +FlashDreams aims to offer researchers a codebase that they can utilize to extend and develop novel video and world models. Our vision is for users to establish a *standalone repository* that imports FlashDreams as a dependency and overrides pipeline components (such as encoders, transformers, or decoders) to cater to specific functionality requirements of the new approach. We encourage you to maintain your method externally rather than pushing changes directly into the `integrations_v2/ `_ directory of this repository. However, if any of your new features require modifications to the core FlashDreams infra or introduce generally useful components (such as the :mod:`TAEHV decoder `), we encourage you to submit a PR to enable others to benefit from them. diff --git a/flashdreams/test_v2/README.md b/flashdreams/test_v2/README.md index e18bcf326..1e1988fd8 100644 --- a/flashdreams/test_v2/README.md +++ b/flashdreams/test_v2/README.md @@ -26,20 +26,20 @@ CPU-only tests for the v2 protocols themselves: skipped when the serving packages are missing, which is also why a run writing a file does not import them. - [`apps/t2v/tests`](../../apps/t2v/tests) covers the reusable text-to-video - application, session, model loop, and stand-in model checks. -- `test_cli.py` covers `flashdreams-run-v2`: finding an application, splitting - the command line at `--`, choosing a window, describing the session to ask for, - and running one into a real MP4 with a stand-in for a model. An application - that describes no session of its own is run there too, since running more than + application, session, model loop, and stand-in model checks. Its + [`test_cli.py`](../../apps/t2v/tests/test_cli.py) also covers + `flashdreams-run-v2` itself: finding an application, splitting the command + line at `--`, choosing a window, describing the session to ask for, and + running one into a real MP4 with a stand-in for a model. An application that + describes no session of its own is run there too, since running more than text-to-video is the point of the command. - `test_metrics_output_sink.py` covers the sink that records what a run measured, which is a file another tool reads: what a benchmark expects of it is checked against the reader itself in `flashdreams/tests/test_benchmark_harness.py`. -Application behaviour is tested by the application that owns it — see -`integrations_v2/red_screen/red_screen/tests/` and -`integrations_v2/color_fade/color_fade/tests/`. +Reusable apps keep their tests beside the package, in `apps//tests/` +(see `apps/t2v/tests/` above, and `apps/interactive_drive/tests/`). Run commands from the repository root. diff --git a/integrations_v2/README.md b/integrations_v2/README.md index 049e11003..6774d6c95 100644 --- a/integrations_v2/README.md +++ b/integrations_v2/README.md @@ -18,6 +18,7 @@ follows is already done for you. - `red_screen` — the smallest interactive one, streaming to a browser. - `slangpy_ui_demo` — three applications that draw widgets over model output, and the reference for writing a UI loop. +- `imgui_ui_demo` — an editable ImGui text field rendered over model output. - `lingbot` — the Lingbot World model and its `cam2v-lingbot` binding to the shared interactive camera-to-video application. - `waypoint` — the Waypoint model binding to the shared `apps/action2v` @@ -43,13 +44,13 @@ integrations_v2// pyproject.toml README.md __init__.py - config.py # model's unique pipeline config or config wrapper - impl/ # all model-specific implementation - tests/ # model-specific tests, when needed + config.py # collection of pipeline definitions for a particular `` + impl/ # implementation details of a model + tests/ # validate model implementation apps/ / __init__.py - adapter.py # create_app() -> IApplication + adapter.py # all entry point definitions for `` (ex: `create_app`) README.md # launch instructions only ``` diff --git a/integrations_v2/omnidreams/tests/test_recipe_configs.py b/integrations_v2/omnidreams/tests/test_recipe_configs.py index 264611afe..98c138d98 100644 --- a/integrations_v2/omnidreams/tests/test_recipe_configs.py +++ b/integrations_v2/omnidreams/tests/test_recipe_configs.py @@ -5,11 +5,24 @@ from collections.abc import Callable from pathlib import Path -from typing import cast +from typing import Any, cast import pytest import tomli as tomllib +from crazy_robotaxi.application import CrazyRobotaxiApplication from interactive_drive import InteractiveDriveApplication, InteractiveDriveConfig +from omnidreams.apps.crazy_robotaxi.adapter import ( + OMNIDREAMS_CRAZY_ROBOTAXI_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_RESPONSIVE_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_RESPONSIVE_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_RESPONSIVE_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_PERF_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_PERF_RESPONSIVE_DEFAULTS, + OMNIDREAMS_CRAZY_ROBOTAXI_RESPONSIVE_DEFAULTS, +) from omnidreams.apps.interactive_drive.adapter import ( OMNIDREAMS_INTERACTIVE_DRIVE_DEFAULTS, OMNIDREAMS_INTERACTIVE_DRIVE_FAST_PERF_DEFAULTS, @@ -200,10 +213,123 @@ def test_application_defaults_are_owned_by_each_adapter() -> None: OMNIDREAMS_INTERACTIVE_DRIVE_FAST_PERF_DEFAULTS, OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG, ), + (OMNIDREAMS_CRAZY_ROBOTAXI_DEFAULTS, OMNIDREAMS_PIPELINE_CONFIG), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_PERF_DEFAULTS, + OMNIDREAMS_PERF_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS, + OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_DEFAULTS, + OMNIDREAMS_OPTIMIZED_GB300_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_DEFAULTS, + OMNIDREAMS_OPTIMIZED_RTX_PRO_6000_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_RESPONSIVE_DEFAULTS, + OMNIDREAMS_RESPONSIVE_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_PERF_RESPONSIVE_DEFAULTS, + OMNIDREAMS_PERF_RESPONSIVE_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_RESPONSIVE_DEFAULTS, + OMNIDREAMS_FAST_PERF_RESPONSIVE_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_GB300_RESPONSIVE_DEFAULTS, + OMNIDREAMS_OPTIMIZED_GB300_RESPONSIVE_PIPELINE_CONFIG, + ), + ( + OMNIDREAMS_CRAZY_ROBOTAXI_OPTIMIZED_RTX_PRO_6000_RESPONSIVE_DEFAULTS, + OMNIDREAMS_OPTIMIZED_RTX_PRO_6000_RESPONSIVE_PIPELINE_CONFIG, + ), ): assert defaults.pipeline_config is pipeline_config +def test_fast_perf_combines_native_dit_and_native_vae_paths() -> None: + """Moved from apps/crazy_robotaxi/tests/test_application.py: pure OmniDreams + pipeline-config assertions, no Crazy Robotaxi app involved.""" + pipeline: Any = OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG + perf_pipeline: Any = OMNIDREAMS_PERF_PIPELINE_CONFIG + assert pipeline.name == "omnidreams-fast-perf" + assert pipeline.diffusion_model.seed is None + assert pipeline.decoder.use_compile is perf_pipeline.decoder.use_compile + assert pipeline.decoder.use_cuda_graph is True + assert pipeline.image_encoder.native_vae_acceleration == "required" + assert pipeline.image_encoder.native_vae_backend == "fp8" + assert pipeline.image_encoder.native_vae_fp8_auto_export is True + assert pipeline.encoder.native_vae_acceleration == "required" + assert pipeline.encoder.native_vae_backend == "fp8" + assert pipeline.encoder.native_vae_fp8_auto_export is True + assert pipeline.diffusion_model.transformer.native_dit_acceleration == "required" + assert ( + pipeline.diffusion_model.transformer.native_dit_backend == "fp8_kvcache_cudnn" + ) + assert pipeline.diffusion_model.transformer.native_dit_attention_backend == "cudnn" + + +def test_crazy_robotaxi_fast_perf_honors_explicit_pipeline_overrides() -> None: + """Moved from apps/crazy_robotaxi/tests/test_application.py: tests that Crazy + Robotaxi's CLI parsing correctly mutates OmniDreams's pipeline config, which + is inherently an adapter-level (app x model) concern.""" + app = CrazyRobotaxiApplication( + defaults=OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS + ) + + app.init( + [ + "--seed", + "7", + "--no-compile", + "--profile-pipeline", + ] + ) + + pipeline = cast(Any, app._pipeline_config) + transformer = pipeline.diffusion_model.transformer + assert pipeline.diffusion_model.seed == 7 + assert transformer.compile_network is False + assert transformer.native_dit_acceleration == "required" + assert transformer.skip_finalize_kv_cache is True + assert pipeline.diffusion_model.scheduler.denoising_timesteps == [1000, 100] + assert pipeline.enable_sync_and_profile is True + + +def test_crazy_robotaxi_map_context_disables_only_native_dit_on_selected_preset() -> ( + None +): + """Moved from apps/crazy_robotaxi/tests/test_application.py; same reasoning + as test_crazy_robotaxi_fast_perf_honors_explicit_pipeline_overrides.""" + app = CrazyRobotaxiApplication( + defaults=OMNIDREAMS_CRAZY_ROBOTAXI_FAST_PERF_DEFAULTS + ) + + app.init(["--live-edit-map-context"]) + + pipeline = cast(Any, app._pipeline_config) + original: Any = OMNIDREAMS_FAST_PERF_PIPELINE_CONFIG + transformer = pipeline.diffusion_model.transformer + assert app._config is not None + assert app._config.scene_request.use_prompt_context + assert pipeline.name == original.name + assert transformer.native_dit_acceleration == "disabled" + assert transformer.native_dit_backend == ( + original.diffusion_model.transformer.native_dit_backend + ) + assert transformer.skip_finalize_kv_cache is True + assert pipeline.diffusion_model.scheduler == original.diffusion_model.scheduler + assert pipeline.image_encoder.native_vae_acceleration == "required" + assert pipeline.encoder.native_vae_acceleration == "required" + + @pytest.mark.parametrize( ("factory", "resolution_wh"), [ diff --git a/pyproject.toml b/pyproject.toml index 7798b5be8..1535140cd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,13 +1,9 @@ [tool.uv.workspace] members = [ "flashdreams", - # Each populated subdir under ``integrations/`` or ``integrations_v2/`` - # is a standalone external integration repo (e.g. ``causal_forcing`` / - # ``self_forcing`` - # plugins, ``omnidreams`` / ``lingbot`` integrations); empty stub - # folders alongside them are intentional placeholders for future - # extractions and don't ship a ``pyproject.toml`` yet. - "integrations/*", + # Each populated subdir under ``integrations_v2/`` is a standalone + # external integration repo (e.g. ``causal_forcing`` / ``self_forcing`` + # plugins, ``omnidreams`` / ``lingbot`` integrations). "integrations_v2/*", "apps/*", # Nested sub-packages that the integration globs do not reach. @@ -16,7 +12,6 @@ members = [ exclude = [ "apps/__init__.py", "apps/README.md", - "integrations/omnidreams", "integrations_v2/README.md", ] @@ -126,6 +121,8 @@ extra-paths = [ exclude = [ "**/protos/*pb2*", "apps/interactive_drive/interactive_drive/**", + # Same files as above; tests live beside the package, not inside it. + "apps/interactive_drive/tests/**", # These map/archive parsers and numerical PhysX adapters intentionally # accept dynamically shaped YAML, Arrow, and NumPy values. Keep the new # V2 application, session, engine contracts, rollout, input, and renderer