From cca26e97b3e0b2c3027105edbeb61458bd10cd12 Mon Sep 17 00:00:00 2001 From: Zhiyu Li Date: Tue, 28 Jul 2026 14:57:19 -0700 Subject: [PATCH 01/52] feat(swe): nano SWE recipe with honoured TransferQueue data plane Reproducible 6-node GB200 smoke that runs agentic SWE RL on Nemotron-3-Nano-30B-A3B with rollouts flowing through TransferQueue. Verified (Slurm 5648757): run_grpo_single_controller.py + nano_swe_teacher_sc.yaml completed train steps 1-5 with live TransferQueueController + 2 SimpleStorageUnit actors serving PUT_DATA / KV_RETRIEVE_META, no traceback and no OOM. Rewards are 0.0 throughout -- the base model solves no SWE instance in a 5-step smoke, so this validates the loop, not model quality. Only two entrypoints honour the data plane: grpo_train_sync and run_grpo_single_controller.py. run_grpo_nemo_gym.py's async path builds the in-memory ReplayBuffer (grpo.py:3898) and silently ignores data_plane, so async + real TQ means SingleController. Contents: - examples/configs/ultra/nano_swe_teacher_sc.yaml -- the recipe, plus its inheritance chain (swe_teacher, _nano_smoke_gb200.inc, nano_swe_teacher, nano_swe_teacher_qwen3mesh) and the sync+TQ variant - launchers: swe_nano.env plus interactive and batch wrappers over ultra_launch.sh, which gains an NRL_ENTRYPOINT override so a run can use run_grpo_single_controller.py - docs/guides/nano-swe-transferqueue.md -- setup, the read-only paths to reuse vs the per-user write paths to change, why each SingleController setting exists, and how to confirm TQ is actually engaged - nemo_rl/utils/config.py: skip mandatory (???) values during config merge. Required here: swe_teacher.yaml declares sif_dir: ??? and resolving it mid-merge raises MissingMandatoryValue before the child config supplies the value. - pin 3rdparty/Gym-workspace/Gym to v0.4.0 to match the container's prebuilt gym venvs (skip_venv_if_present) Signed-off-by: Zhiyu Li --- 3rdparty/Gym-workspace/Gym | 2 +- docs/guides/nano-swe-transferqueue.md | 206 ++++ docs/index.md | 8 + .../configs/ultra/_nano_smoke_gb200.yaml.inc | 97 ++ examples/configs/ultra/nano_swe_teacher.yaml | 36 + .../ultra/nano_swe_teacher_qwen3mesh.yaml | 71 ++ .../configs/ultra/nano_swe_teacher_sc.yaml | 58 ++ .../ultra/nano_swe_teacher_sync_tq.yaml | 46 + examples/configs/ultra/swe_teacher.yaml | 478 +++++++++ swe_nano.env | 110 +++ swe_nano_interactive.sh | 37 + swe_nano_sc.sh | 39 + swe_nano_sc_interactive.sh | 48 + swe_nano_tq_interactive.sh | 35 + ultra_launch.sh | 910 ++++++++++++++++++ 15 files changed, 2180 insertions(+), 1 deletion(-) create mode 100644 docs/guides/nano-swe-transferqueue.md create mode 100644 examples/configs/ultra/_nano_smoke_gb200.yaml.inc create mode 100644 examples/configs/ultra/nano_swe_teacher.yaml create mode 100644 examples/configs/ultra/nano_swe_teacher_qwen3mesh.yaml create mode 100644 examples/configs/ultra/nano_swe_teacher_sc.yaml create mode 100644 examples/configs/ultra/nano_swe_teacher_sync_tq.yaml create mode 100644 examples/configs/ultra/swe_teacher.yaml create mode 100644 swe_nano.env create mode 100644 swe_nano_interactive.sh create mode 100644 swe_nano_sc.sh create mode 100644 swe_nano_sc_interactive.sh create mode 100644 swe_nano_tq_interactive.sh create mode 100755 ultra_launch.sh diff --git a/3rdparty/Gym-workspace/Gym b/3rdparty/Gym-workspace/Gym index bb9b26de3da..b6051536d9a 160000 --- a/3rdparty/Gym-workspace/Gym +++ b/3rdparty/Gym-workspace/Gym @@ -1 +1 @@ -Subproject commit bb9b26de3dac1294faccccb5ed70f3386c16de10 +Subproject commit b6051536d9ac569c853200051accf81ad8af0085 diff --git a/docs/guides/nano-swe-transferqueue.md b/docs/guides/nano-swe-transferqueue.md new file mode 100644 index 00000000000..97f2d7299d1 --- /dev/null +++ b/docs/guides/nano-swe-transferqueue.md @@ -0,0 +1,206 @@ +# Nano SWE RL with the TransferQueue Data Plane + +A reproducible 6-node smoke recipe that runs agentic SWE RL on +Nemotron-3-Nano-30B-A3B with rollouts flowing through the **TransferQueue (TQ) +data plane** instead of an in-process buffer. + +It exists to answer one question end to end: *which NeMo-RL entrypoint actually +puts SWE rollouts through TransferQueue, and what does a config need for that to +work?* The answer is `examples/run_grpo_single_controller.py`, and this recipe is +the smallest configuration where you can watch it happen. + +## Verified result + +Run on 6 GB200 NVL72 nodes (Slurm job 5648757, 2026-07-28): + +| | | +|---|---| +| Entrypoint | `examples/run_grpo_single_controller.py` | +| Config | `examples/configs/ultra/nano_swe_teacher_sc.yaml` | +| Model | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` | +| Shape | train 4 nodes (TP2·PP2·CP4, EP1) / gen 2 nodes (vLLM TP2 → 4 engines) | +| Overrides | `grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 grpo.max_num_steps=5` | +| Outcome | train steps 1–5 completed, no traceback, no OOM | +| Launched via | `swe_nano_sc_interactive.sh` (attach + run the driver by hand); `swe_nano_sc.sh` submits the same driver command as a batch job | + +The TQ actors were live and serving the rollout→train hop throughout: +`TransferQueueController` plus two `SimpleStorageUnit` actors logging `PUT_DATA`, +`GET_DATA`, `KV_RETRIEVE_META`, `SET_CUSTOM_META` and `CLEAR_DATA`. + +The final step reported `loss=7.7e-05`, `grad_norm=0.013`, +`global_valid_toks=112555`. **Rewards were 0.0 for every rollout** — the base +Nano model solves no SWE-bench instance in a 5-step smoke. This recipe validates +the *loop*, not model quality. + +## Which paths honour the data plane + +This is the part that is easy to get wrong: enabling `data_plane` in a config +does not mean it is used. Verified against the source: + +| Entrypoint / mode | TQ honoured? | Why | +|---|---|---| +| `run_grpo_nemo_gym.py`, `async_grpo.enabled=true` | **No** | `async_grpo_train` builds the in-memory `ReplayBuffer` (`nemo_rl/algorithms/grpo.py:3898`); the TQ-backed `TQReplayBuffer` (`nemo_rl/algorithms/async_utils/replay_buffer.py:638`) is never constructed. The `data_plane` block is silently a no-op. | +| `run_grpo_nemo_gym.py`, `async_grpo.enabled=false` | Yes | `grpo_train_sync` reads/writes through the data plane. | +| `run_grpo_single_controller.py` | Yes, and **required** | The entrypoint raises `ValueError` unless `data_plane.enabled=true`, and `SingleControllerActor` commits each group via `TQReplayBuffer` → `dp_client.put_samples` (`nemo_rl/algorithms/single_controller.py:259`). | + +Since the production SWE paradigm is async (sync leaves the 16 training GPUs +idle for the ~11 minutes a 64-rollout SWE batch takes), **SingleController is the +only path that gives async *and* a real data plane** — hence the recipe below. + +## Quick start + +All launchers share `swe_nano.env` and override the entrypoint and config. Run +them from a networked shell at the repo root (the Slurm controller is +unreachable from sandboxes). + +Unattended reproduction, one command: + +```bash +DRY_RUN=0 bash swe_nano_sc.sh grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 +``` + +Without `DRY_RUN=0` it prints the resolved driver command and exits, which is +worth doing once. Interactive mode — allocate, keep Ray up, run the driver by +hand, edit, re-run — is what the verified run used and is the better choice +whenever you expect to change anything: + +```bash +# async GRPO via SingleController + honoured TransferQueue (the verified recipe) +bash swe_nano_sc_interactive.sh + +# sync GRPO + honoured TransferQueue (simplest data-plane smoke) +bash swe_nano_tq_interactive.sh + +# async GRPO, no data plane (baseline) +bash swe_nano_interactive.sh +``` + +The batch and interactive paths build the identical driver command; batch just +lets `ray.sub` run it instead of handing it to you. + +Each interactive launcher allocates 6 nodes, starts Ray, idles, and prints: + +```bash +bash -attach.sh # shell on the head node, Ray already up +source -run-cmd.sh # run the driver; edit + re-source to iterate +``` + +Iterating inside one allocation is the point — a cold start pays for the ~60 GB +checkpoint download plus Megatron conversion and vLLM graph capture before the +first rollout, so you do not want to requeue for every config fix. + +To reach a training step quickly instead of waiting for the default 64-rollout +batch: + +```bash +bash swe_nano_sc_interactive.sh grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 +``` + +Cancel with `scancel `. + +## What to change before you run it + +`swe_nano.env` splits its paths into two blocks: + +**Shared read-only — reuse as is.** The training container, the nemo-skills +sandbox image, the SWE prompt set (`swe.jsonl`, 7816 instances) and the SWE-bench +`.sif` images are all world-readable on Lustre. They total hundreds of GB; do not +copy them. + +**Per-user write — change all of these.** They are owned by `zhiyul` and you +cannot write to them: + +| Variable | What lands there | +|---|---| +| `CODE_DIR` | your checkout of this branch — it is mounted into the container, so it must be the tree you are editing | +| `WORKSPACE_DIR` | results, Ray logs, checkpoints | +| `HF_HOME` | HuggingFace cache (~60 GB for the Nano BF16 checkpoint) | +| `PERSISTENT_CACHE` | vLLM / Triton / Inductor compile caches, reused across jobs | +| `SLURM_ACCOUNT` | an account you can charge | +| `EXP_NAME` | namespaces `results/`, `ray_logs/` and the W&B run | + +Export `HF_TOKEN` too if you can — the verified run downloaded the checkpoint +unauthenticated and was rate limited. + +## Config chain + +``` +nano_swe_teacher_sc.yaml SingleController + TQ (the recipe) +└── nano_swe_teacher_qwen3mesh.yaml TP2·PP2·CP4, EP1, alltoall, vLLM TP2, data_plane block + └── nano_swe_teacher.yaml 49k context, gym venv reuse, sif_dir + ├── swe_teacher.yaml the Ultra SWE stage (rewards, data, optimizer) + └── _nano_smoke_gb200.yaml.inc Nano/GB200 mesh + smoke sizing +``` + +`nano_swe_teacher_sync_tq.yaml` is the sync variant, branching off +`nano_swe_teacher_qwen3mesh.yaml`. + +### Why each SingleController-specific setting is there + +Every one of these came from a crash on the way to the working run: + +| Setting | Without it | +|---|---| +| `policy.draft.enabled: false` | `KeyError: 'draft'` — `run_grpo_single_controller.py` reads `config.policy["draft"]["enabled"]` unconditionally, and the SWE/Ultra chain has no draft block. | +| `loss_fn.reference_policy_kl_penalty: 0.01` | `AttributeError: 'MegatronPolicyWorker' object has no attribute 'reference_state_dict'` — the SC refit swaps in reference weights, but the reference model is only initialized when the KL penalty is > 0 (`single_controller_utils/setup.py:208`). The SWE base sets it to 0. | +| `grpo.val_period: 0`, `checkpointing.enabled: false` | SC supports neither validation nor checkpointing yet. | +| `async_rl` block | SC uses its own async config, not `grpo.async_grpo`. `min_groups_for_streaming_train` decides how many groups buffer before the first training step. | +| `moe_token_dispatcher_type: alltoall` (in the mesh config) | `AssertionError: hybrid-ep kernel ... at least 2 ranks, but got 1` — the inherited flex/hybridep dispatcher requires EP≥2, and this recipe runs EP1. | +| `data_plane.global_segment_size` / `local_buffer_size` | `DataPlaneConfig` requires them even though only the `mooncake_cpu` backend reads them; the `simple` backend ignores the values. | + +Two more constraints that are not config fields: + +- **Batch invariant.** `num_prompts_per_step × num_generations_per_prompt` must + equal `train_global_batch_size` on the SC split path (one RL step = one + optimizer step). Overriding one without the other raises `ValueError`. +- **Absolute entrypoint path.** `run_grpo_single_controller.py` must be spawned + from `${CODE_DIR}` by absolute path. The launcher mounts only `nemo_rl/` and + `examples/configs` over the container, so the container's baked `examples/` has + no such file and `uv run ./examples/...` fails with "Failed to spawn". + +## Verifying TQ is really engaged + +Config echo alone is not evidence — the async `run_grpo_nemo_gym.py` path prints +`data_plane={'enabled': True, ...}` and then ignores it. Look for **runtime +actors** in the driver log: + +``` +(TransferQueueController pid=...) Per-operation statistics: NOTIFY_DATA_UPDATE: ... +(SimpleStorageUnit pid=..., ip=...) Per-operation statistics: PUT_DATA: req_count=... GET_DATA: ... +(TransferQueueController pid=...) ... KV_RETRIEVE_META: ... SET_CUSTOM_META: ... +``` + +`PUT_DATA` is a rollout committing to TQ; `KV_RETRIEVE_META` is the trainer +consuming it. Then confirm training progresses: + +``` +🚀 Launching SingleControllerActor +train step 5/5 trainer_v=5 lag=1 +step_metrics={'loss': ..., 'grad_norm': ..., ...} +``` + +## Known limits + +- **Rewards are 0.** The base model solves no SWE instance in a smoke run. Real + signal needs a capable policy and many more steps. +- **EP1 is memory-hungry.** Expert parallelism of 1 replicates every expert on + each MP rank. With the reference model loaded (required by SC, see above) peak + usage was ~170 GB/rank of 189 GB. If you scale up context, batch, or model, + expect the initial weight sync to OOM first; raise EP, enable + `optimizer_cpu_offload`, or shorten the sequence. +- **Two-hour walltime.** `WALLTIME=1:59:00` is the max for the `short` QOS and is + the practical floor for this recipe. Shorter allocations expired mid-warmup + during development. For scale: with `num_prompts_per_step=2` a 4-rollout batch + took 6–8 minutes, and the default 64-rollout batch took ~11 minutes. +- **Submodule pin.** This branch pins `3rdparty/Gym-workspace/Gym` to `v0.4.0` to + match the container's prebuilt gym venvs (`skip_venv_if_present: true`). + Bumping it without rebuilding the container forces a concurrent nemo-gym + rebuild and a uv cache lock timeout. + +## Related + +- `nemo_rl/data_plane/factory.py` — data-plane client construction and backends +- `nemo_rl/data_plane/docs/data-plane-async-proposal.md` — design notes +- `docs/guides/async-grpo.md` — the async trainer this recipe replaces +- `examples/configs/grpo_math_1B_megatron_single_controller.yaml` — the math + SingleController reference config the SC blocks were adapted from diff --git a/docs/index.md b/docs/index.md index 5d066eb95db..ff0d1ad1718 100644 --- a/docs/index.md +++ b/docs/index.md @@ -142,6 +142,13 @@ Train Qwen2.5-Omni-7B with GRPO on PhilipC/IntentTrain (audio-visual intent reco Train Qwen3-30B-A3B-Thinking into a SWE agent with a pivot stage plus end-to-end agentic RL on SWE-bench. ::: +:::{grid-item-card} {octicon}`database` Nano SWE with TransferQueue +:link: guides/nano-swe-transferqueue +:link-type: doc + +Six-node SWE RL smoke on Nemotron-3-Nano-30B-A3B with rollouts flowing through the TransferQueue data plane via SingleController. +::: + :::{grid-item-card} {octicon}`plus-circle` Adding New Models :link: adding-new-models :link-type: doc @@ -297,6 +304,7 @@ guides/lora.md guides/cispo.md guides/prorlv2.md guides/swe-rl-qwen3.md +guides/nano-swe-transferqueue.md guides/grpo.md guides/ppo.md guides/grpo-deepscaler.md diff --git a/examples/configs/ultra/_nano_smoke_gb200.yaml.inc b/examples/configs/ultra/_nano_smoke_gb200.yaml.inc new file mode 100644 index 00000000000..25a2fef553d --- /dev/null +++ b/examples/configs/ultra/_nano_smoke_gb200.yaml.inc @@ -0,0 +1,97 @@ +# Shared Nemotron 3 Nano v3 (30B-A3B) overrides for smoke-testing the Ultra +# pipeline on GB200 NVL72 nodes (4 GPUs/node). Include this AFTER the Ultra +# stage config so it overrides the Ultra-scale mesh and sizing, e.g.: +# +# defaults: +# - student_rlvr1.yaml # Ultra stage config (rewards, data, optimizer) +# - _nano_smoke_gb200.yaml.inc # this file (shrink to a Nano smoke shape) +# +# Stage-specific sequence lengths, rewards, data routing, and optimizer settings +# are inherited from the corresponding Ultra config. This file ONLY changes what +# is common to every Nano smoke run: the GPU mesh, the smoke sizing, and the +# Nano-vs-Ultra architecture toggles (alltoall dispatcher, no MTP, no fused +# weighted-squared-ReLU). Per-stage judge shrinking lives in the nano_.yaml +# files, because each stage serves a different set of Gym judge models. + +# ============================================================================= +# Cluster — GB200 NVL72, 4 GPUs/node. num_nodes is overridden by the launcher +# (NUM_ACTOR_NODES). +# +# segment_size MUST be disabled (null). It is an NVL72 topology filter: at Ultra +# scale (segment_size=16) the virtual cluster keeps only nodes that form a +# complete 16-node NVLink segment and DISCARDS the rest — per +# virtual_cluster._filter_bundles_by_segment: +# usable_nodes = (domain_node_count // segment_size) * segment_size +# On a ~5-node smoke allocation that is (5 // 16) * 16 = 0 usable nodes, so every +# placement bundle is discarded and nothing can schedule. null takes the early +# return in build_topology_aware_ordering and skips topology filtering entirely. +# (Distinct from the launcher's SEGMENT_SIZE env var, which is only the sbatch +# divisibility check.) +# ============================================================================= +cluster: + gpus_per_node: 4 + num_nodes: 5 # train(4) + gen(1); overridden by ultra_launch.sh + segment_size: null # disable NVLink-domain segment filtering for small allocs + +# ============================================================================= +# GRPO — shrink the step so a smoke run completes in minutes. +# ============================================================================= +grpo: + num_prompts_per_step: 16 + num_generations_per_prompt: 4 # GBS = 16 * 4 = 64 (see policy below) + num_val_generations_per_prompt: 2 + val_period: 100000 # validation effectively off for the smoke run + val_at_start: false + val_at_end: false + overlong_filtering: true # drop prompts longer than the (small) context + +# ============================================================================= +# Checkpointing — save less often so the 30B save doesn't dominate a short run. +# ============================================================================= +checkpointing: + save_period: 100 + checkpoint_must_save_by: null # let walltime end the job naturally + +# ============================================================================= +# Policy — Nano v3 30B-A3B, small batch + short context. +# model_name / tokenizer are overridden by the launcher (MODEL_PATH). +# ============================================================================= +policy: + train_global_batch_size: 64 # = num_prompts_per_step * num_generations_per_prompt + max_total_sequence_length: 16384 # down from the Ultra stage lengths + + megatron_cfg: + # --- GPU mesh grid (the core smoke-test change) ------------------------- + # world = TP*PP*CP*DP = 2*1*4*2 = 16 (4 GB200 nodes); experts sharded over + # EP=8 (EDP=2). Nano has far fewer experts than Ultra, so EP shrinks too. + tensor_model_parallel_size: 2 # was 8 + expert_tensor_parallel_size: 1 + expert_model_parallel_size: 8 # was 32/64 + context_parallel_size: 4 # was 8/32 + pipeline_model_parallel_size: 1 # unchanged (mirror Ultra) + + # Use the proven Nano dispatcher path instead of Ultra's flex/hybridep, + # which depends on the custom vLLM/kernels build. EP ranks must land in one + # NVLink domain (set SEGMENT_SIZE= in the env so sbatch + # --segment keeps the job on one NVL72 rack), else the EP collective hangs. + # moe_token_dispatcher_type: "alltoall" + # use_fused_weighted_squared_relu: false + + # --- MTP off for the Nano base checkpoint ------------------------------- + # The Nano base checkpoint is not guaranteed to ship MTP heads. + mtp_num_layers: 0 + mtp_loss_scaling_factor: 0.0 + + generation: + max_new_tokens: 4096 # cap generation length for a fast smoke run + vllm_cfg: + tensor_parallel_size: 4 # was 8 -> one GB200 node + expert_parallel_size: 1 # was 8 (EP=TP not needed at this scale) + pipeline_parallel_size: 1 + max_model_len: 16384 + + colocated: + enabled: false + resources: + gpus_per_node: 4 + num_nodes: 1 # overridden by launcher (NUM_GEN_NODES) diff --git a/examples/configs/ultra/nano_swe_teacher.yaml b/examples/configs/ultra/nano_swe_teacher.yaml new file mode 100644 index 00000000000..e88899bc031 --- /dev/null +++ b/examples/configs/ultra/nano_swe_teacher.yaml @@ -0,0 +1,36 @@ +# ============================================================================= +# Nemotron 3 NANO v3 (30B-A3B) — SWE teacher smoke test — GB200 +# ============================================================================= +# Inherits, in order: +# 1. swe_teacher.yaml — the Ultra stage +# 2. _nano_smoke_gb200.yaml.inc — Nano+GB200 mesh & smoke sizing (shared) +# +# SWE has NO GPU-served Gym judges (env.nemo_gym.num_gpu_nodes = 0). Rewards come +# from apptainer sandboxes, so there is no judge parallelism to shrink here. The +# launcher must still set SIF_DIR (sif_dir=${SIF_DIR}) — the SWE agents resolve +# `${sif_dir}//{instance_id}.sif`. +# +# Everything smoke-relevant (mesh, batch, context, MTP off, alltoall) comes from +# _nano_smoke_gb200.yaml.inc, so this file only wires the inheritance. +# ============================================================================= + +defaults: + - swe_teacher.yaml + - _nano_smoke_gb200.yaml.inc + +env: + nemo_gym: + # Reuse the container's baked gym venvs (swe_teacher.yaml sets this false, + # which forces a concurrent nemo-gym rebuild -> uv cache lock timeout). + skip_venv_if_present: true + +policy: + # For smoke, don't keep SWE's full 192k context unless you specifically want + # to test long-context memory pressure. Keep generation max_model_len in sync + # (the .inc pins it to 16384, which would reject longer SWE prompts). + max_total_sequence_length: 49152 + generation: + vllm_cfg: + max_model_len: 49152 + +sif_dir: /lustre/fsw/portfolios/llmservice/users/sdevare/images \ No newline at end of file diff --git a/examples/configs/ultra/nano_swe_teacher_qwen3mesh.yaml b/examples/configs/ultra/nano_swe_teacher_qwen3mesh.yaml new file mode 100644 index 00000000000..6d177bf0251 --- /dev/null +++ b/examples/configs/ultra/nano_swe_teacher_qwen3mesh.yaml @@ -0,0 +1,71 @@ +# ============================================================================= +# Nemotron 3 NANO v3 (30B-A3B) — SWE Teacher — Qwen3-MoE parallelism (GB200) +# ============================================================================= +# Same Nano SWE smoke as nano_swe_teacher.yaml, but with the TRAIN/GEN mesh +# mirrored from the tuned Qwen3-30B-A3B MoE SWE recipe (both are ~30B MoE, async +# GRPO, non-colocated), adapted to GB200 NVL72 (4 GPUs/node). +# +# Reference: examples/nemo_gym/grpo_qwen3_30ba3b_thinking_swe1.yaml (16n×8g H100) +# Train: TP2 · PP2 · CP4 (no expert-parallel), MoE router frozen +# Gen: vLLM TP2, non-colocated +# +# GB200 adaptation (minimal, DP1): +# Train 4 nodes (16 GPUs) = TP2·PP2·CP4·DP1 +# Gen 2 nodes ( 8 GPUs) = vLLM TP2 -> 4 engines +# Total 6 nodes (set SEGMENT_SIZE=6 so sbatch keeps it on one rack) +# +# NOTE: PP2 and EP1 are inherited from the Qwen3 reference and are UNVERIFIED for +# the specific Nano v3 checkpoint — confirm on a DRY_RUN / first smoke (the model +# must support 2 pipeline stages; EP1 replicates experts per MP rank). +# ============================================================================= + +defaults: + - nano_swe_teacher.yaml + +policy: + megatron_cfg: + tensor_model_parallel_size: 2 # Qwen3: 2 + pipeline_model_parallel_size: 2 # Qwen3: 2 (nano .inc had 1) + context_parallel_size: 4 # Qwen3: 4 + expert_model_parallel_size: 1 # Qwen3: no EP (nano .inc had 8) + # swe_teacher.yaml inherits flex/hybridep, whose deep_ep kernel REQUIRES EP>=2 + # (AssertionError "hybrid-ep kernel ... at least 2 ranks, but got 1"). With EP1 + # use the proven Nano alltoall dispatcher (the _nano_smoke .inc's intent). + moe_token_dispatcher_type: "alltoall" + + # MoE router settings from the Qwen3 reference + freeze_moe_router: true + moe_router_bias_update_rate: 0.001 + moe_aux_loss_coeff: 0.0 + moe_router_enable_expert_bias: true + + generation: + vllm_cfg: + tensor_parallel_size: 2 # Qwen3: 2 (nano .inc had 4) + colocated: + enabled: false + resources: + gpus_per_node: 4 + num_nodes: 2 # overridden by launcher (NUM_GEN_NODES) + +# TransferQueue data plane. Defined here so the children can inherit it, but +# note WHICH entrypoint actually honours it: +# run_grpo_nemo_gym.py + async_grpo.enabled=true -> IGNORES this block. The +# async trainer builds the in-memory ReplayBuffer (grpo.py:3898), not the +# TQ-backed TQReplayBuffer (async_utils/replay_buffer.py:638). +# run_grpo_nemo_gym.py + async_grpo.enabled=false -> honoured by grpo_train_sync +# (see nano_swe_teacher_sync_tq.yaml). +# run_grpo_single_controller.py -> honoured, and REQUIRED +# (the entrypoint raises when data_plane.enabled is false). This is the +# async + TQ combination — see nano_swe_teacher_sc.yaml. +# Leaving it enabled on this async config is a harmless no-op; it is kept so the +# two child configs get the same storage settings. +data_plane: + enabled: true + impl: transfer_queue + backend: "simple" + storage_capacity: 1000000 + num_storage_units: 2 + claim_meta_poll_interval_s: 0.5 + global_segment_size: 549755813888 # 512 GiB (only used by mooncake_cpu backend) + local_buffer_size: 68719476736 # 64 GiB (only used by mooncake_cpu backend) diff --git a/examples/configs/ultra/nano_swe_teacher_sc.yaml b/examples/configs/ultra/nano_swe_teacher_sc.yaml new file mode 100644 index 00000000000..85d657e72a3 --- /dev/null +++ b/examples/configs/ultra/nano_swe_teacher_sc.yaml @@ -0,0 +1,58 @@ +# ============================================================================= +# Nano v3 30B-A3B SWE — SingleController async GRPO + HONOURED TransferQueue +# ============================================================================= +# THE path that gives all three: SWE (NeMo-Gym) + async + real TQ mediation. +# Runs via examples/run_grpo_single_controller.py (NOT run_grpo_nemo_gym.py) → +# SingleControllerActor, which drives async rollout↔train through TQReplayBuffer +# (dp_client.put_samples → TransferQueue). `run_grpo_nemo_gym.py`'s async path +# (async_grpo_train) uses an in-memory ReplayBuffer and does NOT honour TQ. +# +# uv run ./examples/run_grpo_single_controller.py --config +# +# data_plane.enabled=true is MANDATORY for this launcher (it raises otherwise); +# inherited from nano_swe_teacher_qwen3mesh.yaml (impl: transfer_queue). +# +# VERIFIED on 6 GB200 nodes (train 4 / gen 2), Nemotron-3-Nano-30B-A3B-BF16: +# ran train steps 1-5 with live TransferQueueController + 2 SimpleStorageUnit +# actors serving PUT_DATA / KV_RETRIEVE_META / SET_CUSTOM_META. Launch with +# swe_nano_sc_interactive.sh; see docs/guides/nano-swe-transferqueue.md. +# ============================================================================= + +defaults: + - nano_swe_teacher_qwen3mesh.yaml # SWE env + Nano mesh (TP2·PP2·CP4, alltoall, EP1) + data_plane:transfer_queue + +# SingleController's own async config (SC uses this, not grpo.async_grpo). Mirrors +# examples/configs/grpo_math_1B_megatron_single_controller.yaml. +async_rl: + sampler: + name: in_order + max_lookahead_versions: 1 # how far generation may run ahead of the trainer + recompute_kv_cache_after_weight_updates: false + min_groups_for_streaming_train: ${grpo.num_prompts_per_step} + max_inflight_prompts: ${grpo.num_prompts_per_step} + max_buffered_rollouts: 64 # DataPlane backpressure cap + diagnostics: false + +grpo: + val_period: 0 # SC does not support validation yet + +checkpointing: + enabled: false # SC does not support checkpointing yet + +policy: + # run_grpo_single_controller.py:111 reads policy.draft.enabled (speculative + # draft weights). The Ultra/SWE chain has no draft block; add it, disabled. + draft: + enabled: false + model_name: null + loss_weight: 0.1 + num_layers: null + aux_layer_indices: null + +loss_fn: + # SC initializes the reference model only when kl_penalty>0 (single_controller_utils/ + # setup.py:208: init_reference_model = reference_policy_kl_penalty > 0), and its refit + # unconditionally swaps in reference_state_dict (megatron_policy_worker.py:1611). The SWE + # base sets kl_penalty=0 → no reference model → AttributeError 'reference_state_dict'. + # Enable a small KL (matches grpo_math_1B_megatron_single_controller.yaml). + reference_policy_kl_penalty: 0.01 diff --git a/examples/configs/ultra/nano_swe_teacher_sync_tq.yaml b/examples/configs/ultra/nano_swe_teacher_sync_tq.yaml new file mode 100644 index 00000000000..a875fe93f3a --- /dev/null +++ b/examples/configs/ultra/nano_swe_teacher_sync_tq.yaml @@ -0,0 +1,46 @@ +# ============================================================================= +# Nano v3 30B-A3B SWE — SYNC GRPO + TransferQueue data plane (GB200) +# ============================================================================= +# Purpose: exercise the TransferQueue data plane (data_plane.impl=transfer_queue) +# under the SYNC trainer (grpo_train_sync). Of the two run_grpo_nemo_gym.py +# paths, only the sync one puts rollouts through TQ — async_grpo_train uses an +# in-memory ReplayBuffer — so this variant DISABLES async_grpo. +# +# Inherits the nano Qwen3-mesh recipe (TP2·PP2·CP4, vLLM TP2, 6-node shape) and +# only changes the trainer + data plane. +# +# VERIFIED: reached "🚀 Running synchronous GRPO training" → Step 1 (rewards → +# advantages → logprobs → forward/backward) on the 6-node nano shape, with a +# 64-rollout SWE batch taking ~11 min. +# +# Sync leaves the 16 training GPUs idle during that rollout, which is why the +# production SWE path is async. For async + honoured TQ use +# nano_swe_teacher_sc.yaml (SingleController) instead; this file is the simplest +# way to smoke-test the data plane itself. +# ============================================================================= + +defaults: + - nano_swe_teacher_qwen3mesh.yaml + +grpo: + async_grpo: + enabled: false # REQUIRED: TQ data plane runs under grpo_train_sync + +# TransferQueue-mediated bulk transfer between rollout and train. +# See nemo_rl/data_plane/factory.py. 'simple' = CPU backend (no mooncake). +data_plane: + enabled: true + impl: transfer_queue + backend: "simple" + storage_capacity: 1000000 + num_storage_units: 2 + claim_meta_poll_interval_s: 0.5 + # Required by DataPlaneConfig (ints, no defaults). Only read when + # backend == "mooncake_cpu"; the "simple" backend ignores them. + global_segment_size: 549755813888 # 512 GiB + local_buffer_size: 68719476736 # 64 GiB + +policy: + generation: + vllm_cfg: + async_engine: true # keep vLLM async engine (per request); independent of async_grpo diff --git a/examples/configs/ultra/swe_teacher.yaml b/examples/configs/ultra/swe_teacher.yaml new file mode 100644 index 00000000000..cc129c192ae --- /dev/null +++ b/examples/configs/ultra/swe_teacher.yaml @@ -0,0 +1,478 @@ +# ============================================================================= +# Nemotron 3 Ultra — SWE Teacher (code-execution RLVR, GBS=512, 192k context) +# ============================================================================= +# RLVR teacher for software-engineering benchmarks. Trains the policy against +# multi-turn coding agents (`swe_agents`) that execute candidate fixes inside +# apptainer (.sif) container images for SWE-Bench, SWE-Gym, etc., +# and reward based on test pass/fail — no LLM judges in this stage. +# +# Trained at 192k context with a +# small batch (GBS=512, PPS=32, GPP=16), large CP=32, and EP=32. +# +# Cluster shape (128 nodes × 4 GPUs on GB200 NVL72 = 512 GPUs): +# Training 64 / vLLM 64 / Gym 0 (no LLM judges). Override via +# NUM_TRAIN_NODES, NUM_GEN_NODES, NUM_GYM_NODES. +# +# SIF images: the SWE agent's `container_formatter` paths use the top-level +# `sif_dir` Hydra key — set via `SIF_DIR=...` in the launcher. The released +# blend uses two benchmarks, so the directory needs `swerebench/` and `swegym/` +# subdirectories with per-instance `{instance_id}.sif` files. See the public +# guide for how to build these for ARM64 (GB200). +# ============================================================================= +# +# - gpus_per_node: 4 +# - TP: 8, CP: 8, EP: 32, PP: 1 +# - vLLM TP: 8 +# - Non-colocated async inference with 64 generation nodes +# - Max sequence length: 131072 +# ============================================================================= + +# ============================================================================= +# Cluster — overridden by launch script for parallelism changes +# ============================================================================= +cluster: + gpus_per_node: 4 + num_nodes: 128 + segment_size: 16 + +# ============================================================================= +# SIF image root — required, set by the launcher via SIF_DIR. The SWE agent's +# container_formatter (in env.nemo_gym.swe_agents_train) resolves +# `${sif_dir}//{instance_id}.sif` per SWE benchmark family. +# ============================================================================= +sif_dir: ??? + +# ============================================================================= +# Checkpointing +# ============================================================================= +checkpointing: + enabled: true + checkpoint_dir: "results/grpo_ultra_e2e" + metric_name: "val:total_reward/mean" + higher_is_better: true + keep_top_k: 1000000 + save_period: 5 + ft_keep_latest_k: 1 + ft_save_period: 1 + checkpoint_must_save_by: "00:03:30:00" + model_save_format: "safetensors" + save_consolidated: false + save_optimizer: true + +# ============================================================================= +# GRPO Algorithm +# ============================================================================= +grpo: + num_prompts_per_step: 32 + num_generations_per_prompt: 16 + num_val_generations_per_prompt: 2 + max_rollout_turns: 1 + max_num_epochs: 4 + max_num_steps: 1000000 + normalize_rewards: true + use_leave_one_out_baseline: true + advantage_clip_low: -100 + advantage_clip_high: 100 + val_period: 10000 + val_at_start: false + val_at_end: false + overlong_filtering: false + max_val_samples: null + val_batch_size: 256 + seed: 42 + + use_dynamic_sampling: false + dynamic_sampling_max_gen_batches: 10 + batch_multiplier: 1 + + penalize_invalid_tool_call: true + invalid_tool_call_advantage: -5.0 + penalize_malformed_thinking: true + malformed_thinking_advantage: -5.0 + + reward_shaping: + enabled: false + overlong_buffer_length: 128 + overlong_buffer_penalty: 1 + max_response_length: ${policy.max_total_sequence_length} + stop_properly_penalty_coef: null + reward_scaling: + enabled: false + source_min: 0.0 + source_max: 1.0 + target_min: 0.0 + target_max: 1.0 + + async_grpo: + enabled: true + max_trajectory_age_steps: 1 + in_flight_weight_updates: true + recompute_kv_cache_after_weight_updates: false + + use_best_at_k: false + best_at_k_k: 8 + best_at_k_m: 1000 + + use_combined_training: false + combined_training_weight_mode: "auto" + combined_training_best_at_k_weight: 0.2 + combined_training_pass_at_1_weight: 1.0 + + dynamic_sampling_oversample_ratio: 1.0 + seq_logprob_error_threshold: 2 + +# ============================================================================= +# Loss Function +# ============================================================================= +loss_fn: + reference_policy_kl_penalty: 0.0 + reference_policy_kl_type: "k3" + kl_input_clamp_value: null + kl_output_clamp_value: null + + ratio_clip_min: 0.2 + ratio_clip_max: 0.28 + ratio_clip_c: null + use_on_policy_kl_approximation: true + use_importance_sampling_correction: true + truncated_importance_sampling_ratio: 5 + truncated_importance_sampling_ratio_min: 0.2 + truncated_importance_sampling_type: tis + sequence_level_importance_ratios: false + token_level_loss: true + force_on_policy_ratio: true + use_kl_in_reward: false + +# ============================================================================= +# Policy +# ============================================================================= +policy: + model_name: null + tokenizer: + name: ${policy.model_name} + chat_template_kwargs: null + hf_config_overrides: {} + + train_global_batch_size: 512 + train_micro_batch_size: 1 + generation_batch_size: 64 + logprob_batch_size: 1 + max_total_sequence_length: 196608 + precision: "bfloat16" + logprob_chunk_size: 1024 + offload_optimizer_for_logprob: false + + dtensor_cfg: + _v2: true + enabled: false + cpu_offload: false + sequence_parallel: false + activation_checkpointing: false + tensor_parallel_size: 1 + context_parallel_size: 1 + custom_parallel_plan: null + + megatron_cfg: + enabled: true + empty_unused_memory_level: 2 + activation_checkpointing: true + + # TP=8 spans 2 GB200 nodes (4 GPUs each) via intra-rack NVLink. + tensor_model_parallel_size: 8 + expert_tensor_parallel_size: 1 + expert_model_parallel_size: 32 + pipeline_model_parallel_size: 1 + num_layers_in_first_pipeline_stage: null + num_layers_in_last_pipeline_stage: null + context_parallel_size: 32 + pipeline_dtype: ${policy.precision} + sequence_parallel: true + + # MoE + freeze_moe_router: true + moe_router_dtype: "fp32" + moe_router_load_balancing_type: "none" + moe_router_bias_update_rate: 1.0e-3 + moe_router_enable_expert_bias: true + moe_permute_fusion: true + moe_enable_deepep: false + moe_token_dispatcher_type: "flex" + moe_flex_dispatcher_backend: "hybridep" + moe_hybridep_num_sms: 32 + moe_aux_loss_coeff: 0.0 + moe_shared_expert_overlap: false + use_gloo_process_groups: false + + # Compute + apply_rope_fusion: true + use_fused_weighted_squared_relu: true + bias_activation_fusion: false + gradient_accumulation_fusion: false + defer_fp32_logits: true + + # Logging + track_moe_metrics: true + moe_per_layer_logging: true + do_not_average_loss: true + cp_normalize: true + calculate_per_token_loss: true + scale_loss_by_dp_cp_size: false + + # MTP — disabled + mtp_loss_scaling_factor: 0.0 + mtp_use_repeated_layer: true + mtp_num_layers: 5 + mtp_detach_heads: true + + optimizer: + optimizer: "adam" + lr: 3.0e-6 + min_lr: 3.0e-6 + weight_decay: 0.0 + bf16: true + fp16: false + params_dtype: "float32" + + adam_beta1: 0.9 + adam_beta2: 0.999 + adam_eps: 1e-8 + + sgd_momentum: 0.9 + + use_distributed_optimizer: true + use_precision_aware_optimizer: true + + clip_grad: ${policy.max_grad_norm} + + optimizer_cpu_offload: false + optimizer_offload_fraction: 0.0 + + scheduler: + start_weight_decay: ${policy.megatron_cfg.optimizer.weight_decay} + end_weight_decay: ${policy.megatron_cfg.optimizer.weight_decay} + weight_decay_incr_style: "constant" + lr_decay_style: "constant" + lr_decay_iters: null + lr_warmup_iters: 10 + lr_warmup_init: 3e-7 + override_opt_param_scheduler: true + + distributed_data_parallel_config: + grad_reduce_in_fp32: false + overlap_grad_reduce: false + overlap_param_gather: true + average_in_collective: false + use_custom_fsdp: false + data_parallel_sharding_strategy: "optim_grads_params" + + # FP8 — overridden by precision recipe in launch script + fp8_cfg: + enabled: false + fp8: "e4m3" + fp8_recipe: "mxfp8" + fp8_param: false + + first_last_layers_bf16: true + num_layers_at_start_in_bf16: 1 + num_layers_at_end_in_bf16: 1 + + checkpoint: + async_save: true + ckpt_assume_constant_structure: true + fully_parallel_save_process_group: "ep_dp" + fully_parallel_load_process_group: "ep_dp" + fully_parallel_load_exchange_algo: "broadcast" + + env_vars: null + + # --------------------------------------------------------------------------- + # Sequence Packing + # --------------------------------------------------------------------------- + dynamic_batching: + enabled: false + train_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.train_micro_batch_size}} + logprob_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.logprob_batch_size}} + sequence_length_round: 64 + + sequence_packing: + enabled: true + train_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.train_micro_batch_size}} + logprob_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.logprob_batch_size}} + algorithm: "modified_first_fit_decreasing" + sequence_length_round: 64 + fuse_loss: true + + make_sequence_length_divisible_by: ${mul:${policy.megatron_cfg.context_parallel_size}, ${mul:2, ${policy.megatron_cfg.tensor_model_parallel_size}}} + max_grad_norm: 1.0 + optimizer: null + scheduler: null + + # --------------------------------------------------------------------------- + # Generation (vLLM) — Non-colocated, async + # --------------------------------------------------------------------------- + generation: + port_range_low: 11001 + port_range_high: 15000 + backend: "vllm" + max_new_tokens: ${policy.max_total_sequence_length} + temperature: 1.0 + top_p: 1.0 + top_k: null + stop_token_ids: null + stop_strings: null + vllm_cfg: + async_engine: true + precision: ${policy.precision} + kv_cache_dtype: "auto" + tensor_parallel_size: 8 + pipeline_parallel_size: 1 + expert_parallel_size: 1 + gpu_memory_utilization: 0.8 + max_model_len: ${policy.max_total_sequence_length} + enforce_eager: false + use_deep_gemm: false + num_last_layers_in_bf16: 0 + num_first_layers_in_bf16: 0 + enable_vllm_metrics_logger: true + vllm_metrics_logger_interval: 0.5 + expose_http_server: true + skip_tokenizer_init: false + reasoning_parser_plugin: nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py + http_server_serving_chat_kwargs: + enable_auto_tools: true + tool_parser: qwen3_coder + reasoning_parser: nano_v3 + + vllm_kwargs: + attention_backend: FLASH_ATTN + mamba_ssm_cache_dtype: "float32" + # Hybrid Mamba: one state slot per running seq; the vLLM default (1024) can + # exceed available slots. 256 matches the Ultra GB200 serving reference + # (Gym nemotron_3_ultra_dev_nemorl_gb200.yaml). + max_num_seqs: 256 + compilation_config: + cudagraph_capture_sizes: [1,2,4,8,16,32,64] + cudagraph_mode: PIECEWISE + pass_config: + fuse_allreduce_rms: false + + colocated: + enabled: false + resources: + gpus_per_node: 4 + num_nodes: 64 + +# ============================================================================= +# Data +# ============================================================================= +data: + max_input_seq_length: null + shuffle: false + num_workers: 1 + use_multiple_dataloader: false + train: + data_path: null # Set by launch script + validation: + data_path: null # Set by launch script + default: + dataset_name: NemoGymDataset + env_name: "nemo_gym" + prompt_file: null + system_prompt_file: null + processor: "nemo_gym_data_processor" + +# ============================================================================= +# Environment — NeMo Gym + SWE Agents +# ============================================================================= +env: + should_use_nemo_gym: true + should_log_nemo_gym_responses: true + nemo_gym: + nemo_gym_log_dir: "logs/nemo_gym" + skip_venv_if_present: false + # NeMo-RL-side effort shaping; stripped before forwarding to Gym. + effort_levels: + low_string: "{reasoning effort: efficient}" + low_weight: 0.1 + low_penalty: 1 + low_ub: 15000 + num_gpu_nodes: 0 + port_range_low: 15001 + port_range_high: 20000 + invalid_tool_call_patterns: + - "" + - "" + - "" + - "" + thinking_tags: + - "" + - "" + config_paths: + - responses_api_models/vllm_model/configs/vllm_model_for_training.yaml + - responses_api_agents/swe_agents/configs/swebench_openhands_training.yaml + # Model-specific (Nemotron chat template): keep reasoning in the multi-turn SWE + # agent history so the trajectory grows monotonically for token-in-token-out RL. + policy_model: + responses_api_models: + vllm_model: + chat_template_kwargs: + enable_thinking: true + truncate_history_thinking: false + sequential_reasoning_allowed: false + swe_agents_train: + responses_api_agents: + swe_agents: + agent_max_turns: 200 + concurrency: ${mul:2, ${mul:${grpo.num_prompts_per_step}, ${grpo.num_generations_per_prompt}}} + swebench_agent_timeout: 3600 + run_with_mixed_prompts: true + dataset_path: ${data.train.data_path} + apptainer_memory_limit_mb: 65536 + container_formatter: + - "${sif_dir}/swerebench/{instance_id}.sif" + - "${sif_dir}/swegym/sweb.eval.arm64.{instance_id}.sif" + swe_agents_val: + responses_api_agents: + swe_agents: + agent_max_turns: 200 + concurrency: ${mul:2, ${mul:${grpo.num_prompts_per_step}, ${grpo.num_generations_per_prompt}}} + swebench_agent_timeout: 3600 + dataset_path: ${data.validation.data_path} + apptainer_memory_limit_mb: 65536 + container_formatter: ${env.nemo_gym.swe_agents_train.responses_api_agents.swe_agents.container_formatter} + use_absolute_ip: true + +# ============================================================================= +# Logger +# ============================================================================= +logger: + log_dir: "logs" + num_val_samples_to_print: 0 + wandb_enabled: false + tensorboard_enabled: false + mlflow_enabled: false + monitor_gpus: true + swanlab_enabled: false + wandb: + project: "ultra-v3-swe-e2e" + name: "grpo-ultra-e2e" + tensorboard: {} + mlflow: + experiment_name: "grpo-ultra-e2e" + run_name: "grpo-ultra-e2e" + gpu_monitoring: + collection_interval: 10 + flush_interval: 10 + +# ============================================================================= +# Reward Penalties (set reward to 0 when triggered) +# ============================================================================= +reward_penalties: + penalize_duplicated_reasoning: true # reasoning content == final answer + penalize_empty_final_answer: true # last message output has empty content + penalize_unwanted_tokens: true # unwanted token appears in generation + penalize_malformed_think_tag: true # /<\/think> count != 1 per turn + token_ids: + unwanted: [2] # + think_open: 12 # + think_close: 13 # diff --git a/swe_nano.env b/swe_nano.env new file mode 100644 index 00000000000..4efe23c23be --- /dev/null +++ b/swe_nano.env @@ -0,0 +1,110 @@ +# ============================================================================= +# Nemotron 3 NANO v3 (30B-A3B) — SWE Teacher — MINIMAL 5-node smoke — GB200 +# ============================================================================= +# Small-scale SWE: swaps Ultra 550B for Nano v3 30B-A3B and shrinks to 6 nodes +# (train 4 + gen 2), GBS 64, 49k context. Shared by the three nano SWE +# launchers; each one overrides CONFIG_PATH (and the entrypoint) for its variant: +# +# swe_nano_interactive.sh async GRPO, no TransferQueue (baseline) +# swe_nano_tq_interactive.sh sync GRPO, TransferQueue honoured +# swe_nano_sc_interactive.sh async GRPO via SingleController, TQ honoured +# +# See docs/guides/nano-swe-transferqueue.md for which code paths actually +# honour the data plane and for the verified 5-step run. +# +# ----------------------------------------------------------------------------- +# READ vs WRITE directories — WHAT TO CHANGE WHEN YOU REUSE THIS RECIPE +# ----------------------------------------------------------------------------- +# The "shared read-only" block below (container images, SWE dataset, SIF images) +# is world-readable on Lustre — REUSE IT AS IS. Do not copy those multi-GB +# artifacts into your own tree. +# +# The "per-user write" block is owned by zhiyul and is NOT writable by you. +# Point every variable in that block at a directory you own before launching. +# ============================================================================= +export NRL_FORCE_REBUILD_VENVS=false +export NCCL_MAX_NCHANNELS=1 +export NCCL_NVLS_ENABLE=0 + +# ----------------------------------------------------------------------------- +# SHARED READ-ONLY — reuse as is (world-readable; nothing to copy or rebuild) +# ----------------------------------------------------------------------------- +# Training container (GB200 aarch64, prefetched venvs) and the nemo-skills +# sandbox image. Both are plain squashfs files read by pyxis at job start. +export CONTAINER=/lustre/fsw/portfolios/llmservice/users/zhiyul/enroot-images/gitlab-master.nvidia.com+yifuw+images+rl+ultra_gb200_forcenccl_20260724_prefetch_venvs.2026-07-25.squashfs +export SANDBOX_CONTAINER=/lustre/fsw/portfolios/coreai/users/cye/enroot/nemo-rl:skills-sandbox-latest.squashfs + +# SWE prompt set (7816 instances). Read-only; no need to duplicate. +export DATA_DIR=/lustre/fsw/portfolios/coreai/users/cye/code/RL-debug-ultra/workspace/data + +# SWE rewards come from executing each instance in its apptainer .sif image +# (no GPU judges). Expects swerebench/ and swegym/ subdirectories. +export SIF_DIR=/lustre/fsw/portfolios/llmservice/users/sdevare/images + +# Nano v3 30B-A3B — HF repo id (Nemotron model → matches the nano_v3 reasoning +# parser in swe_teacher.yaml). Downloaded to HF_HOME on first run (~60 GB BF16); +# export HF_TOKEN before launching if the repo is gated (also makes the download +# much faster — the verified run downloaded unauthenticated and was rate limited). +export MODEL_PATH=nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 + +# ----------------------------------------------------------------------------- +# PER-USER WRITE — CHANGE THESE. They are zhiyul-owned and not writable by you. +# Substitute your own Lustre paths (any writable location works): +# CODE_DIR your checkout of this branch; mounted into the container, +# so it must be the tree you are actually editing +# WORKSPACE_DIR results, ray logs, checkpoints (grows to tens of GB) +# HF_HOME HuggingFace cache — ~60 GB for the Nano BF16 checkpoint +# PERSISTENT_CACHE vLLM/Triton/Inductor compile caches reused across jobs +# ----------------------------------------------------------------------------- +# Code from this checkout (the fsw path is a symlink to the fs1 RL-copy tree). +export CODE_DIR=/lustre/fsw/portfolios/llmservice/users/zhiyul/RL-copy +export WORKSPACE_DIR=${CODE_DIR}/workspace + +export HF_HOME=/lustre/fsw/portfolios/llmservice/users/zhiyul/hf_cache +export PERSISTENT_CACHE=/lustre/fsw/portfolios/llmservice/users/zhiyul/persistent_cache +export NRL_MEGATRON_CHECKPOINT_DIR=${PERSISTENT_CACHE}/megatron_ckpt_cache + +# Slurm identity — change SLURM_ACCOUNT to an account you can charge. +export SLURM_PARTITION=batch +export SLURM_ACCOUNT=nemotron_sw_post +export GPUS_PER_NODE=4 # GB200 NVL72 + +# Nano SWE shape mirroring the Qwen3-30B-A3B MoE mesh (6 nodes total): +# Train 4 nodes (16 GPUs) = TP2 · PP2 · CP4 · DP1 (no EP) +# Gen 2 nodes ( 8 GPUs) = vLLM TP2 -> 4 engines (non-colocated) +# Gym 0 nodes = code-execution SIF rewards +export NUM_TRAIN_NODES=4 +export NUM_GEN_NODES=2 +export NUM_GYM_NODES=0 +# Total = 6; SEGMENT_SIZE must divide it (sbatch --segment). The nano .inc sets +# cluster.segment_size=null to disable the virtual-cluster NVLink segment filter. +export SEGMENT_SIZE=6 +export WALLTIME=1:59:00 # 2h (max for QOS short). 1h is NOT enough: the + # first run downloads ~60 GB from HF and converts + # to Megatron before the first rollout starts. +export SLURM_QOS=short +export NRL_MAX_STEPS=5 + +# Nano base checkpoint: MTP off, container's stock vLLM (no Ultra fork needed). +export ENABLE_MTP_INFERENCE=0 +export USE_CUSTOM_VLLM=0 + +# Mount RL-copy's own Gym (writable; SWE/OpenHands harness writes here at runtime). +export EXTRA_MOUNTS=/lustre:/lustre,${CODE_DIR}/3rdparty/Gym-workspace/Gym:/opt/nemo-rl/3rdparty/Gym-workspace/Gym +export WANDB_PROJ=nano-swe-smoke + +# Start safe: inspect the resolved launch command before submission. +export DRY_RUN=1 +export USE_SNAPSHOT=0 + +# Default variant = async GRPO baseline. The tq/sc launchers override both. +# EXP_NAME namespaces results/, ray_logs/ and the W&B run — rename it per user. +export EXP_NAME=nano-swe-teacher-qwen3mesh-zhiyul +export CONFIG_PATH=${CODE_DIR}/examples/configs/ultra/nano_swe_teacher_qwen3mesh.yaml +export TRAIN_PATH=${DATA_DIR}/swe.jsonl +export VAL_PATH=${DATA_DIR}/swe.jsonl + +export RESULTS_DIR=${WORKSPACE_DIR}/results/${EXP_NAME} +export BASE_LOG_DIR=${WORKSPACE_DIR}/ray_logs/${EXP_NAME} + +export NRL_WG_USE_RAY_REF=1 diff --git a/swe_nano_interactive.sh b/swe_nano_interactive.sh new file mode 100644 index 00000000000..220f6aea24b --- /dev/null +++ b/swe_nano_interactive.sh @@ -0,0 +1,37 @@ +#!/bin/bash +# ============================================================================= +# swe_nano_interactive.sh — interactive launch of the minimal Nano SWE smoke. +# +# Nano v3 30B-A3B, 6-node Qwen3-mesh recipe: async GRPO through +# run_grpo_nemo_gym.py. This is the NO-TransferQueue baseline — that path drives +# an in-memory ReplayBuffer and ignores data_plane. For a honoured data plane +# use swe_nano_sc_interactive.sh (async) or swe_nano_tq_interactive.sh (sync). +# +# Allocates the nodes, starts Ray, and idles so you can attach and run/edit/ +# re-run the driver WITHOUT requeueing. +# +# Run from a NETWORKED shell (the sandbox can't reach slurmctld): +# bash swe_nano_interactive.sh +# bash swe_nano_interactive.sh grpo.max_num_steps=1 # extra hydra overrides pass through +# +# On submit it prints: +# bash -attach.sh # shell on the head node (Ray already up) +# source -run-cmd.sh # run the recipe inside that shell; edit + re-source to iterate +# Cancel with: scancel +# +# First run downloads ~60 GB (Nemotron-3-Nano-30B-A3B-BF16) + converts to +# Megatron, so give the alloc room, e.g.: +# INTERACTIVE_WALLTIME=2:00:00 bash swe_nano_interactive.sh +# ============================================================================= +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +set -a +# shellcheck disable=SC1091 +source "${HERE}/swe_nano.env" +set +a + +# INTERACTIVE_WAIT=1 blocks until Ray is up then prints the attach command; +# set INTERACTIVE_WAIT=0 to submit and return immediately. +INTERACTIVE=1 DRY_RUN=0 INTERACTIVE_WAIT="${INTERACTIVE_WAIT:-1}" \ + bash "${HERE}/ultra_launch.sh" "$@" diff --git a/swe_nano_sc.sh b/swe_nano_sc.sh new file mode 100644 index 00000000000..594b91c4cc2 --- /dev/null +++ b/swe_nano_sc.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# ============================================================================= +# swe_nano_sc.sh — BATCH launch of the nano SWE SingleController + TransferQueue +# recipe (ray.sub runs the driver directly; no interactive idle, no attach). +# +# Same 6-node shape, entrypoint and config as swe_nano_sc_interactive.sh — the +# driver command ray.sub runs is byte-for-byte the one the interactive path +# writes to -run-cmd.sh. Use this for an unattended reproduction; use the +# interactive script when you expect to iterate on the config, since a cold +# start pays the ~60 GB checkpoint download and Megatron conversion every time. +# +# Run from a NETWORKED shell at the repo root: +# bash swe_nano_sc.sh # DRY_RUN inherited from swe_nano.env (=1): inspect first +# DRY_RUN=0 bash swe_nano_sc.sh # submit +# DRY_RUN=0 bash swe_nano_sc.sh grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 +# +# Logs land in ${WORKSPACE_DIR}/results/${EXP_NAME}/runs/latest/slurm/. +# ============================================================================= +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# Capture DRY_RUN passed on the command line BEFORE sourcing swe_nano.env, which +# exports DRY_RUN=1 and would otherwise clobber the caller's value. +_DRY_RUN_IN="${DRY_RUN:-}" + +set -a +# shellcheck disable=SC1091 +source "${HERE}/swe_nano.env" + +# --- SingleController + TransferQueue overrides ------------------------------ +EXP_NAME=nano-swe-sc-tq-zhiyul +NRL_ENTRYPOINT="${CODE_DIR}/examples/run_grpo_single_controller.py" +CONFIG_PATH="${CODE_DIR}/examples/configs/ultra/nano_swe_teacher_sc.yaml" +RESULTS_DIR="${WORKSPACE_DIR}/results/${EXP_NAME}" +BASE_LOG_DIR="${WORKSPACE_DIR}/ray_logs/${EXP_NAME}" +[ -n "${_DRY_RUN_IN}" ] && DRY_RUN="${_DRY_RUN_IN}" +set +a + +bash "${HERE}/ultra_launch.sh" "$@" diff --git a/swe_nano_sc_interactive.sh b/swe_nano_sc_interactive.sh new file mode 100644 index 00000000000..5e68ef9b0a5 --- /dev/null +++ b/swe_nano_sc_interactive.sh @@ -0,0 +1,48 @@ +#!/bin/bash +# ============================================================================= +# swe_nano_sc_interactive.sh — nano SWE via SingleController async GRPO with a +# HONOURED TransferQueue data plane. This is the VERIFIED recipe (see +# docs/guides/nano-swe-transferqueue.md). +# +# Same 6-node nano shape as swe_nano_interactive.sh (train 4 / gen 2), but: +# entrypoint examples/run_grpo_single_controller.py (NOT run_grpo_nemo_gym.py) +# config examples/configs/ultra/nano_swe_teacher_sc.yaml +# +# run_grpo_nemo_gym.py's async path (async_grpo_train) drives an in-memory +# ReplayBuffer and ignores data_plane; only grpo_train_sync and the +# SingleController path put rollouts through TransferQueue. +# +# The entrypoint is spawned by ABSOLUTE path from ${CODE_DIR}: the container +# only has nemo_rl/ and examples/configs mounted over it, so the baked +# examples/ has no run_grpo_single_controller.py. +# +# Run from a NETWORKED shell (the sandbox cannot reach slurmctld): +# bash swe_nano_sc_interactive.sh +# +# On submit it prints: +# bash -attach.sh # shell on the head node (Ray already up) +# source -run-cmd.sh # run the driver; edit + re-source to iterate +# Cancel with: scancel +# +# Extra hydra overrides pass through, e.g. a fast first training step: +# bash swe_nano_sc_interactive.sh grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 +# Keep the invariant num_prompts_per_step × num_generations_per_prompt == +# train_global_batch_size — the SingleController split path enforces it. +# ============================================================================= +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +set -a +# shellcheck disable=SC1091 +source "${HERE}/swe_nano.env" + +# --- override for the SingleController + TransferQueue variant --------------- +EXP_NAME=nano-swe-sc-tq-zhiyul +NRL_ENTRYPOINT="${CODE_DIR}/examples/run_grpo_single_controller.py" +CONFIG_PATH="${CODE_DIR}/examples/configs/ultra/nano_swe_teacher_sc.yaml" +RESULTS_DIR="${WORKSPACE_DIR}/results/${EXP_NAME}" +BASE_LOG_DIR="${WORKSPACE_DIR}/ray_logs/${EXP_NAME}" +set +a + +INTERACTIVE=1 DRY_RUN=0 INTERACTIVE_WAIT="${INTERACTIVE_WAIT:-1}" \ + bash "${HERE}/ultra_launch.sh" "$@" diff --git a/swe_nano_tq_interactive.sh b/swe_nano_tq_interactive.sh new file mode 100644 index 00000000000..2fcec20f305 --- /dev/null +++ b/swe_nano_tq_interactive.sh @@ -0,0 +1,35 @@ +#!/bin/bash +# ============================================================================= +# swe_nano_tq_interactive.sh — interactive nano SWE run with the TransferQueue +# data plane ENABLED (sync GRPO). +# +# Same 6-node nano shape as swe_nano_interactive.sh, but points at the sync + TQ +# config (nano_swe_teacher_sync_tq.yaml): async_grpo disabled, data_plane.enabled +# =true, impl=transfer_queue. Use this to CHECK the data plane end to end. +# +# Run from a NETWORKED shell: +# bash swe_nano_tq_interactive.sh +# bash swe_nano_tq_interactive.sh grpo.max_num_steps=1 # extra hydra overrides pass through +# +# Then: bash -attach.sh ; source -run-cmd.sh (edit + re-source to iterate) +# +# Verify TQ engaged: the driver log must show "🚀 Running synchronous GRPO +# training" (not "Running async GRPO training") plus TransferQueueController / +# SimpleStorageUnit actors. For async + TQ use swe_nano_sc_interactive.sh. +# ============================================================================= +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +set -a +# shellcheck disable=SC1091 +source "${HERE}/swe_nano.env" + +# --- override for the sync + TransferQueue variant -------------------------- +EXP_NAME=nano-swe-tq-zhiyul +CONFIG_PATH="${CODE_DIR}/examples/configs/ultra/nano_swe_teacher_sync_tq.yaml" +RESULTS_DIR="${WORKSPACE_DIR}/results/${EXP_NAME}" +BASE_LOG_DIR="${WORKSPACE_DIR}/ray_logs/${EXP_NAME}" +set +a + +INTERACTIVE=1 DRY_RUN=0 INTERACTIVE_WAIT="${INTERACTIVE_WAIT:-1}" \ + bash "${HERE}/ultra_launch.sh" "$@" diff --git a/ultra_launch.sh b/ultra_launch.sh new file mode 100755 index 00000000000..b37b2180bae --- /dev/null +++ b/ultra_launch.sh @@ -0,0 +1,910 @@ +#!/bin/bash +set -euo pipefail + +# ============================================================================= +# ultra_launch.sh +# +# Public launcher for Nemotron 3 Ultra post-training stages on a SLURM cluster. +# +# Each training stage (Student RLVR, teacher RLVR/RLHF stages, MOPD) has a +# matching YAML config under examples/configs/ultra/. The stage-specific +# hyperparameters (batch size, advantage clip, MoE parallelism, etc.) live +# in the YAML; this launcher only handles orchestration: SLURM submission, +# code snapshotting, persistent cache management, container mounts, and the +# Hydra overrides that vary per run (data paths, model checkpoint, judge +# endpoints, log directories). +# +# Usage: +# +# EXP_NAME=ultra-student-rlvr-001 \ +# CONFIG_PATH=examples/configs/ultra/student_rlvr.yaml \ +# MODEL_PATH=/path/to/sft_checkpoint \ +# TRAIN_PATH=/path/to/train.jsonl \ +# VAL_PATH=/path/to/val.jsonl \ +# CONTAINER=nvcr.io/nvidia/nemo-rl: \ +# SANDBOX_CONTAINER=/path/to/nemo-skills-sandbox.sqsh \ +# PERSISTENT_CACHE=/path/to/persistent/cache \ +# SLURM_PARTITION=batch \ +# SLURM_ACCOUNT=your_account \ +# GENRM_MODEL= # Required for local and remote GenRM +# NL2BASH_JUDGE_MODEL=Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 \ +# SAFETY_JUDGE_MODEL=/path/to/safety_checkpoint \ +# bash ultra_launch.sh +# +# Optional knobs: +# WALLTIME=4:00:00 Slurm --time +# SLURM_QOS= Slurm --qos; defaults to short when +# WALLTIME is under two hours +# SLURM_RESERVATION= Slurm --reservation +# SLURM_DEPENDENCY= Extra Slurm dependency, merged with +# singleton (e.g. afterany:) +# GENRM_BASE_URL= Remote GenRM OpenAI base URL +# GENRM_BASE_URL_FILE= Shared file containing a remote GenRM +# host:port or OpenAI base URL. Resolved +# after the job allocation starts. +# GENRM_WAIT_TIMEOUT_SECONDS=3600 Endpoint-file readiness timeout +# EXCLUDE_NODES= Slurm --exclude +# NUM_TRAIN_NODES=64 Training (Megatron) nodes +# NUM_GEN_NODES=172 vLLM generation nodes +# NUM_GYM_NODES=20 NeMo Gym (judge) nodes +# ENABLE_MTP_INFERENCE=0 1 to enable MTP speculative decoding +# NUM_SPECULATIVE_TOKENS=5 MTP speculative tokens +# MAX_NUM_BATCHED_TOKENS=8480 vLLM max batched tokens (MTP) +# NRL_MAX_STEPS= Override grpo.max_num_steps +# EXTRA_MOUNTS= Comma-separated host:container pairs +# USE_SNAPSHOT=1 Snapshot source tree at submission +# USE_CUSTOM_VLLM=1 1 to require the Ultra vLLM fork; +# 0 to use the container's regular vLLM +# DRY_RUN=0 1 to print TRAIN_CMD and exit +# INTERACTIVE=0 1 to bring up Ray and idle for attach +# (no training driver) for debugging +# INTERACTIVE_WAIT=1 0 to submit and return immediately +# INTERACTIVE_WALLTIME= override WALLTIME for the interactive alloc +# HF_HOME= HuggingFace cache root (recommended) +# HF_TOKEN= HuggingFace API token +# WANDB_API_KEY= Weights & Biases API key +# WANDB_PROJ=nemotron-3-ultra W&B project +# WANDB_ENTITY= W&B entity +# +# Hydra overrides are forwarded verbatim as positional arguments: +# bash ultra_launch.sh policy.megatron_cfg.optimizer.lr=1e-6 grpo.val_period=50 +# +# GB200 NVL72 nodes have 4 GPUs each. SLURM total = NUM_TRAIN + NUM_GEN + NUM_GYM +# and must be a multiple of SEGMENT_SIZE (default 16, one NVLink domain group). +# ============================================================================= + +# ============================================================================= +# Required environment +# ============================================================================= +: "${EXP_NAME:?EXP_NAME is required (used for job name, W&B run, checkpoint/log dirs)}" +: "${CONFIG_PATH:?CONFIG_PATH is required (e.g. examples/configs/ultra/student_rlvr.yaml)}" +: "${MODEL_PATH:?MODEL_PATH is required (initial policy checkpoint, HF repo id or local path)}" +: "${TRAIN_PATH:?TRAIN_PATH is required (training data jsonl path)}" +: "${VAL_PATH:?VAL_PATH is required (validation data jsonl path)}" +: "${CONTAINER:?CONTAINER is required (NGC image URI or .sqsh path)}" +: "${SANDBOX_CONTAINER:?SANDBOX_CONTAINER is required (nemo-skills sandbox image)}" +: "${PERSISTENT_CACHE:?PERSISTENT_CACHE is required (Lustre dir for vLLM/Triton/Inductor caches)}" +: "${SLURM_PARTITION:?SLURM_PARTITION is required}" +: "${SLURM_ACCOUNT:?SLURM_ACCOUNT is required}" +# Judge models are recipe-specific. Most teachers (student RLVR, IFBench, RLHF, +# Reasoning) need all three (GenRM, NL2Bash, Safety). The SWE teacher uses +# code-execution rewards and needs none of them. Set per recipe; unset vars +# skip the corresponding override. +NL2BASH_JUDGE_MODEL="${NL2BASH_JUDGE_MODEL:-}" +SAFETY_JUDGE_MODEL="${SAFETY_JUDGE_MODEL:-}" +GENRM_BASE_URL="${GENRM_BASE_URL:-}" +GENRM_BASE_URL_FILE="${GENRM_BASE_URL_FILE:-}" +GENRM_WAIT_TIMEOUT_SECONDS="${GENRM_WAIT_TIMEOUT_SECONDS:-3600}" +GENRM_MODEL="${GENRM_MODEL:-}" +GENRM_OVERRIDE="" +GENRM_RUNTIME_SETUP="" +if [[ -n "${GENRM_BASE_URL}" && -n "${GENRM_BASE_URL_FILE}" ]]; then + echo "ERROR: set only one of GENRM_BASE_URL or GENRM_BASE_URL_FILE" >&2 + exit 1 +fi +if [[ -n "${GENRM_BASE_URL}" || -n "${GENRM_BASE_URL_FILE}" ]]; then + if [[ -z "${GENRM_MODEL}" ]]; then + echo "ERROR: GENRM_MODEL is required when using a remote GenRM endpoint" >&2 + exit 1 + fi +fi +if [[ ! "${GENRM_WAIT_TIMEOUT_SECONDS}" =~ ^[1-9][0-9]*$ ]]; then + echo "ERROR: GENRM_WAIT_TIMEOUT_SECONDS must be a positive integer" >&2 + exit 1 +fi +if [[ -n "${GENRM_BASE_URL_FILE}" ]]; then + GENRM_RUNTIME_SETUP="\ +GENRM_BASE_URL_FILE='${GENRM_BASE_URL_FILE}' ; \ +GENRM_WAIT_TIMEOUT_SECONDS=${GENRM_WAIT_TIMEOUT_SECONDS} ; \ +GENRM_WAIT_DEADLINE=\$((SECONDS + GENRM_WAIT_TIMEOUT_SECONDS)) ; \ +echo \"Waiting for GenRM endpoint file: \${GENRM_BASE_URL_FILE}\" ; \ +while true; do \ + if [[ -s \"\${GENRM_BASE_URL_FILE}\" ]]; then \ + GENRM_ENDPOINT=\$(tr -d '[:space:]' < \"\${GENRM_BASE_URL_FILE}\") ; \ + case \"\${GENRM_ENDPOINT}\" in \ + http://*|https://*) GENRM_RUNTIME_BASE_URL=\"\${GENRM_ENDPOINT%/}\" ;; \ + *) GENRM_RUNTIME_BASE_URL=\"http://\${GENRM_ENDPOINT%/}\" ;; \ + esac ; \ + case \"\${GENRM_RUNTIME_BASE_URL}\" in \ + */v1) ;; \ + *) GENRM_RUNTIME_BASE_URL=\"\${GENRM_RUNTIME_BASE_URL}/v1\" ;; \ + esac ; \ + if curl --fail --silent --show-error --connect-timeout 5 --max-time 10 \ + \"\${GENRM_RUNTIME_BASE_URL}/models\" >/dev/null; then \ + echo \"GenRM is ready at \${GENRM_RUNTIME_BASE_URL}\" ; \ + break ; \ + fi ; \ + fi ; \ + if (( SECONDS >= GENRM_WAIT_DEADLINE )); then \ + echo \"ERROR: timed out waiting for GenRM via \${GENRM_BASE_URL_FILE}\" >&2 ; \ + exit 1 ; \ + fi ; \ + sleep 10 ; \ +done ; " + GENRM_OVERRIDE="env.nemo_gym.genrm_model.responses_api_models.genrm_model.base_url=\${GENRM_RUNTIME_BASE_URL} env.nemo_gym.genrm_model.responses_api_models.genrm_model.model=${GENRM_MODEL}" +elif [[ -n "${GENRM_BASE_URL}" ]]; then + GENRM_BASE_URL="${GENRM_BASE_URL%/}" + [[ "${GENRM_BASE_URL}" == */v1 ]] || GENRM_BASE_URL="${GENRM_BASE_URL}/v1" + GENRM_OVERRIDE="env.nemo_gym.genrm_model.responses_api_models.genrm_model.base_url=${GENRM_BASE_URL} env.nemo_gym.genrm_model.responses_api_models.genrm_model.model=${GENRM_MODEL}" +elif [[ -n "${GENRM_MODEL}" ]]; then + GENRM_OVERRIDE="env.nemo_gym.genrm_model.responses_api_models.genrm_model.model=${GENRM_MODEL}" +fi + +# SIF_DIR: for the SWE teacher recipe — directory containing apptainer .sif +# images for SWE-Bench / SWE-Gym / R2E-Gym instances. The yaml's +# container_formatter uses `${sif_dir}/...` paths. Unset for non-SWE recipes. +SIF_DIR="${SIF_DIR:-}" + +if [[ ! -f "${CONFIG_PATH}" ]]; then + echo "ERROR: CONFIG_PATH does not exist: ${CONFIG_PATH}" >&2 + exit 1 +fi + +# ============================================================================= +# Project root and code root +# ============================================================================= +PROJECT_ROOT=$(realpath "$PWD") +cd "${PROJECT_ROOT}" + +# ============================================================================= +# Job identity — fixed name for singleton. +# Slurm --dependency=singleton serialises queued submissions with the same name +# so a resubmission after preemption resumes from the latest checkpoint instead +# of running in parallel. +# ============================================================================= +JOB_NAME="${EXP_NAME}" + +# ============================================================================= +# Output directories +# ============================================================================= +RESULTS_DIR="${RESULTS_DIR:-results/${EXP_NAME}}" +CHECKPOINT_DIR="${CHECKPOINT_DIR:-${RESULTS_DIR}/checkpoints}" + +# Per-submission dirs for logs and Slurm output (timestamped for history). +RUN_DIR="${RESULTS_DIR}/runs/$(date +%Y%m%d-%H%M)" +LOG_DIR="${RUN_DIR}/logs" +SLURM_LOG_DIR="${RUN_DIR}/slurm" +mkdir -p "${CHECKPOINT_DIR}" "${LOG_DIR}" "${SLURM_LOG_DIR}" +ln -sfn "$(realpath "${RUN_DIR}")" "${RESULTS_DIR}/runs/latest" + +# ray.sub reads BASE_LOG_DIR and creates $BASE_LOG_DIR/$SLURM_JOB_ID-logs/ for +# ray infrastructure logs (ray-head.log, ray-driver.log, ray-worker-*.log, +# topology probes, attach scripts, etc.). +export BASE_LOG_DIR="${BASE_LOG_DIR:-${RESULTS_DIR}/ray_logs}" + +# ============================================================================= +# SLURM configuration +# ============================================================================= +WALLTIME="${WALLTIME:-4:00:00}" +SLURM_QOS="${SLURM_QOS:-}" +SLURM_RESERVATION="${SLURM_RESERVATION:-}" +EXCLUDE_NODES="${EXCLUDE_NODES:-}" + +slurm_walltime_seconds() { + local value="$1" + local days=0 + local -a fields + + if [[ "${value}" == *-* ]]; then + days="${value%%-*}" + value="${value#*-}" + fi + [[ "${days}" =~ ^[0-9]+$ ]] || return 1 + + IFS=: read -r -a fields <<< "${value}" + for field in "${fields[@]}"; do + [[ "${field}" =~ ^[0-9]+$ ]] || return 1 + done + + case "${#fields[@]}" in + 1) + if (( days > 0 )); then + echo $((10#${days} * 86400 + 10#${fields[0]} * 3600)) + else + echo $((10#${fields[0]} * 60)) + fi + ;; + 2) + if (( days > 0 )); then + echo $((10#${days} * 86400 + 10#${fields[0]} * 3600 + 10#${fields[1]} * 60)) + else + echo $((10#${fields[0]} * 60 + 10#${fields[1]})) + fi + ;; + 3) + echo $((10#${days} * 86400 + 10#${fields[0]} * 3600 + 10#${fields[1]} * 60 + 10#${fields[2]})) + ;; + *) return 1 ;; + esac +} + +if [[ -z "${SLURM_QOS}" ]]; then + if WALLTIME_SECONDS="$(slurm_walltime_seconds "${WALLTIME}")"; then + if (( WALLTIME_SECONDS < 2 * 60 * 60 )); then + SLURM_QOS=short + fi + else + echo "[WARN] Could not parse WALLTIME=${WALLTIME}; leaving SLURM_QOS unset." >&2 + fi +fi +# INTERACTIVE=1 brings up the Ray cluster and idles for attachment (no training +# driver), so you can run/debug the recipe by hand. INTERACTIVE_WAIT=1 (default) +# blocks until Ray is ready; INTERACTIVE_WALLTIME overrides WALLTIME for the alloc. +INTERACTIVE="${INTERACTIVE:-0}" +INTERACTIVE_WAIT="${INTERACTIVE_WAIT:-1}" +# If set (format DD:HH:MM:SS), training stops early to reserve time for a final +# checkpoint save before walltime. Unset to use the YAML's default and let +# slurm walltime end the job naturally — fine when each step checkpoints. +CHECKPOINTING_SAVE_BY="${CHECKPOINTING_SAVE_BY:-}" + +# ============================================================================= +# Container & mounts +# ============================================================================= +export CONTAINER +MOUNTS="${MOUNTS:-}" + +# GB200 NVL72 defaults to 4 GPUs/node. Allow H100 smoke configs to request +# their native 8-GPU node shape through the launch environment. +export GPUS_PER_NODE="${GPUS_PER_NODE:-4}" +export CPUS_PER_WORKER="${CPUS_PER_WORKER:-144}" + +# ============================================================================= +# HuggingFace configuration +# ============================================================================= +if [[ -n "${HF_HOME:-}" ]]; then + export HF_HOME + export HF_HUB_CACHE="${HF_HUB_CACHE:-${HF_HOME}/hub}" + export HF_DATASETS_CACHE="${HF_DATASETS_CACHE:-${HF_HOME}/hub}" +else + echo "[WARN] HF_HOME is not set — HuggingFace will use the default cache (~/.cache/huggingface) per-node." >&2 +fi + +# ============================================================================= +# W&B configuration +# ============================================================================= +WANDB_PROJ="${WANDB_PROJ:-nemotron-3-ultra}" +WANDB_NAME="${EXP_NAME}" +WANDB_ENABLED=False +if [[ -n "${WANDB_API_KEY:-}" ]]; then + export WANDB_API_KEY + WANDB_ENABLED=True + if [[ -n "${WANDB_ENTITY:-}" ]]; then + export WANDB_ENTITY + fi +else + echo "[WARN] WANDB_API_KEY is not set — W&B logging will be disabled." >&2 +fi + +# ============================================================================= +# Training overrides +# ============================================================================= +NRL_MAX_STEPS="${NRL_MAX_STEPS:-}" + +# ============================================================================= +# MTP speculative decoding (optional) +# ============================================================================= +ENABLE_MTP_INFERENCE="${ENABLE_MTP_INFERENCE:-0}" +NUM_SPECULATIVE_TOKENS="${NUM_SPECULATIVE_TOKENS:-5}" +MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-8480}" +MTP_EXTRA_ARGS="" +if [[ "${ENABLE_MTP_INFERENCE}" == "1" ]]; then + MTP_EXTRA_ARGS="\ +++policy.generation.vllm_cfg.enable_prefix_caching=true \ +++policy.generation.vllm_kwargs.enable_chunked_prefill=true \ +++policy.generation.vllm_kwargs.max_num_batched_tokens=${MAX_NUM_BATCHED_TOKENS} \ +++policy.generation.vllm_kwargs.mamba_cache_mode=align \ +~policy.generation.vllm_kwargs.compilation_config.cudagraph_capture_sizes \ +++policy.generation.vllm_kwargs.speculative_config.num_speculative_tokens=${NUM_SPECULATIVE_TOKENS} \ +++policy.generation.vllm_kwargs.speculative_config.method=mtp" + echo "MTP speculative decoding ENABLED (num_speculative_tokens=${NUM_SPECULATIVE_TOKENS})" +fi + +# ============================================================================= +# Job shape — defaults match the 256-node student_rlvr.yaml +# +# Training: 64 nodes ( 256 GPUs) — Megatron training backend +# vLLM: 172 nodes ( 688 GPUs) — async generation, EP=8 instances at TP=8 +# Gym: 20 nodes ( 80 GPUs) — judges (GenRM, NL2Bash, Safety) +# +# Override via NUM_TRAIN_NODES / NUM_GEN_NODES / NUM_GYM_NODES. +# +# For STAGE_TYPE=mopd, additional teacher nodes are allocated for the +# non-colocated teacher panel: NUM_UNIQUE_TEACHERS × NUM_NODES_PER_TEACHER. +# ============================================================================= +NUM_TRAIN_NODES="${NUM_TRAIN_NODES:-64}" +NUM_GEN_NODES="${NUM_GEN_NODES:-172}" +NUM_GYM_NODES="${NUM_GYM_NODES:-20}" + +STAGE_TYPE="${STAGE_TYPE:-grpo}" +NUM_TEACHER_NODES=0 +MOPD_OVERRIDES="" +if [[ "${STAGE_TYPE}" == "mopd" ]]; then + : "${NRL_GENERAL_TEACHER_PATH:?NRL_GENERAL_TEACHER_PATH is required for STAGE_TYPE=mopd (path to the Student RLVR output checkpoint)}" + NUM_UNIQUE_TEACHERS="${NUM_UNIQUE_TEACHERS:-5}" + NUM_NODES_PER_TEACHER="${NUM_NODES_PER_TEACHER:-4}" + NUM_TEACHER_NODES=$((NUM_UNIQUE_TEACHERS * NUM_NODES_PER_TEACHER)) + TEACHER_TP="${TEACHER_TP:-8}" + TEACHER_CP="${TEACHER_CP:-2}" + TEACHER_PP="${TEACHER_PP:-1}" + TEACHER_EP="${TEACHER_EP:-16}" + + # _teachers.general is required; other slots fall back via the YAML's + # interpolation. Pass only the slots the user explicitly set. + MOPD_OVERRIDES="_teachers.general=${NRL_GENERAL_TEACHER_PATH}" + for _slot in RLHF IFBENCH REASONING SWE; do + _var="NRL_${_slot}_TEACHER_PATH" + _val="${!_var:-}" + if [[ -n "${_val}" ]]; then + MOPD_OVERRIDES="${MOPD_OVERRIDES} _teachers.$(echo ${_slot} | tr A-Z a-z)=${_val}" + fi + done + + # Teacher parallelism + per-teacher node count + MOPD_OVERRIDES="${MOPD_OVERRIDES} \ +on_policy_distillation.non_colocated_teachers.default_teacher_cfg.tensor_model_parallel_size=${TEACHER_TP} \ +on_policy_distillation.non_colocated_teachers.default_teacher_cfg.context_parallel_size=${TEACHER_CP} \ +on_policy_distillation.non_colocated_teachers.default_teacher_cfg.pipeline_model_parallel_size=${TEACHER_PP} \ +on_policy_distillation.non_colocated_teachers.default_teacher_cfg.expert_model_parallel_size=${TEACHER_EP} \ +on_policy_distillation.non_colocated_teachers.default_teacher_cfg.num_nodes=${NUM_NODES_PER_TEACHER}" + + echo "MOPD: ${NUM_UNIQUE_TEACHERS} teacher pools × ${NUM_NODES_PER_TEACHER} nodes = ${NUM_TEACHER_NODES} teacher nodes" +fi + +NUM_ACTOR_NODES=$((NUM_TRAIN_NODES + NUM_GEN_NODES + NUM_TEACHER_NODES)) +NUM_TOTAL_NODES=$((NUM_ACTOR_NODES + NUM_GYM_NODES)) + +if (( NUM_TRAIN_NODES <= 0 )); then + echo "ERROR: NUM_TRAIN_NODES must be > 0 (got ${NUM_TRAIN_NODES})" >&2; exit 1 +fi +if (( NUM_GEN_NODES <= 0 )); then + echo "ERROR: NUM_GEN_NODES must be > 0 (got ${NUM_GEN_NODES})" >&2; exit 1 +fi +if (( NUM_GYM_NODES < 0 )); then + echo "ERROR: NUM_GYM_NODES must be >= 0 (got ${NUM_GYM_NODES})" >&2; exit 1 +fi + +# GB200 NVL72 topology: 18 nodes per NVLink domain, allocate in groups of 16. +SEGMENT_SIZE="${SEGMENT_SIZE:-16}" +if (( NUM_TOTAL_NODES < SEGMENT_SIZE )); then + echo "ERROR: NUM_TOTAL_NODES=${NUM_TOTAL_NODES} < SEGMENT_SIZE=${SEGMENT_SIZE}" >&2 + exit 1 +fi +if (( NUM_TOTAL_NODES % SEGMENT_SIZE != 0 )); then + echo "ERROR: NUM_TOTAL_NODES=${NUM_TOTAL_NODES} is not divisible by SEGMENT_SIZE=${SEGMENT_SIZE}." >&2 + echo " Training=${NUM_TRAIN_NODES} + Generation=${NUM_GEN_NODES} + Gym=${NUM_GYM_NODES} + Teachers=${NUM_TEACHER_NODES} = ${NUM_TOTAL_NODES}" >&2 + echo " Adjust node counts so the total is a multiple of ${SEGMENT_SIZE}." >&2 + exit 1 +fi + +# ============================================================================= +# NeMo Skills sandbox (for math_formal_lean, ns_tools, etc.) +# ============================================================================= +export SANDBOX_CONTAINER +export SANDBOX_COMMAND="${SANDBOX_COMMAND:-/start-with-nginx.sh}" +export NEMO_SKILLS_SANDBOX_PORT="${NEMO_SKILLS_SANDBOX_PORT:-6000}" + +# ============================================================================= +# Ray log sync +# ============================================================================= +export RAY_LOG_SYNC_FREQUENCY="${RAY_LOG_SYNC_FREQUENCY:-60}" + +CODE_ROOT="/opt/nemo-rl" +USE_CUSTOM_VLLM="${USE_CUSTOM_VLLM:-1}" +case "${USE_CUSTOM_VLLM}" in + 1) + VLLM_ENV_SOURCE="source /opt/nemo-rl/3rdparty/vllm/nemo-rl.env && " + ;; + 0) + VLLM_ENV_SOURCE="" + ;; + *) + echo "ERROR: USE_CUSTOM_VLLM must be 0 or 1, got: ${USE_CUSTOM_VLLM}" >&2 + exit 1 + ;; +esac + +# ============================================================================= +# Persistent cache directories +# ============================================================================= +# Lustre holds the warm persistent cache. At job start, SETUP_COMMAND clears +# stale /tmp caches then seeds node-local /tmp from Lustre. JIT writes go to +# /tmp to avoid Lustre metadata contention from parallel compilation. +_vllm_cache_precision="bf16" +CACHE_READ_DIR="${PERSISTENT_CACHE}/cache_read" +CACHE_WRITE_DIR="${PERSISTENT_CACHE}/cache_write" +LUSTRE_VLLM_CACHE="${CACHE_WRITE_DIR}/vllm_compile_cache_${_vllm_cache_precision}" +LUSTRE_FLASHINFER_CUBIN_CACHE="${PERSISTENT_CACHE}/flashinfer_cubins" +FLASHINFER_CUBIN_CACHE="/tmp/nemo_rl_flashinfer_cubins" +FLASHINFER_WS_BASE="${PERSISTENT_CACHE}/flashinfer_workspace" +LUSTRE_INDUCTOR_CACHE="${PERSISTENT_CACHE}/inductor_cache" +LUSTRE_TRITON_CACHE="${PERSISTENT_CACHE}/triton_cache" +NRL_VLLM_LOCAL_CACHE_DIR="/tmp/nemo_rl_vllm_cache" +NRL_VLLM_CACHE_SEED_DIR="/tmp/nemo_rl_vllm_cache_warm" +INDUCTOR_CACHE_DIR="/tmp/nemo_rl_inductor_cache" +TRITON_CACHE_DIR="/tmp/nemo_rl_triton_cache" +CACHE_SYNC_FREQUENCY="${CACHE_SYNC_FREQUENCY:-0}" + +export LUSTRE_VLLM_CACHE +export LUSTRE_INDUCTOR_CACHE +export LUSTRE_TRITON_CACHE +export CACHE_READ_DIR +export CACHE_WRITE_DIR +export NRL_VLLM_LOCAL_CACHE_DIR +export INDUCTOR_CACHE_DIR +export TRITON_CACHE_DIR +export CACHE_SYNC_FREQUENCY + +mkdir -p "${LUSTRE_FLASHINFER_CUBIN_CACHE}" "${FLASHINFER_WS_BASE}" \ + "${LUSTRE_INDUCTOR_CACHE}" "${LUSTRE_TRITON_CACHE}" \ + "${CACHE_READ_DIR}" "${CACHE_WRITE_DIR}" + +# Read path : cache_read/*.tar.zst — compute nodes extract tarballs (hundreds of concurrent reads) +# Write path : cache_write/*/ — sidecar rsyncs individual files (one sequential writer) +# Splitting reads (tarball) from writes (directory) avoids Lustre MDT invalidation storms +# and lets rsync accumulate the union of all roles' kernels across jobs. +for _name in inductor_cache triton_cache; do + _write_dir="${CACHE_WRITE_DIR}/${_name}" + _old_dir="${PERSISTENT_CACHE}/${_name}" + + # One-time migration: move legacy dir → cache_write/ (instant rename, same FS) + if ([ ! -d "$_write_dir" ] || [ -z "$(ls -A "$_write_dir" 2>/dev/null)" ]) \ + && [ -d "$_old_dir" ] && [ -n "$(ls -A "$_old_dir" 2>/dev/null)" ]; then + [ -d "$_write_dir" ] && rmdir "$_write_dir" 2>/dev/null + mv "$_old_dir" "$_write_dir" 2>/dev/null \ + && echo "[CACHE] Moved legacy ${_name}/ → cache_write/${_name}/" \ + || echo "[CACHE] Failed to move legacy ${_name}/" + fi +done + +# vLLM: migrate the most recent legacy seed dir → cache_write/ (one-time, instant rename) +_vllm_write="${CACHE_WRITE_DIR}/vllm_compile_cache_${_vllm_cache_precision}" +_vllm_read_tar="${CACHE_READ_DIR}/vllm_compile_cache_${_vllm_cache_precision}.tar.zst" + +if [ ! -d "$_vllm_write" ] || [ -z "$(ls -A "$_vllm_write" 2>/dev/null)" ]; then + _best="$(ls -1dt \ + "${PERSISTENT_CACHE}/vllm_compile_cache_${_vllm_cache_precision}" \ + "${PERSISTENT_CACHE}/vllm_compile_cache_${_vllm_cache_precision}_"* \ + 2>/dev/null \ + | while IFS= read -r d; do + [ -d "$d" ] && [ -n "$(ls -A "$d" 2>/dev/null)" ] && echo "$d" && break + done + )" || true + if [ -n "$_best" ]; then + [ -d "$_vllm_write" ] && rmdir "$_vllm_write" 2>/dev/null || true + mv "$_best" "$_vllm_write" 2>/dev/null \ + && echo "[CACHE] Moved $(basename "$_best") → cache_write/vllm_compile_cache_${_vllm_cache_precision}/" \ + || echo "[CACHE] Failed to move vLLM cache" + fi +fi + +# Purge redundant legacy vLLM cache directories. +# The old sidecar wrote every vLLM seed as a separate directory on Lustre +# (e.g. vllm_compile_cache_bf16_2058, _3072, ...). With cache_write/ + tarball, +# only cache_write/vllm_compile_cache_{precision}/ matters. All seed copies are +# content-addressed duplicates — safe to remove after migration. +_purge_count=0 +for _d in "${PERSISTENT_CACHE}/vllm_compile_cache_${_vllm_cache_precision}" \ + "${PERSISTENT_CACHE}/vllm_compile_cache_${_vllm_cache_precision}_"*; do + [ -d "$_d" ] || continue + rm -rf "$_d" 2>/dev/null && (( _purge_count++ )) || true +done +for _d in "${PERSISTENT_CACHE}"/vllm_compile_cache_[0-9]*/; do + [ -d "$_d" ] || continue + rm -rf "$_d" 2>/dev/null && (( _purge_count++ )) || true +done +for _d in "${PERSISTENT_CACHE}/vllm_compile_cache" \ + "${PERSISTENT_CACHE}/vllm_compile_cache_warm"; do + [ -d "$_d" ] || continue + rm -rf "$_d" 2>/dev/null && (( _purge_count++ )) || true +done +if (( _purge_count > 0 )); then + echo "[CACHE] Purged ${_purge_count} redundant legacy vLLM cache directories from ${PERSISTENT_CACHE}/" +fi + +# ============================================================================= +# Code snapshot +# ============================================================================= +# Snapshot the git-tracked source tree so the code is frozen at submission time. +# This guarantees we know exactly which code was used for a given experiment. +# Set USE_SNAPSHOT=0 to skip (runs from container built-in or live checkout). +# Interactive mode defaults to the live checkout for fast iteration; batch snapshots. +if [[ "${INTERACTIVE}" == "1" ]]; then + USE_SNAPSHOT="${USE_SNAPSHOT:-0}" +else + USE_SNAPSHOT="${USE_SNAPSHOT:-1}" +fi + +if [[ "${USE_SNAPSHOT}" == "1" ]]; then + if [[ ! -f "${PROJECT_ROOT}/tools/code_snapshot.sh" ]]; then + echo "ERROR: tools/code_snapshot.sh not found at ${PROJECT_ROOT}/tools/code_snapshot.sh" >&2 + echo " Set USE_SNAPSHOT=0 to run from the live checkout instead." >&2 + exit 1 + fi + SNAPSHOT_DIR=$(bash "${PROJECT_ROOT}/tools/code_snapshot.sh" "${JOB_NAME}") + + if [[ -d "${PROJECT_ROOT}/3rdparty/vllm" ]] && [[ ! -e "${SNAPSHOT_DIR}/3rdparty/vllm" ]]; then + mkdir -p "${SNAPSHOT_DIR}/3rdparty" + ln -s "${PROJECT_ROOT}/3rdparty/vllm" "${SNAPSHOT_DIR}/3rdparty/vllm" + fi + + echo "Code snapshot: ${SNAPSHOT_DIR}" + OVERLAY_SOURCE="${SNAPSHOT_DIR}" +else + OVERLAY_SOURCE="${PROJECT_ROOT}" +fi + +# ============================================================================= +# Container mounts +# ============================================================================= +# By default, nemo_rl (Python package) and examples/configs (YAML configs) from +# the code snapshot are overlaid into the container. Everything else uses the +# container's built-in code at /opt/nemo-rl. +# +# To overlay additional components (e.g. a local Megatron-LM checkout), pass +# EXTRA_MOUNTS as a comma-separated list of host:container pairs: +# +# EXTRA_MOUNTS="/path/to/Megatron-LM:/opt/nemo-rl/3rdparty/Megatron-LM-workspace/Megatron-LM" bash ultra_launch.sh +# +# Container paths for reference: +# /opt/nemo-rl/nemo_rl — Python package +# /opt/nemo-rl/examples/configs — YAML configs +# /opt/nemo-rl/3rdparty/Megatron-LM-workspace/Megatron-LM — Megatron-LM +# /opt/nemo-rl/3rdparty/Megatron-Bridge-workspace/Megatron-Bridge — Megatron-Bridge +# /opt/nemo-rl/3rdparty/Gym-workspace/Gym — NeMo-Gym +# /opt/nemo-rl/3rdparty/vllm — vLLM +# ============================================================================= +_append_mount() { + if [[ -z "${MOUNTS}" ]]; then + MOUNTS="$1" + else + MOUNTS="${MOUNTS},$1" + fi +} + +if [[ -d "${OVERLAY_SOURCE}/nemo_rl" ]]; then + _append_mount "${OVERLAY_SOURCE}/nemo_rl:/opt/nemo-rl/nemo_rl" + echo " Mount: nemo_rl → /opt/nemo-rl/nemo_rl" +fi +if [[ -d "${OVERLAY_SOURCE}/examples/configs" ]]; then + _append_mount "${OVERLAY_SOURCE}/examples/configs:/opt/nemo-rl/examples/configs" + echo " Mount: configs → /opt/nemo-rl/examples/configs" +fi +if [[ -d "${OVERLAY_SOURCE}/3rdparty/Gym-workspace/Gym" ]]; then + _append_mount "${OVERLAY_SOURCE}/3rdparty/Gym-workspace/Gym:/opt/nemo-rl/3rdparty/Gym-workspace/Gym" + echo " Mount: Gym → /opt/nemo-rl/3rdparty/Gym-workspace/Gym" +fi + +if [[ "${USE_SNAPSHOT}" == "1" ]]; then + _append_mount "${SNAPSHOT_DIR}:${SNAPSHOT_DIR}" +fi + +if [[ -n "${EXTRA_MOUNTS:-}" ]]; then + _append_mount "${EXTRA_MOUNTS}" + echo " Extra mounts: ${EXTRA_MOUNTS}" +fi + +export MOUNTS + +# ============================================================================= +# Resolve ray.sub +# ============================================================================= +RAY_SUB="${RAY_SUB:-${PROJECT_ROOT}/ray.sub}" +if [[ ! -f "${RAY_SUB}" ]]; then + echo "ERROR: ray.sub not found at ${RAY_SUB}" >&2 + exit 1 +fi + +# ============================================================================= +# Per-node cache seeding (SETUP_COMMAND) +# ============================================================================= +# Triton, Inductor, and FlashInfer cubins compile/download to node-local /tmp to +# avoid Lustre race conditions and file lock contention during concurrent JIT +# compilation. To avoid cold-start penalties, we seed /tmp from a warm Lustre +# cache before Ray starts. +# +# IMPORTANT: Stale /tmp caches from previous jobs can cause hangs (e.g. the +# Triton bundler skipping non-empty temp dirs). We rm -rf /tmp caches first, +# then seed fresh from Lustre. +# ============================================================================= +read -r -d '' SETUP_COMMAND </dev/null 2>&1 || { apt-get update -qq && apt-get install -y -qq zstd; } 2>/dev/null || true +echo "[CACHE SEED] Clearing stale /tmp caches and seeding from Lustre..." +WARM_SEED="${NRL_VLLM_CACHE_SEED_DIR}" +LOCAL_IND="${INDUCTOR_CACHE_DIR}" +LOCAL_TRI="${TRITON_CACHE_DIR}" +CACHE_READ="${CACHE_READ_DIR}" + +# vLLM caches are per-instance (VLLM_CACHE_ROOT_{seed}). Clear ALL from prior jobs. +rm -rf /tmp/nemo_rl_vllm_cache /tmp/nemo_rl_vllm_cache_* +rm -rf "\$LOCAL_IND" "\$LOCAL_TRI" +mkdir -p "\$LOCAL_IND" "\$LOCAL_TRI" + +_seed_cache() { + local tarball="\$1" local_dir="\$2" name="\$3" + if [ -f "\$tarball" ]; then + tar --zstd -xf "\$tarball" -C "\$local_dir" \ + && echo "[CACHE SEED] \$name: seeded from tarball (\$(du -sh "\$local_dir" 2>/dev/null | cut -f1))" \ + || echo "[CACHE SEED] \$name: tarball extract failed (non-fatal)" + else + echo "[CACHE SEED] \$name: no warm cache on Lustre yet" + fi +} + +# Seed vLLM compile cache from cache_read/ tarball (one per precision). +rm -rf "\$WARM_SEED" +_vllm_tar="\$CACHE_READ/vllm_compile_cache_${_vllm_cache_precision}.tar.zst" +if [ -f "\$_vllm_tar" ]; then + mkdir -p "\$WARM_SEED" + tar --zstd -xf "\$_vllm_tar" -C "\$WARM_SEED" \ + && echo "[CACHE SEED] vLLM (${_vllm_cache_precision}): seeded from tarball (\$(du -sh "\$WARM_SEED" 2>/dev/null | cut -f1))" \ + || echo "[CACHE SEED] vLLM: tarball extract failed (non-fatal)" +else + echo "[CACHE SEED] vLLM: no warm cache on Lustre yet" +fi + +_seed_cache "\$CACHE_READ/inductor_cache.tar.zst" "\$LOCAL_IND" "Inductor" +_seed_cache "\$CACHE_READ/triton_cache.tar.zst" "\$LOCAL_TRI" "Triton" + +echo "[CACHE SEED] Done." +SETUPEOF +export SETUP_COMMAND + +# ============================================================================= +# Build the training command +# ============================================================================= +# Stage-specific hyperparameters (batch sizes, advantage clip, MoE parallelism, +# learning rate, etc.) live in CONFIG_PATH. The launcher only passes the +# per-run overrides: cluster shape, paths, judge endpoints, logging. +# ============================================================================= +TRAIN_CMD="cd ${CODE_ROOT} && date ; \ +${GENRM_RUNTIME_SETUP}\ +${VLLM_ENV_SOURCE}\ +OMP_NUM_THREADS=16 \ +RAY_DEDUP_LOGS=1 \ +WANDB_INIT_TIMEOUT=300 \ +VLLM_CACHE_ROOT=${NRL_VLLM_LOCAL_CACHE_DIR} \ +NRL_VLLM_CACHE_SEED_DIR=${NRL_VLLM_CACHE_SEED_DIR} \ +DG_JIT_CACHE_DIR=${NRL_VLLM_LOCAL_CACHE_DIR}/deep_gemm \ +TORCHINDUCTOR_CACHE_DIR=${INDUCTOR_CACHE_DIR} \ +TRITON_CACHE_DIR=${TRITON_CACHE_DIR} \ +UV_CACHE_DIR=/tmp/nemo-gym-uv-cache-\${SLURM_JOB_ID:-default} \ +UV_LOCK_TIMEOUT=1800 \ +RAY_ENABLE_UV_RUN_RUNTIME_ENV=0 \ +UV_HTTP_TIMEOUT=10 \ +VLLM_USE_FLASHINFER_MOE_FP8=1 \ +VLLM_FLASHINFER_MOE_BACKEND=latency \ +NRL_VLLM_ASYNC_TIMEOUT_SECONDS=1800 \ +NRL_WG_USE_RAY_REF=1 \ +HF_HOME=${HF_HOME:-} \ +HF_TOKEN=${HF_TOKEN:-} \ +NRL_USE_FASTOKENS=${NRL_USE_FASTOKENS:-1} \ +uv run ${NRL_ENTRYPOINT:-./examples/nemo_gym/run_grpo_nemo_gym.py} \ +--config ${CONFIG_PATH} \ +policy.model_name=${MODEL_PATH} \ +cluster.num_nodes=${NUM_ACTOR_NODES} \ +policy.generation.colocated.resources.num_nodes=${NUM_GEN_NODES} \ +env.nemo_gym.num_gpu_nodes=${NUM_GYM_NODES} \ +checkpointing.checkpoint_dir=${CHECKPOINT_DIR} \ +${CHECKPOINTING_SAVE_BY:+checkpointing.checkpoint_must_save_by=${CHECKPOINTING_SAVE_BY}} \ +data.train.data_path=${TRAIN_PATH} \ +data.validation.data_path=${VAL_PATH} \ +${GENRM_OVERRIDE:+${GENRM_OVERRIDE}} \ +${NL2BASH_JUDGE_MODEL:+env.nemo_gym.nl2bash_judge_model.responses_api_models.local_vllm_model.model=${NL2BASH_JUDGE_MODEL}} \ +${SAFETY_JUDGE_MODEL:+env.nemo_gym.safety_judge_model.responses_api_models.local_vllm_model.model=${SAFETY_JUDGE_MODEL}} \ +${SIF_DIR:+sif_dir=${SIF_DIR}} \ +env.nemo_gym.nemo_gym_log_dir=${LOG_DIR}/nemo_gym \ +logger.log_dir=${LOG_DIR} \ +logger.wandb_enabled=${WANDB_ENABLED} \ +logger.wandb.name=${WANDB_NAME} \ +logger.wandb.project=${WANDB_PROJ} \ +${NRL_MAX_STEPS:+grpo.max_num_steps=${NRL_MAX_STEPS}} \ +${MTP_EXTRA_ARGS} \ +${MOPD_OVERRIDES} \ +${*}" + +export COMMAND="${TRAIN_CMD}" + +# ============================================================================= +# Summary +# ============================================================================= +echo "" +echo "================================================================" +echo " Nemotron 3 Ultra — ${EXP_NAME} (${NUM_TOTAL_NODES}-node)" +echo "================================================================" +echo " Job name: ${JOB_NAME} (singleton — only one runs at a time)" +echo " Config: ${CONFIG_PATH}" +echo " Nodes: ${NUM_TOTAL_NODES} total (segment=${SEGMENT_SIZE})" +echo " Training: ${NUM_TRAIN_NODES} ($((NUM_TRAIN_NODES * GPUS_PER_NODE)) GPUs)" +echo " vLLM gen: ${NUM_GEN_NODES} ($((NUM_GEN_NODES * GPUS_PER_NODE)) GPUs)" +echo " Gym: ${NUM_GYM_NODES} ($((NUM_GYM_NODES * GPUS_PER_NODE)) GPUs)" +if (( NUM_TEACHER_NODES > 0 )); then +echo " Teachers: ${NUM_TEACHER_NODES} ($((NUM_TEACHER_NODES * GPUS_PER_NODE)) GPUs)" +fi +echo " Walltime: ${WALLTIME}" +echo "" +echo " Checkpoints: ${CHECKPOINT_DIR} (stable — auto-resumes across jobs)" +echo " Run dir: ${RUN_DIR}" +echo " Logs: ${LOG_DIR}" +echo " Slurm logs: ${SLURM_LOG_DIR}" +echo " W&B: ${WANDB_PROJ} / ${WANDB_NAME} (enabled=${WANDB_ENABLED})" +echo "" +echo " Model: ${MODEL_PATH}" +echo " Train data: ${TRAIN_PATH}" +echo " Val data: ${VAL_PATH}" +echo " Container: ${CONTAINER}" +echo " Custom vLLM: ${USE_CUSTOM_VLLM}" +echo " Sandbox: ${SANDBOX_CONTAINER}" +if [[ -n "${GENRM_BASE_URL_FILE}" ]]; then + echo " GenRM: runtime endpoint from ${GENRM_BASE_URL_FILE}" +elif [[ -n "${GENRM_BASE_URL}" ]]; then + echo " GenRM: remote endpoint ${GENRM_BASE_URL}" +elif [[ -n "${GENRM_MODEL}" ]]; then + echo " GenRM: local model ${GENRM_MODEL}" +fi +if [[ "${USE_SNAPSHOT}" == "1" ]]; then +echo " Snapshot: ${SNAPSHOT_DIR}" +fi +echo "" +echo " Monitor: squeue -u \$USER -n ${JOB_NAME}" +echo " Logs: tail -f ${SLURM_LOG_DIR}/*.out" +echo " Latest: ls -la ${RESULTS_DIR}/runs/latest" +echo "" +echo "================================================================" +echo "" + +# ============================================================================= +# Record code provenance in the run directory +# ============================================================================= +{ + echo "timestamp: $(date -Iseconds)" + echo "branch: $(git -C "${PROJECT_ROOT}" rev-parse --abbrev-ref HEAD 2>/dev/null || echo unknown)" + echo "commit: $(git -C "${PROJECT_ROOT}" rev-parse HEAD 2>/dev/null || echo unknown)" + echo "dirty: $(git -C "${PROJECT_ROOT}" status --porcelain 2>/dev/null | head -20)" + echo "snapshot: ${USE_SNAPSHOT}" + if [[ "${USE_SNAPSHOT}" == "1" ]]; then + echo "snapshot_dir: ${SNAPSHOT_DIR}" + fi + echo "container: ${CONTAINER}" + echo "config: ${CONFIG_PATH}" + echo "genrm_base_url_file: ${GENRM_BASE_URL_FILE}" + echo "command: ${TRAIN_CMD}" +} > "${RUN_DIR}/provenance.txt" + +# ============================================================================= +# Dry-run mode: print everything, don't submit +# ============================================================================= +DRY_RUN="${DRY_RUN:-0}" +if [[ "${DRY_RUN}" == "1" ]]; then + echo "DRY_RUN=1 — printing TRAIN_CMD and exiting without submission." + echo "" + echo "--- TRAIN_CMD ---" + echo "${TRAIN_CMD}" + echo "--- end ---" + exit 0 +fi + +# ============================================================================= +# Interactive mode: bring up Ray and idle for attachment (no training driver) +# ============================================================================= +# With COMMAND empty, ray.sub starts the Ray cluster, writes -attach.sh, +# then idles. We save the driver command to -run-cmd.sh so you can attach +# and run it by hand, edit it, and re-run without requeueing. +if [[ "${INTERACTIVE}" == "1" ]]; then + unset COMMAND 2>/dev/null || true # empty COMMAND -> ray.sub idle/interactive mode + WALLTIME="${INTERACTIVE_WALLTIME:-${WALLTIME}}" + + echo "" + echo "================================================================" + echo " INTERACTIVE MODE — ${NUM_TOTAL_NODES}-node allocation (walltime ${WALLTIME})" + echo " Ray will start and idle until you attach." + echo "================================================================" + + SBATCH_OUTPUT=$(sbatch \ + --nodes="${NUM_TOTAL_NODES}" \ + --account="${SLURM_ACCOUNT}" \ + --job-name="interactive-${JOB_NAME}" \ + --partition="${SLURM_PARTITION}" \ + --time="${WALLTIME}" \ + --gres=gpu:${GPUS_PER_NODE} \ + --exclusive \ + --mem=0 \ + --segment="${SEGMENT_SIZE}" \ + --output="${SLURM_LOG_DIR}/%j.out" \ + --error="${SLURM_LOG_DIR}/%j.err" \ + ${SLURM_QOS:+--qos="${SLURM_QOS}"} \ + ${EXCLUDE_NODES:+--exclude="${EXCLUDE_NODES}"} \ + ${SLURM_RESERVATION:+--reservation="${SLURM_RESERVATION}"} \ + --comment='{"OccupiedIdleGPUsJobReaper":{"exemptIdleTimeMins":"60","reason":"interactive","description":"interactive debugging"}}' \ + "${RAY_SUB}") + echo "${SBATCH_OUTPUT}" + JOB_ID=$(echo "${SBATCH_OUTPUT}" | grep -oP '\d+$') + [[ -z "${JOB_ID}" ]] && { echo "ERROR: could not parse job ID from sbatch output." >&2; exit 1; } + + LAUNCH_DIR="$(pwd)" + ATTACH_SCRIPT="${LAUNCH_DIR}/${JOB_ID}-attach.sh" + CMD_FILE="${LAUNCH_DIR}/${JOB_ID}-run-cmd.sh" + cat > "${CMD_FILE}" </dev/null || true) + [[ -z "${state}" ]] && { echo " Job ${JOB_ID} left the queue. Check: sacct -j ${JOB_ID}"; exit 1; } + [[ "${state}" != "${prev_state}" ]] && { echo " [$(date +%H:%M:%S)] state: ${state}"; prev_state="${state}"; } + sleep 15 + done + echo "" + echo " Ray is ready — attach: bash ${ATTACH_SCRIPT}" + fi + exit 0 +fi + +# ============================================================================= +# Submit +# ============================================================================= +# Always serialise same-name submissions via singleton; optionally chain after +# another job with SLURM_DEPENDENCY (e.g. "afterany:3044848" or "afterok:JOBID"). +SLURM_DEPENDENCY="${SLURM_DEPENDENCY:-}" +DEPENDENCY="singleton" +[[ -n "${SLURM_DEPENDENCY}" ]] && DEPENDENCY="singleton,${SLURM_DEPENDENCY}" + +SBATCH_OUTPUT=$(sbatch \ + --nodes="${NUM_TOTAL_NODES}" \ + --account="${SLURM_ACCOUNT}" \ + --job-name="${JOB_NAME}" \ + --partition="${SLURM_PARTITION}" \ + --time="${WALLTIME}" \ + --gres=gpu:${GPUS_PER_NODE} \ + --exclusive \ + --mem=0 \ + --dependency="${DEPENDENCY}" \ + --segment="${SEGMENT_SIZE}" \ + --output="${SLURM_LOG_DIR}/%j.out" \ + --error="${SLURM_LOG_DIR}/%j.err" \ + ${SLURM_QOS:+--qos="${SLURM_QOS}"} \ + ${EXCLUDE_NODES:+--exclude="${EXCLUDE_NODES}"} \ + ${SLURM_RESERVATION:+--reservation="${SLURM_RESERVATION}"} \ + --comment='{"OccupiedIdleGPUsJobReaper":{"exemptIdleTimeMins":"60","reason":"interactive","description":"interactive debugging"}}' \ + "${RAY_SUB}") + +echo "${SBATCH_OUTPUT}" +JOB_ID=$(echo "${SBATCH_OUTPUT}" | grep -oP '\d+$') + +if [[ -n "${JOB_ID}" ]]; then + echo "" + echo " Ray logs: ${BASE_LOG_DIR}/${JOB_ID}-logs/" + echo "" +fi From bdb60d453671a6b7fab0e968f5136c8faaae6f49 Mon Sep 17 00:00:00 2001 From: Zhiyu Li Date: Tue, 28 Jul 2026 15:16:29 -0700 Subject: [PATCH 02/52] fix(swe): drop the short QOS from the nano SWE recipe A 6-node submission was held with Reason=QOSMaxNodePerUserLimit: the short QOS enforces a per-user node cap that a larger concurrent job of the same user had already consumed. short also caps walltime at 2h, which is only just enough for a cold start (~60 GB checkpoint download plus Megatron conversion) followed by five steps. Submit with no QOS and WALLTIME=3:59:00, the batch partition maximum. ultra_launch.sh only auto-selects short below 2h, so an empty SLURM_QOS means no --qos flag. Opting back in is one env var. Signed-off-by: Zhiyu Li --- docs/guides/nano-swe-transferqueue.md | 11 +++++++---- swe_nano.env | 14 ++++++++++---- 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/docs/guides/nano-swe-transferqueue.md b/docs/guides/nano-swe-transferqueue.md index 97f2d7299d1..2fa5d1709f5 100644 --- a/docs/guides/nano-swe-transferqueue.md +++ b/docs/guides/nano-swe-transferqueue.md @@ -188,10 +188,13 @@ step_metrics={'loss': ..., 'grad_norm': ..., ...} usage was ~170 GB/rank of 189 GB. If you scale up context, batch, or model, expect the initial weight sync to OOM first; raise EP, enable `optimizer_cpu_offload`, or shorten the sequence. -- **Two-hour walltime.** `WALLTIME=1:59:00` is the max for the `short` QOS and is - the practical floor for this recipe. Shorter allocations expired mid-warmup - during development. For scale: with `num_prompts_per_step=2` a 4-rollout batch - took 6–8 minutes, and the default 64-rollout batch took ~11 minutes. +- **Walltime and QOS.** `WALLTIME=3:59:00` (the `batch` partition maximum) with + no QOS. Two hours is the practical floor — shorter allocations expired + mid-warmup during development. The recipe deliberately avoids the `short` QOS: + it trades a priority boost for a 2h ceiling and a per-user node cap, and a + 6-node job here was held with `Reason=QOSMaxNodePerUserLimit` while a larger + job of the same user was running. For scale: with `num_prompts_per_step=2` a + 4-rollout batch took 6–8 minutes; the default 64-rollout batch took ~11. - **Submodule pin.** This branch pins `3rdparty/Gym-workspace/Gym` to `v0.4.0` to match the container's prebuilt gym venvs (`skip_venv_if_present: true`). Bumping it without rebuilding the container forces a concurrent nemo-gym diff --git a/swe_nano.env b/swe_nano.env index 4efe23c23be..4bbe9e027fb 100644 --- a/swe_nano.env +++ b/swe_nano.env @@ -79,10 +79,16 @@ export NUM_GYM_NODES=0 # Total = 6; SEGMENT_SIZE must divide it (sbatch --segment). The nano .inc sets # cluster.segment_size=null to disable the virtual-cluster NVLink segment filter. export SEGMENT_SIZE=6 -export WALLTIME=1:59:00 # 2h (max for QOS short). 1h is NOT enough: the - # first run downloads ~60 GB from HF and converts - # to Megatron before the first rollout starts. -export SLURM_QOS=short +# 3:59:00 is the max for the `batch` partition. 1h is NOT enough: the first run +# downloads ~60 GB from HF and converts to Megatron before the first rollout. +export WALLTIME=3:59:00 +# Deliberately NOT the `short` QOS. short gives higher priority but caps walltime +# at 2h and enforces a per-user node limit — a 6-node job of ours got held with +# Reason=QOSMaxNodePerUserLimit while a large job of the same user was running. +# ultra_launch.sh only auto-selects short when WALLTIME < 2h, so leaving this +# empty submits with no --qos. Set SLURM_QOS=short (and WALLTIME under 2h) if you +# want the priority boost and know your node budget is free. +export SLURM_QOS= export NRL_MAX_STEPS=5 # Nano base checkpoint: MTP off, container's stock vLLM (no Ultra fork needed). From 9b5bdf68af9132110b03ff30979ad165f78eb7bc Mon Sep 17 00:00:00 2001 From: Pranav Prashant Thombre Date: Fri, 31 Jul 2026 15:50:10 -0700 Subject: [PATCH 03/52] feat(sc): gate-authoritative token capture on the Gym tokidcap stack MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Token-in/token-out capture for NeMo-Gym rollouts in the async SingleController GRPO path, rebuilt on the nano SWE recipe branch (3fcc69666, main + the TQ-honoured SWE recipe) per docs/design-docs/tq-gym-tokidcap-migration.md §9b.1a. The Gym submodule is pinned to pthombre/tq-tokidcap-capture (b6051536 = upstream tokidcap stack top 81ac2736/#2182 + the 7-commit gate/staging series; companion Gym PR NVIDIA-NeMo/Gym#2278). This squashes the capture series (S1-S5 primitives/worker/gate/receipts/ verification + the stack migration) into one commit ported onto the main-lineage SC: the granular history lives on the prior branch head 624bb277d. Highlights: - TQTokenSink/TQTokenSource: per-call token deltas staged from the vLLM worker straight into TransferQueue, durable before the response releases; BlackboxFinalizer rebuilds digest-verified always-N training rows via Gym's terminal-aware linearize over run_builder. - Identity: rollout ids ride the run body as the opaque _ng_rollout_id key (agents stamp /ng-rollout/; all agent impls unmodified); the middleware-minted model_call_id is the call id; TQ sample ids ARE the capture keys. - Gate hosting config: LineageIndex capacity derived from the training config; per-run control-plane bearer token; hard per-call control deadlines (gate death surfaces as failed dispatches + placeholders, not a silent retry stall); base capture dir under the run's log dir. - Receipt-mode dispatch in RolloutManager (_generate_and_finalize: reserve-with-ids, seal -> receipt, finalize_group, commit_finalized, always-N with placeholders), weight-version fan-out to workers, gate metrics logging (token_in_rate, fallback-by-cause). - Port adaptations onto main's structures: streaming run_rollouts (receipt postprocess branch + picklable aiohttp error re-raise), commit()'s pre-write and evicted-during-write guards keep main's semantics, legacy failure path keeps main's remove_group. Dormant by default: everything is behind token_capture.enabled=false. Working log: docs/design-docs/tq-gym-tokidcap-migration-log.md. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Prashant Thombre --- .../nemo_gym/grpo_qwen3_30ba3b_instruct.yaml | 37 ++ examples/swe_bench/README.md | 85 +++ .../swe_bench/grpo_qwen3_30b_async_swe.yaml | 436 ++++++++++++++ .../swe_bench/run_grpo_qwen3_30b_async_swe.sh | 560 ++++++++++++++++++ nemo_rl/algorithms/advantage_estimator.py | 9 +- .../algorithms/async_utils/replay_buffer.py | 184 ++++-- nemo_rl/algorithms/single_controller.py | 34 ++ .../single_controller_utils/config.py | 52 +- .../single_controller_utils/setup.py | 132 +++++ nemo_rl/data_plane/tq_token_sink.py | 188 ++++++ nemo_rl/distributed/virtual_cluster.py | 4 + nemo_rl/environments/nemo_gym.py | 225 ++++++- nemo_rl/experience/blackbox_finalizer.py | 374 ++++++++++++ nemo_rl/experience/rollout_manager.py | 217 ++++++- nemo_rl/experience/row_dump.py | 77 +++ .../models/generation/vllm/vllm_generation.py | 31 + .../generation/vllm/vllm_worker_async.py | 139 ++++- nemo_rl/utils/http_byte_counter.py | 82 +++ pyrefly.toml | 5 + swe/EXPERIMENT_LAUNCH.md | 129 ++++ swe/SWE_RUN.md | 193 ++++++ swe/aggregate_perf.py | 113 ++++ swe/launch_swe_ab.sh | 75 +++ swe/make_capture_config.py | 50 ++ .../L1_Functional_Tests_SingleController.sh | 3 + .../algorithms/test_advantage_validity.py | 54 ++ .../data_plane/test_blackbox_finalizer.py | 247 ++++++++ tests/unit/data_plane/test_tq_token_sink.py | 111 ++++ tests/unit/experience/test_rollout_manager.py | 212 +++++++ .../test_vllm_token_capture_hosting.py | 312 ++++++++++ .../test_single_controller.py | 8 +- .../test_single_controller_setup.py | 1 + .../test_tq_replay_buffer.py | 206 +++++++ 33 files changed, 4483 insertions(+), 102 deletions(-) create mode 100644 examples/swe_bench/README.md create mode 100644 examples/swe_bench/grpo_qwen3_30b_async_swe.yaml create mode 100644 examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh create mode 100644 nemo_rl/data_plane/tq_token_sink.py create mode 100644 nemo_rl/experience/blackbox_finalizer.py create mode 100644 nemo_rl/experience/row_dump.py create mode 100644 nemo_rl/utils/http_byte_counter.py create mode 100644 swe/EXPERIMENT_LAUNCH.md create mode 100644 swe/SWE_RUN.md create mode 100644 swe/aggregate_perf.py create mode 100755 swe/launch_swe_ab.sh create mode 100644 swe/make_capture_config.py create mode 100644 tests/unit/algorithms/test_advantage_validity.py create mode 100644 tests/unit/data_plane/test_blackbox_finalizer.py create mode 100644 tests/unit/data_plane/test_tq_token_sink.py create mode 100644 tests/unit/models/generation/test_vllm_token_capture_hosting.py diff --git a/examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml b/examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml index 68ff288e2a4..67cf722c662 100644 --- a/examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml +++ b/examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml @@ -172,3 +172,40 @@ checkpointing: # 4. The checkpoint time for this model is around 10 mins. checkpoint_must_save_by: "00:03:30:00" save_period: 1 + +# Gate-authoritative token capture (token-in/token-out via NeMo-Gym; see +# docs/design-docs/tq-gym-tokidcap-migration.md). Dormant by default: with +# enabled=false every legacy codepath behaves exactly as before. Requires the +# SingleController path with data_plane.enabled=true and the async vLLM +# backend; defaults live on TokenCaptureConfig +# (nemo_rl/algorithms/single_controller_utils/config.py). +token_capture: + enabled: false + # TQ partition holding per-call staged token deltas (finalizer-cleared). + staging_partition: "rollout_staging" + # continue: a failed worker-side stage poisons the rollout (placeholder row); + # abort: fails the whole rollout at the gate. + on_capture_failure: "continue" + # allow: train groups whose calls span a refit (staleness = group's oldest + # call version); reject: placeholder such rollouts. + mixed_weight_version_policy: "allow" + # Drop the whole group when fewer than this fraction of its rollouts + # produced valid rows (null keeps every group). + min_valid_fraction_per_group: null + # Gate-side cleanup backstops (seconds). + registration_ttl_s: 3600.0 + staging_ttl_s: 3600.0 + # Gym LineageIndex capacity (each in-flight rollout's cumulative tokens + # live in the gate process). null = derived from the training config: + # rollouts = 2 x max in-flight, tokens = rollouts x max sequence length. + # Size explicitly for agentic workloads with deep per-rollout call trees. + lineage_max_rollouts: null + lineage_max_tokens: null + # Bearer token for the gate's /ng-control/* routes. null = minted per run. + control_auth_token: null + # Hard deadline per control-plane call (gate death must surface as failed + # dispatches + placeholders, not a silent retry stall). + control_timeout_s: 60.0 + # Directory for the Gym base capture layer the gate rides on (stays + # essentially empty on the gate path). null = /gym_token_capture. + capture_dir: null diff --git a/examples/swe_bench/README.md b/examples/swe_bench/README.md new file mode 100644 index 00000000000..3f30f5f4520 --- /dev/null +++ b/examples/swe_bench/README.md @@ -0,0 +1,85 @@ +# Async GRPO on SWE-bench (Qwen3-30B-A3B) + +Launcher and recipe for agentic RL on SWE-bench via NeMo-Gym/OpenHands: +16 nodes by default (8 training + 8 generation, non-colocated), async GRPO +with staleness window 1. + +| File | Purpose | +|---|---| +| `run_grpo_qwen3_30b_async_swe.sh` | SLURM launcher (submits `ray.sub`) | +| `grpo_qwen3_30b_async_swe.yaml` | Recipe config | + +The launcher supports two entrypoints: + +- `SC_MODE=1` (default): `examples/run_grpo_single_controller.py` — + single-controller with the TransferQueue data plane. +- `SC_MODE=0`: `examples/nemo_gym/run_grpo_nemo_gym.py` — classic async GRPO + (async behavior comes from the yaml's `grpo.async_grpo` block). + +## Prerequisites + +1. A NeMo-RL checkout (or code snapshot) containing `ray.sub` at its root — + the launcher submits from the root it lives under. +2. An enroot container image of a recent NeMo-RL nightly. +3. The SWE train dataset (`.jsonl`). +4. The SWE sandbox images (apptainer/singularity `.sif` files for the + swe-bench / sweap instances). **Edit the `container_formatter` lists in the + yaml** to point at your local copies (they ship as `/path/to/...` + placeholders). +5. A HF checkpoint to train from. + +## Quick start + +Run from the repo root: + +```bash +ACCOUNT= \ +CONTAINER=/path/to/nemo-rl-nightly.sqsh \ +MODEL_PATH=/path/to/hf_checkpoint \ +TRAIN_DATA_PATH=/path/to/swe_train.jsonl \ +EXTRA_MOUNTS=/path/to/shared_fs:/path/to/shared_fs \ +bash examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh +``` + +`EXTRA_MOUNTS` must make the model / data / `.sif` locations visible inside +the container (the default mounts only cover the repo tree and the gym +source). Add `DRY_RUN=1` to print the sbatch command and config without +submitting. + +Secrets (`WANDB_API_KEY`, `HUGGINGFACE_TOKEN`, ...) are read from the calling +environment and never stored in this directory. The recommended pattern is a +small personal wrapper script that exports your site-specific paths, cluster +account, and secrets, then calls this launcher. + +## Common variations + +```bash +# Classic (non-single-controller) entrypoint +SC_MODE=0 bash examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh + +# Different training tensor parallelism +TP=2 bash examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh + +# streaming-2 dispatch semantics (over-generation + out-of-order consumption; +# default is streaming-1: strict repro of classic async_grpo age-1 dispatch) +OVER_SAMPLING=true FORCE_IN_ORDER=false \ +bash examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh + +# Finer intra-step dispatch: start training once 2 prompt groups are ready +# instead of waiting for the full batch (SC only; PPS=8 by default) +MIN_PROMPT_GROUPS=2 bash examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh + +# Smaller / shorter run +NUM_NODES=8 NUM_GEN_NODES=4 TIME=2:0:0 MAX_NUM_STEPS=5 \ +bash examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh +``` + +The full knob list (parallelism, batch sizes, staleness, agent turn limits, +caches, ...) is documented in the launcher's header comment. + +## Monitoring + +The submit banner prints the experiment name, W&B target, and log location. +The driver log lands at `/logs/slurm/-logs/ray-driver.log`; +training progress appears there as `train step ...` / `step_metrics=...` +lines and in W&B under `train/reward` and `timing/train/*`. diff --git a/examples/swe_bench/grpo_qwen3_30b_async_swe.yaml b/examples/swe_bench/grpo_qwen3_30b_async_swe.yaml new file mode 100644 index 00000000000..aa774156245 --- /dev/null +++ b/examples/swe_bench/grpo_qwen3_30b_async_swe.yaml @@ -0,0 +1,436 @@ +# ============================================================================ +# Async GRPO SWE RL Training: Qwen3-30B-A3B-Thinking-2507 +# +# Model: Qwen3-30B-A3B-Thinking-2507 (MoE, 30B total / 3B active, thinking) +# Train data: R2E-Gym (r2e-gym subset, 4518 samples) +# Eval data: SWE-bench Verified +# Mode: Async GRPO with non-colocated generation +# Entry: examples/nemo_gym/run_grpo_nemo_gym.py +# Env: swe_agents (OpenHands agent, singularity sandbox) +# +# Based on: bihu/nemo-rl-qwen-swe/grpo_qwen3_30b_thinking_swe.yaml +# Gym: main branch (nemo-rl-async-swe repo) +# ============================================================================ + +checkpointing: + enabled: true + checkpoint_dir: "results/grpo-qwen3-30b-thinking-swe-rl" + metric_name: "train:total_reward/mean" + higher_is_better: true + keep_top_k: 100 + save_period: 5 + checkpoint_must_save_by: "00:03:35:00" + model_save_format: "safetensors" + save_consolidated: false + save_optimizer: true + +grpo: + num_prompts_per_step: 16 + num_generations_per_prompt: 16 + num_val_generations_per_prompt: 1 + max_rollout_turns: 1 + max_num_epochs: 100 + max_num_steps: 1000000 + normalize_rewards: true + use_leave_one_out_baseline: true + advantage_clip_low: -100 + advantage_clip_high: 100 + val_period: 10 + val_at_start: false + val_at_end: false + overlong_filtering: true + max_val_samples: null + val_batch_size: 256 + seed: 42 + invalid_tool_call_strategy: "" + + use_dynamic_sampling: false + dynamic_sampling_max_gen_batches: 10 + batch_multiplier: 1 + + penalize_invalid_tool_call: true + invalid_tool_call_advantage: -5.0 + penalize_malformed_thinking: true + malformed_thinking_advantage: -5.0 + + reward_shaping: + enabled: false + overlong_buffer_length: 128 + overlong_buffer_penalty: 1 + max_response_length: ${policy.max_total_sequence_length} + stop_properly_penalty_coef: null + reward_scaling: + enabled: false + source_min: 0.0 + source_max: 1.0 + target_min: 0.0 + target_max: 1.0 + + async_grpo: + enabled: true + max_trajectory_age_steps: 1 + in_flight_weight_updates: true + recompute_kv_cache_after_weight_updates: false + + seq_logprob_error_threshold: 2 + +loss_fn: + reference_policy_kl_penalty: 0.0 + reference_policy_kl_type: "k3" + kl_input_clamp_value: null + kl_output_clamp_value: null + ratio_clip_min: 0.2 + ratio_clip_max: 0.28 + ratio_clip_c: null + use_on_policy_kl_approximation: true + use_importance_sampling_correction: true + truncated_importance_sampling_ratio: 5.0 + truncated_importance_sampling_ratio_min: null + truncated_importance_sampling_type: tis + sequence_level_importance_ratios: false + token_level_loss: true + force_on_policy_ratio: true + use_kl_in_reward: false + +policy: + model_name: "Qwen/Qwen3-30B-A3B-Thinking-2507" + tokenizer: + name: ${policy.model_name} + chat_template_kwargs: + enable_thinking: true + hf_config_overrides: {} + train_global_batch_size: 256 + train_micro_batch_size: 1 + generation_batch_size: 64 + logprob_batch_size: 1 + max_total_sequence_length: 131072 + precision: "bfloat16" + logprob_chunk_size: 2048 + offload_optimizer_for_logprob: false + + dtensor_cfg: + _v2: true + enabled: false + cpu_offload: False + sequence_parallel: false + activation_checkpointing: false + tensor_parallel_size: 1 + context_parallel_size: 1 + custom_parallel_plan: null + + megatron_cfg: + enabled: true + gradient_accumulation_fusion: false + empty_unused_memory_level: 1 + activation_checkpointing: true + tensor_model_parallel_size: 2 + expert_tensor_parallel_size: 1 + expert_model_parallel_size: 8 + pipeline_model_parallel_size: 2 + num_layers_in_first_pipeline_stage: null + num_layers_in_last_pipeline_stage: null + context_parallel_size: 4 + pipeline_dtype: ${policy.precision} + sequence_parallel: true + freeze_moe_router: true + moe_router_dtype: "fp32" + moe_router_load_balancing_type: "none" + moe_router_bias_update_rate: 1.0e-3 + moe_permute_fusion: true + moe_enable_deepep: false + moe_token_dispatcher_type: "alltoall" + moe_aux_loss_coeff: 0.0 + moe_router_enable_expert_bias: true + moe_shared_expert_overlap: false + apply_rope_fusion: True + bias_activation_fusion: False + defer_fp32_logits: True + moe_per_layer_logging: True + + optimizer: + optimizer: "adam" + lr: 1.0e-6 + min_lr: 1.0e-6 + weight_decay: 0.0 + bf16: true + fp16: false + params_dtype: "float32" + adam_beta1: 0.9 + adam_beta2: 0.999 + adam_eps: 1e-8 + sgd_momentum: 0.9 + use_distributed_optimizer: true + use_precision_aware_optimizer: true + clip_grad: ${policy.max_grad_norm} + optimizer_cpu_offload: false + optimizer_offload_fraction: 0.0 + + scheduler: + start_weight_decay: ${policy.megatron_cfg.optimizer.weight_decay} + end_weight_decay: ${policy.megatron_cfg.optimizer.weight_decay} + weight_decay_incr_style: "constant" + lr_decay_style: "constant" + lr_decay_iters: 1000000 + lr_warmup_iters: 0 + lr_warmup_init: 0 + + distributed_data_parallel_config: + grad_reduce_in_fp32: false + overlap_grad_reduce: false + overlap_param_gather: false + use_custom_fsdp: false + data_parallel_sharding_strategy: "optim_grads_params" + + mtp_loss_scaling_factor: 0.0 + mtp_use_repeated_layer: false + mtp_num_layers: 0 + mtp_detach_heads: false + + fp8_cfg: + enabled: false + fp8: "e4m3" + fp8_recipe: "blockwise" + fp8_param: false + + env_vars: null + + dynamic_batching: + enabled: False + train_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.train_micro_batch_size}} + logprob_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.logprob_batch_size}} + sequence_length_round: 64 + + sequence_packing: + enabled: True + train_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.train_micro_batch_size}} + logprob_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.logprob_batch_size}} + algorithm: "modified_first_fit_decreasing" + sequence_length_round: 64 + + make_sequence_length_divisible_by: 8 + max_grad_norm: 1.0 + + optimizer: null + scheduler: null + + generation: + port_range_low: 11001 + port_range_high: 15000 + backend: "vllm" + max_new_tokens: ${policy.max_total_sequence_length} + temperature: 1.0 + top_p: 1.0 + top_k: null + stop_token_ids: null + stop_strings: null + vllm_cfg: + enable_prefix_caching: true + async_engine: true + precision: ${policy.precision} + kv_cache_dtype: "auto" + tensor_parallel_size: 2 + pipeline_parallel_size: 1 + expert_parallel_size: 1 + gpu_memory_utilization: 0.8 + max_model_len: ${policy.max_total_sequence_length} + enforce_eager: False + enforce_monotonicity: false + use_deep_gemm: False + num_last_layers_in_bf16: 0 + num_first_layers_in_bf16: 0 + enable_vllm_metrics_logger: true + vllm_metrics_logger_interval: 0.5 + expose_http_server: true + skip_tokenizer_init: false + enable_thinking: true + http_server_serving_chat_kwargs: + enable_auto_tools: true + tool_parser: hermes + reasoning_parser: deepseek_r1 + chat_template: | + {%- if tools %} + {{- '<|im_start|>system\n' }} + {%- if messages[0].role == 'system' %} + {{- messages[0].content + '\n\n' }} + {%- endif %} + {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n" }} + {%- for tool in tools %} + {{- "\n" }} + {{- tool | tojson }} + {%- endfor %} + {{- "\n\n\nFor each function call, return a json object with function name and arguments within XML tags:\n\n{\"name\": , \"arguments\": }\n<|im_end|>\n" }} + {%- else %} + {%- if messages[0].role == 'system' %} + {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }} + {%- endif %} + {%- endif %} + {%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %} + {%- for message in messages[::-1] %} + {%- set index = (messages|length - 1) - loop.index0 %} + {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %} + {%- set ns.multi_step_tool = false %} + {%- set ns.last_query_index = index %} + {%- endif %} + {%- endfor %} + {%- for message in messages %} + {%- if message.content is string %} + {%- set content = message.content %} + {%- else %} + {%- set content = '' %} + {%- endif %} + {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} + {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }} + {%- elif message.role == "assistant" %} + {%- set reasoning_content = '' %} + {%- if message.reasoning_content is string %} + {%- set reasoning_content = message.reasoning_content %} + {%- else %} + {%- if '' in content %} + {%- set reasoning_content = content.split('')[0].rstrip('\n').split('')[-1].lstrip('\n') %} + {%- set content = content.split('')[-1].lstrip('\n') %} + {%- endif %} + {%- endif %} + {%- if reasoning_content %} + {{- '<|im_start|>' + message.role + '\n\n' + reasoning_content.strip('\n') + '\n\n\n' + content.lstrip('\n') }} + {%- else %} + {{- '<|im_start|>' + message.role + '\n' + content }} + {%- endif %} + {%- if message.tool_calls %} + {%- for tool_call in message.tool_calls %} + {%- if (loop.first and content) or (not loop.first) %} + {{- '\n' }} + {%- endif %} + {%- if tool_call.function %} + {%- set tool_call = tool_call.function %} + {%- endif %} + {{- '\n{"name": "' }} + {{- tool_call.name }} + {{- '", "arguments": ' }} + {%- if tool_call.arguments is string %} + {{- tool_call.arguments }} + {%- else %} + {{- tool_call.arguments | tojson }} + {%- endif %} + {{- '}\n' }} + {%- endfor %} + {%- endif %} + {{- '<|im_end|>\n' }} + {%- elif message.role == "tool" %} + {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %} + {{- '<|im_start|>user' }} + {%- endif %} + {{- '\n\n' }} + {{- content }} + {{- '\n' }} + {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %} + {{- '<|im_end|>\n' }} + {%- endif %} + {%- endif %} + {%- endfor %} + {%- if add_generation_prompt %} + {{- '<|im_start|>assistant\n\n' }} + {%- endif %} + default_chat_template_kwargs: + enable_thinking: true + truncate_history_thinking: false + + vllm_kwargs: + mamba_ssm_cache_dtype: "float32" + # The default flashinfer MoE backend (trtllm_bf16_moe) uses a fused/quantized + # expert weight layout that does NOT match the megatron weight-refit broadcast, + # producing garbage generations (and can hang broadcast_weights_for_collective). + # triton uses the standard expert layout the refit broadcast expects. + # Mirrors ruit's grpo_qwen3_30b_async_swe_hsg.yaml on the swe2-scale-gen branch. + moe_backend: triton + compilation_config: + cudagraph_capture_sizes: [1,2,4,8,16,32,64] + + colocated: + enabled: false + resources: + gpus_per_node: 8 + num_nodes: 4 + +data: + max_input_seq_length: null + shuffle: false + num_workers: 1 + use_multiple_dataloader: false + train: + # Site-specific; the launcher always overrides this via ++data.train.data_path. + data_path: "/path/to/swe_train_dataset.jsonl" + validation: + # Site-specific; the launcher always overrides this via ++data.validation.data_path. + data_path: "/path/to/swe_val_dataset.jsonl" + default: + dataset_name: NemoGymDataset + env_name: "nemo_gym" + prompt_file: null + system_prompt_file: null + processor: "nemo_gym_data_processor" + +env: + should_use_nemo_gym: true + should_log_nemo_gym_responses: false + nemo_gym: + skip_venv_if_present: true + port_range_low: 15001 + port_range_high: 20000 + config_paths: + - responses_api_models/vllm_model/configs/vllm_model_for_training.yaml + - responses_api_agents/swe_agents/configs/swebench_openhands_training.yaml + swe_agents_train: + responses_api_agents: + swe_agents: + agent_max_turns: 100 + concurrency: 768 + swebench_agent_timeout: 3600 + run_with_mixed_prompts: true + dataset_path: ${data.train.data_path} + # Site-specific: apptainer/singularity .sif sandbox images for the SWE + # datasets, tried in order per instance. Point these at your local + # copies of the swe-bench / sweap image sets. + container_formatter: + - "/path/to/swe-bench-images/swebench_sweb.eval.x86_64.{instance_id}.sif" + - "/path/to/sweap-images/sweap.{instance_id}.sif" + - "/path/to/swe-bench-images/namanjain12_{instance_id}.sif" + - "/path/to/sweap-images/sweap.{instance_id}.sif" + - "/path/to/swe-bench-images/swebench_sweb.eval.x86_64{instance_id}.sif" + swe_agents_val: + responses_api_agents: + swe_agents: + agent_max_turns: 200 + concurrency: 768 + swebench_agent_timeout: 3600 + dataset_path: ${data.validation.data_path} + # Site-specific: see the note on swe_agents_train.container_formatter. + container_formatter: + - "/path/to/swe-bench-images/swebench_sweb.eval.x86_64.{instance_id}.sif" + - "/path/to/sweap-images/sweap.{instance_id}.sif" + - "/path/to/swe-bench-images/namanjain12_{instance_id}.sif" + - "/path/to/sweap-images/sweap.{instance_id}.sif" + - "/path/to/swe-bench-images/swebench_sweb.eval.x86_64{instance_id}.sif" + use_absolute_ip: true + +logger: + log_dir: "logs" + num_val_samples_to_print: 0 + wandb_enabled: true + tensorboard_enabled: false + mlflow_enabled: false + monitor_gpus: true + swanlab_enabled: false + wandb: + project: "ruit-nemo-rl" + name: "qwen3-30b-thinking-swe-rl" + tensorboard: {} + mlflow: + experiment_name: "qwen3-30b-thinking-swe-rl" + run_name: "qwen3-30b-thinking-swe-rl" + gpu_monitoring: + collection_interval: 10 + flush_interval: 10 + +cluster: + gpus_per_node: 8 + num_nodes: 16 diff --git a/examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh b/examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh new file mode 100644 index 00000000000..182274a3c5a --- /dev/null +++ b/examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh @@ -0,0 +1,560 @@ +#!/bin/bash +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= +# Async GRPO on SWE-bench (NeMo-Gym / OpenHands) — Qwen3-30B-A3B, SLURM launcher. +# +# Submits a 16-node (8 train + 8 generation, non-colocated) async GRPO run via +# ray.sub. Supports both entrypoints: +# SC_MODE=1 (default) examples/run_grpo_single_controller.py +# (single-controller + TransferQueue data plane) +# SC_MODE=0 examples/nemo_gym/run_grpo_nemo_gym.py +# (classic async GRPO; async comes from the yaml's +# grpo.async_grpo block) +# +# This file is environment-agnostic: no secrets and no user-specific paths. +# Site-specific values come from the environment (see REQUIRED below) — wrap +# this script with your own launcher that exports them (see +# test_assets/SWE/grpo_swe_tests.sh for an example wrapper). +# +# --------------------------------------------------------------------------- +# REQUIRED environment: +# ACCOUNT SLURM account +# CONTAINER enroot .sqsh/.squashfs image (a recent NeMo-RL nightly; +# must postdate the TransferQueue pyproject dependency) +# MODEL_PATH HF checkpoint dir to train from +# TRAIN_DATA_PATH SWE train jsonl +# +# Common optional environment (see defaults inline for the full list): +# PARTITION (batch), NUM_NODES (16), NUM_GEN_NODES (8), TIME (4:0:0) +# VAL_DATA_PATH (=TRAIN_DATA_PATH), CONFIG_FILE (yaml next to this script) +# TP/EP/CP/PP/VLLM_TP, SEQLEN, PPS/GPP/GBS, LR, MAX_NUM_STEPS +# SC_MODE (1), MIN_PROMPT_GROUPS (=PPS) +# OVER_SAMPLING (false), FORCE_IN_ORDER (true) +# streaming-1 (default): OVER_SAMPLING=false FORCE_IN_ORDER=true +# — no over-generation, each step consumes the groups dispatched +# for it; 1:1 repro of the classic async_grpo (age=1) dispatch. +# streaming-2: OVER_SAMPLING=true FORCE_IN_ORDER=false +# — generation keeps producing, steps consume any groups within +# the staleness window; stale groups get evicted (wasted). +# PERSISTENT_CACHE compile/uv cache root on shared fs (~/.cache/... default) +# GYM_VENV_DIR /tmp/nemo_gym_venvs default; /opt/gym_venvs if your image +# has the gym server venvs baked in +# MOUNTS / EXTRA_MOUNTS container mounts (make model/data paths visible!) +# WANDB_API_KEY + WANDB_PROJ + EXP_SUFFIX logging & naming +# HUGGINGFACE_TOKEN / GITHUB_TOKEN / GITLAB_TOKEN passed through if set +# REAPER_COMMENT SLURM --comment payload (cluster-specific; empty default) +# DRY_RUN=1 print everything, submit nothing +# +# Usage (from a repo/snapshot root that contains ray.sub): +# ACCOUNT=... CONTAINER=... MODEL_PATH=... TRAIN_DATA_PATH=... \ +# bash examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh +# ============================================================================= + +set -e + +# --------------------------------------------------------------------------- +# Locate the tree to submit from: this script lives at examples/swe_bench/ +# inside the repo (or a code snapshot); ray.sub must exist at its root. +# --------------------------------------------------------------------------- +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +RUN_DIR="${RUN_DIR:-$(cd "${SCRIPT_DIR}/../.." && pwd)}" +if [ ! -f "${RUN_DIR}/ray.sub" ]; then + echo "Error: ${RUN_DIR}/ray.sub not found — RUN_DIR must be a NeMo-RL checkout/snapshot root." >&2 + exit 1 +fi +RUN_COMMIT="unknown" +if git -C "${RUN_DIR}" rev-parse --short HEAD >/dev/null 2>&1; then + RUN_COMMIT="$(git -C "${RUN_DIR}" rev-parse --short HEAD)" +elif [ -f "${RUN_DIR}/commit.txt" ]; then + RUN_COMMIT="$(cut -c1-7 "${RUN_DIR}/commit.txt")" +fi + +# --------------------------------------------------------------------------- +# Required site-specific inputs — fail fast with a clear message. +# --------------------------------------------------------------------------- +missing="" +[ -n "${ACCOUNT:-}" ] || missing+=" ACCOUNT" +[ -n "${CONTAINER:-}" ] || missing+=" CONTAINER" +[ -n "${MODEL_PATH:-}" ] || missing+=" MODEL_PATH" +[ -n "${TRAIN_DATA_PATH:-}" ] || missing+=" TRAIN_DATA_PATH" +if [ -n "${missing}" ]; then + echo "Error: missing required environment variables:${missing}" >&2 + echo "See the header of this script for the full contract." >&2 + exit 1 +fi + +# --------------------------------------------------------------------------- +# Cluster / submission (site-specific, all overridable) +# --------------------------------------------------------------------------- +PARTITION="${PARTITION:-batch}" +GPUS_PER_NODE="${GPUS_PER_NODE:-8}" +CPUS_PER_WORKER="${CPUS_PER_WORKER:-114}" +# SLURM --comment payload (e.g. idle-GPU-reaper exemptions on some clusters). +REAPER_COMMENT="${REAPER_COMMENT:-}" + +# --------------------------------------------------------------------------- +# Scale / walltime +# --------------------------------------------------------------------------- +NUM_NODES="${NUM_NODES:-16}" # total allocation +NUM_GEN_NODES="${NUM_GEN_NODES:-8}" # carved out of NUM_NODES for generation +TIME="${TIME:-4:0:0}" +MAX_NUM_STEPS="${MAX_NUM_STEPS:-}" # empty => use the yaml's value + +# --------------------------------------------------------------------------- +# Paths +# --------------------------------------------------------------------------- +CONFIG_FILE="${CONFIG_FILE:-${SCRIPT_DIR}/grpo_qwen3_30b_async_swe.yaml}" +VAL_DATA_PATH="${VAL_DATA_PATH:-${TRAIN_DATA_PATH}}" + +# ============================ Parallelism ============================ +TP="${TP:-4}"; EP="${EP:-8}"; CP="${CP:-4}"; PP="${PP:-2}"; VLLM_TP="${VLLM_TP:-2}" +MIN_PAD=1 +[ "${CP}" -gt 1 ] && MIN_PAD=$((MIN_PAD * CP * 2)) +[ "${TP}" -gt 1 ] && MIN_PAD=$((MIN_PAD * TP)) +MAKE_SEQ_DIVISIBLE_BY="${MIN_PAD}" +SEQUENCE_PACKING=True + +# ===================== Sequence length ===================== +SEQLEN="${SEQLEN:-131072}" + +# ===== Single-controller async-RL knobs (SC_MODE=1: data_plane + async_rl) ===== +# Maps the classic async_grpo (age=1) 1:1 to async_rl. +MAX_TRAJECTORY_AGE_STEPS="${MAX_TRAJECTORY_AGE_STEPS:-1}" # -> async_rl.max_weight_staleness_versions +BATCH_SELECTION_STRATEGY=staleness_window +# Dispatch semantics ("streaming" modes). Defaults = strict 1:1 repro of the +# classic async_grpo age-1 behavior (streaming-1): no over-generation, rollouts +# consumed strictly by their dispatch-target step. streaming-2 = the code +# defaults (over_sampling=true force_in_order=false): generation keeps +# producing (stale groups get evicted/wasted) and steps consume any groups +# inside the staleness window, out of order. +OVER_SAMPLING="${OVER_SAMPLING:-false}" +FORCE_IN_ORDER="${FORCE_IN_ORDER:-true}" +FORCE_ON_POLICY_RATIO=True +SEQ_LOGPROB_ERROR_THRESHOLD=null +COLOCATED_ENABLED=False +VLLM_GPU_UTIL=0.8 +OVERLAP_GRAD_REDUCE=False +ADVANTAGE_CLIP_LOW=-100 +ADVANTAGE_CLIP_HIGH=100 +TIS_THRESHOLD=5 + +# ========================= GRPO / sampling ========================= +PPS="${PPS:-8}"; GPP="${GPP:-8}"; GBS="${GBS:-64}" +# Intra-step dispatch granularity (SC only): the sampler releases work to the +# trainer once this many complete prompt groups are ready (gradient +# accumulation; the optimizer still steps only after PPS groups). Not passed in +# => same as PPS = fully synchronous within the step; smaller values overlap +# trainer compute with the generation tail. +MIN_PROMPT_GROUPS="${MIN_PROMPT_GROUPS:-${PPS}}" +NORMALIZE_REWARDS=True +OVERLONG_FILTERING=True +VAL_PERIOD="${VAL_PERIOD:-1000}" + +# ========================== Loss function ========================== +KL=0 +CLIP_MIN=0.2 +CLIP_MAX=0.28 +USE_ON_POLICY_KL_APPROXIMATION=True +IMPORTANCE_SAMPLING_CORRECTION=True +SEQ_LEVEL_IS=False +TOKEN_LEVEL_LOSS=True + +# ============================ Optimizer ============================ +LR="${LR:-1e-06}" + +# =============================== MoE =============================== +MOE_FREEZE_ROUTER=True +MOE_PERMUTE_FUSION=True +MOE_ENABLE_DEEPEP=False +MOE_TOKEN_DISPATCHER_TYPE="alltoall" +MOE_AUX_LOSS_COEFF=0 +MOE_ROUTER_LOAD_BALANCING_TYPE="none" +MOE_ROUTER_BIAS_UPDATE_RATE="1e-3" + +# ======================= Generation / vLLM ======================= +TEMPERATURE=1.0 + +# =================== Checkpointing & validation =================== +SAVE_PERIOD="${SAVE_PERIOD:-5}" +KEEP_TOP_K="${KEEP_TOP_K:-2}" +MUST_SAVE_BY="${MUST_SAVE_BY:-00:03:35:00}" # graceful save+exit before TIME + +# ============================ SWE agent ============================ +AGENT_MAX_TURNS="${AGENT_MAX_TURNS:-200}" +AGENT_TIMEOUT="${AGENT_TIMEOUT:-1800}" + +# ============================== Logging ============================== +WANDB_PROJ="${WANDB_PROJ:-nemo-rl-swe-bench}" +LOG_GYM_RESPONSES=true + +# ========================= Experiment naming ========================= +SYNC_MODE="async-age${MAX_TRAJECTORY_AGE_STEPS}" +EXP_SUFFIX="${EXP_SUFFIX:-swe-sc@${RUN_COMMIT}-${SYNC_MODE}-pps${PPS}-gpp${GPP}-gbs${GBS}-lr${LR}-tp${TP}}" +WANDB_NAME="${EXP_SUFFIX}" +EXP_NAME="${EXP_SUFFIX}" +CHECKPOINT_ROOT="${CHECKPOINT_ROOT:-${RUN_DIR}/results}" +CHECKPOINT_DIR="${CHECKPOINT_DIR:-${CHECKPOINT_ROOT}/${EXP_SUFFIX}}" +BASE_LOG_DIR="${BASE_LOG_DIR:-${RUN_DIR}/logs/slurm}" + +# ========================= Runtime env ========================= +# Secrets are passed through from the caller's environment; never set here. +export HUGGINGFACE_TOKEN="${HUGGINGFACE_TOKEN:-${HF_TOKEN:-}}" +export GITHUB_TOKEN="${GITHUB_TOKEN:-}" +export GITLAB_TOKEN="${GITLAB_TOKEN:-}" +export WANDB_API_KEY="${WANDB_API_KEY:-}" +export HF_HOME="${HF_HOME:-${HOME}/.cache/huggingface}" +export HF_DATASETS_CACHE="${HF_DATASETS_CACHE:-${HF_HOME}/datasets}" +# Node-local uv cache (ephemeral). Each node builds its own venv independently — +# a shared-filesystem uv cache corrupts under concurrent multi-node builds. +# Pre-warm LUSTRE_UV_CACHE once with a single process to skip long compiles. +export UV_CACHE_DIR=/tmp/uv_cache +export UV_LOCK_TIMEOUT=3600 +export RAY_DEDUP_LOGS=1 +export SSL_CERT_FILE="${SSL_CERT_FILE:-/etc/ssl/certs/ca-certificates.crt}" +export REQUESTS_CA_BUNDLE="${REQUESTS_CA_BUNDLE:-/etc/ssl/certs/ca-certificates.crt}" +export CURL_CA_BUNDLE="${CURL_CA_BUNDLE:-/etc/ssl/certs/ca-certificates.crt}" +export OMP_NUM_THREADS="${OMP_NUM_THREADS:-16}" + +# ===================== Shared-fs compile caches ===================== +# Runtime caches stay node-local (/tmp); the shared-fs copies are only seeded +# from (read) at startup and written back by a periodic sidecar, so repeat runs +# skip the ~20min triton MoE JIT. +PERSISTENT_CACHE="${PERSISTENT_CACHE:-${HOME}/.cache/nemo_rl_swe_bench}" +export LUSTRE_VLLM_CACHE="${PERSISTENT_CACHE}/vllm_compile_cache" +export LUSTRE_INDUCTOR_CACHE="${PERSISTENT_CACHE}/inductor_cache" +export LUSTRE_TRITON_CACHE="${PERSISTENT_CACHE}/triton_cache" +export LUSTRE_UV_CACHE="${PERSISTENT_CACHE}/uv_cache" +export NRL_VLLM_LOCAL_CACHE_DIR="/tmp/nemo_rl_vllm_cache" +export NRL_VLLM_CACHE_SEED_DIR="/tmp/nemo_rl_vllm_cache_warm" +export INDUCTOR_CACHE_DIR="/tmp/nemo_rl_inductor_cache" +export TRITON_CACHE_DIR="/tmp/nemo_rl_triton_cache" +export CACHE_SYNC_FREQUENCY="${CACHE_SYNC_FREQUENCY:-120}" +mkdir -p "${LUSTRE_VLLM_CACHE}" "${LUSTRE_INDUCTOR_CACHE}" "${LUSTRE_TRITON_CACHE}" "${LUSTRE_UV_CACHE}" + +# ===================== NeMo-Gym server venvs ===================== +# Default: node-local build (safe everywhere). If your container has the gym +# server venvs baked in (vllm_model + swe_agents), export GYM_VENV_DIR=/opt/gym_venvs. +# Do NOT point this at a shared filesystem: the editable nemo-gym build hangs on +# uv's flock there, and interrupted builds leave empty venv shells that +# skip-if-present then reuses -> the gym policy_model server crashes on import. +GYM_VENV_DIR="${GYM_VENV_DIR:-/tmp/nemo_gym_venvs}" +case "${GYM_VENV_DIR}" in /opt/*|/tmp/*) ;; *) mkdir -p "${GYM_VENV_DIR}" ;; esac + +# ===== SETUP_COMMAND: install apptainer + seed caches + uv sync ===== +# Runs on all nodes before Ray starts (consumed by ray.sub). +read -r -d '' SETUP_COMMAND </dev/null || true +RET=1 +RETRIES=3 +for attempt in \$(seq 1 \$RETRIES); do + if command -v apptainer >/dev/null 2>&1 || command -v singularity >/dev/null 2>&1; then + echo "[SETUP] singularity/apptainer already available" + RET=0 + break + fi + cd /tmp && \ + wget --no-check-certificate -q https://github.com/apptainer/apptainer/releases/download/v1.3.1/apptainer_1.3.1_amd64.deb && \ + apt install -y ./apptainer_1.3.1_amd64.deb && \ + ln -sf /usr/bin/apptainer /usr/bin/singularity + if command -v apptainer >/dev/null 2>&1; then + echo "[SETUP] apptainer installed successfully" + RET=0 + break + fi + echo "[SETUP] apptainer install attempt \$attempt failed, retrying..." + sleep 10 +done +if [ \$RET -ne 0 ]; then + echo "[SETUP] WARNING: apptainer installation failed after \$RETRIES attempts" +fi + +echo "[CACHE SEED] Clearing stale /tmp caches and seeding from shared fs..." +rm -rf /tmp/nemo_rl_vllm_cache /tmp/nemo_rl_vllm_cache_* +rm -rf "${INDUCTOR_CACHE_DIR}" "${TRITON_CACHE_DIR}" +mkdir -p "${INDUCTOR_CACHE_DIR}" "${TRITON_CACHE_DIR}" + +find "${LUSTRE_INDUCTOR_CACHE}" -maxdepth 1 -name '.tmp_*' -mmin +30 -exec rm -rf {} + 2>/dev/null || true +find "${LUSTRE_TRITON_CACHE}" -maxdepth 1 -name '.tmp_*' -mmin +30 -exec rm -rf {} + 2>/dev/null || true + +_seed_cache() { + local lustre="\$1" local_dir="\$2" name="\$3" + if [ -d "\$lustre" ] && [ "\$(ls -A "\$lustre" 2>/dev/null)" ]; then + rsync -a --exclude '.tmp_*' "\$lustre/" "\$local_dir/" 2>/dev/null \ + && echo "[CACHE SEED] \$name: seeded from shared fs" \ + || echo "[CACHE SEED] \$name: seed failed (non-fatal)" + else + echo "[CACHE SEED] \$name: no warm cache on shared fs yet" + fi +} + +_seed_cache "${LUSTRE_INDUCTOR_CACHE}" "${INDUCTOR_CACHE_DIR}" "Inductor" +_seed_cache "${LUSTRE_TRITON_CACHE}" "${TRITON_CACHE_DIR}" "Triton" +# uv cache: read-only rsync of single-process-prebuilt wheels into node-local +# /tmp so the uv sync below is a cache hit, not a ~28min build. +# (NOTE: no backticks/'\$(...)' in this heredoc body -- they would be command- +# substituted on the LOGIN node when SETUP_COMMAND is read.) +mkdir -p "${UV_CACHE_DIR}" +_seed_cache "${LUSTRE_UV_CACHE}" "${UV_CACHE_DIR}" "uv (prebuilt wheels)" +echo "[CACHE SEED] Done." + +# ===== Compile-cache WRITE-BACK sidecar ===== +# The seed above only READS shared fs -> /tmp. This sidecar periodically rsyncs +# /tmp -> shared fs (and on TERM/INT) so compiled kernels persist across runs; +# --ignore-existing makes concurrent per-node writes first-writer-wins. +_sync_cache_one() { + local src="\$1" dst="\$2" name="\$3" + mkdir -p "\$dst" + if [ -d "\$src" ] && [ "\$(ls -A "\$src" 2>/dev/null)" ]; then + rsync -a --ignore-existing --exclude '.tmp_*' --exclude 'tmp*' "\$src/" "\$dst/" 2>/dev/null \ + && echo "[CACHE SYNC] \$name: /tmp -> shared fs" \ + || echo "[CACHE SYNC] \$name: sync failed (non-fatal)" + fi +} +_sync_compile_caches_to_lustre() { + _sync_cache_one "${INDUCTOR_CACHE_DIR}" "${LUSTRE_INDUCTOR_CACHE}" "Inductor" + _sync_cache_one "${TRITON_CACHE_DIR}" "${LUSTRE_TRITON_CACHE}" "Triton" +} +_start_cache_sync_sidecar() { + local pidfile="/tmp/nemo_rl_compile_cache_sync.pid" + if [ -f "\$pidfile" ] && kill -0 "\$(cat "\$pidfile" 2>/dev/null)" 2>/dev/null; then + echo "[CACHE SYNC] sidecar already running (pid=\$(cat "\$pidfile" 2>/dev/null))" + return + fi + ( + set +e + trap '_sync_compile_caches_to_lustre; exit 0' TERM INT + echo "[CACHE SYNC] sidecar started, frequency=${CACHE_SYNC_FREQUENCY}s" + while true; do + sleep "${CACHE_SYNC_FREQUENCY}" + _sync_compile_caches_to_lustre + done + ) > /tmp/nemo_rl_compile_cache_sync.log 2>&1 & + echo "\$!" > "\$pidfile" + echo "[CACHE SYNC] sidecar pid=\$!" +} +if [ "${CACHE_SYNC_FREQUENCY}" -gt 0 ] 2>/dev/null; then + _start_cache_sync_sidecar +else + echo "[CACHE SYNC] disabled (CACHE_SYNC_FREQUENCY=${CACHE_SYNC_FREQUENCY})" +fi + +UV_HTTP_TIMEOUT=3600 \ + uv sync --frozen --extra mcore +SETUPEOF +export SETUP_COMMAND + +# Optional extra grpo overrides (only emitted when set, so empty == use yaml). +EXTRA_GRPO="" +[ -n "${MAX_NUM_STEPS}" ] && EXTRA_GRPO="grpo.max_num_steps=${MAX_NUM_STEPS}" + +# ===== Entrypoint switch: SC (single-controller) vs classic async GRPO ===== +# SC_MODE=1 (default): examples/run_grpo_single_controller.py + data_plane/async_rl overrides. +# SC_MODE=0: examples/nemo_gym/run_grpo_nemo_gym.py — the gym-dedicated classic +# entry; async comes from the yaml's native grpo.async_grpo block. +# (The generic examples/run_grpo.py is NOT wired for nemo-gym: it +# discards the gym actor instead of binding task_to_env["nemo_gym"], +# and its configure_generation_config eos injection trips the gym +# rollout stop-criteria assert.) +SC_MODE="${SC_MODE:-1}" +if [ "${SC_MODE}" = "1" ]; then + ENTRYPOINT="./examples/run_grpo_single_controller.py" + SC_OVERRIDES="++data_plane.enabled=true \ + ++data_plane.impl=transfer_queue \ + ++data_plane.backend=simple \ + ++data_plane.storage_capacity=1000000 \ + ++data_plane.num_storage_units=2 \ + ++data_plane.claim_meta_poll_interval_s=0.5 \ + ++data_plane.global_segment_size=549755813888 \ + ++data_plane.local_buffer_size=68719476736 \ + ++async_rl.max_weight_staleness_versions=${MAX_TRAJECTORY_AGE_STEPS} \ + ++async_rl.min_prompt_groups_per_batch=${MIN_PROMPT_GROUPS} \ + ++async_rl.max_inflight_prompts=${PPS} \ + ++async_rl.max_buffered_rollouts=$((PPS * (MAX_TRAJECTORY_AGE_STEPS + 1))) \ + ++async_rl.batch_selection_strategy=${BATCH_SELECTION_STRATEGY} \ + ++async_rl.over_sampling=${OVER_SAMPLING} \ + ++async_rl.force_in_order=${FORCE_IN_ORDER}" +else + ENTRYPOINT="./examples/nemo_gym/run_grpo_nemo_gym.py" + SC_OVERRIDES="" +fi + +# ===== Training command ===== +export COMMAND="NRL_VLLM_USE_V1=1 \ + NRL_WG_USE_RAY_REF=1 \ + WANDB_API_KEY=${WANDB_API_KEY} \ + HUGGINGFACE_TOKEN=${HUGGINGFACE_TOKEN} \ + GITHUB_TOKEN=${GITHUB_TOKEN} \ + GITLAB_TOKEN=${GITLAB_TOKEN} \ + HF_HOME=${HF_HOME} \ + HF_DATASETS_CACHE=${HF_DATASETS_CACHE} \ + UV_CACHE_DIR=${UV_CACHE_DIR} \ + VLLM_ATTENTION_BACKEND=FLASH_ATTN \ + VLLM_CACHE_ROOT=${LUSTRE_VLLM_CACHE} \ + DG_JIT_CACHE_DIR=${LUSTRE_VLLM_CACHE}/deep_gemm \ + VLLM_DEEP_GEMM_WARMUP=skip \ + NRL_FORCE_REBUILD_VENVS=${NRL_FORCE_REBUILD_VENVS:-false} \ + NG_HTTP_BYTES_DIR=${NG_HTTP_BYTES_DIR:-} \ + NRL_HTTP_BYTES_DIR=${NRL_HTTP_BYTES_DIR:-} \ + NRL_SKIP_TQ_RUNTIME_ENV_PATCH=${NRL_SKIP_TQ_RUNTIME_ENV_PATCH:-1} \ + NRL_IGNORE_VERSION_MISMATCH=1 \ + RAY_ENABLE_UV_RUN_RUNTIME_ENV=0 \ + UV_HTTP_TIMEOUT=3600 \ + UV_LOCK_TIMEOUT=900 \ + TORCH_CUDA_ARCH_LIST='9.0 10.0' \ + NEMO_GYM_SKIP_VENV_IF_PRESENT=1 \ + NEMO_GYM_VENV_DIR=${GYM_VENV_DIR} \ + uv run --frozen --extra mcore ${ENTRYPOINT} \ + --config=${CONFIG_FILE} \ + cluster.num_nodes=${NUM_NODES} \ + cluster.gpus_per_node=${GPUS_PER_NODE} \ + ++data.train.data_path=${TRAIN_DATA_PATH} \ + ++data.validation.data_path=${VAL_DATA_PATH} \ + grpo.num_prompts_per_step=${PPS} \ + grpo.num_generations_per_prompt=${GPP} \ + grpo.val_at_start=False \ + grpo.normalize_rewards=${NORMALIZE_REWARDS} \ + grpo.overlong_filtering=${OVERLONG_FILTERING} \ + grpo.val_period=${VAL_PERIOD} \ + grpo.seq_logprob_error_threshold=${SEQ_LOGPROB_ERROR_THRESHOLD} \ + ${EXTRA_GRPO} \ + ${SC_OVERRIDES} \ + ++policy.draft.enabled=false \ + ++policy.draft.model_name=null \ + ++policy.draft.loss_weight=0.1 \ + ++policy.draft.num_layers=null \ + ++policy.draft.aux_layer_indices=null \ + env.should_log_nemo_gym_responses=${LOG_GYM_RESPONSES} \ + policy.generation.colocated.enabled=${COLOCATED_ENABLED} \ + policy.model_name=${MODEL_PATH} \ + policy.max_total_sequence_length=${SEQLEN} \ + policy.dynamic_batching.enabled=False \ + policy.train_global_batch_size=${GBS} \ + policy.make_sequence_length_divisible_by=${MAKE_SEQ_DIVISIBLE_BY} \ + policy.offload_optimizer_for_logprob=true \ + policy.sequence_packing.enabled=${SEQUENCE_PACKING} \ + policy.megatron_cfg.tensor_model_parallel_size=${TP} \ + policy.megatron_cfg.expert_model_parallel_size=${EP} \ + policy.megatron_cfg.context_parallel_size=${CP} \ + policy.megatron_cfg.pipeline_model_parallel_size=${PP} \ + policy.megatron_cfg.sequence_parallel=True \ + policy.megatron_cfg.bias_activation_fusion=False \ + ++policy.megatron_cfg.use_fused_weighted_squared_relu=false \ + policy.megatron_cfg.distributed_data_parallel_config.overlap_grad_reduce=${OVERLAP_GRAD_REDUCE} \ + policy.megatron_cfg.moe_permute_fusion=${MOE_PERMUTE_FUSION} \ + policy.megatron_cfg.moe_enable_deepep=${MOE_ENABLE_DEEPEP} \ + policy.megatron_cfg.moe_token_dispatcher_type=${MOE_TOKEN_DISPATCHER_TYPE} \ + policy.megatron_cfg.moe_aux_loss_coeff=${MOE_AUX_LOSS_COEFF} \ + policy.megatron_cfg.moe_router_load_balancing_type=${MOE_ROUTER_LOAD_BALANCING_TYPE} \ + policy.megatron_cfg.moe_router_bias_update_rate=${MOE_ROUTER_BIAS_UPDATE_RATE} \ + policy.megatron_cfg.freeze_moe_router=${MOE_FREEZE_ROUTER} \ + policy.megatron_cfg.optimizer.lr=${LR} \ + policy.megatron_cfg.optimizer.min_lr=${LR} \ + policy.megatron_cfg.optimizer.weight_decay=0 \ + policy.megatron_cfg.empty_unused_memory_level=2 \ + policy.megatron_cfg.activation_checkpointing=True \ + policy.generation.temperature=${TEMPERATURE} \ + policy.generation.vllm_cfg.tensor_parallel_size=${VLLM_TP} \ + policy.generation.vllm_cfg.gpu_memory_utilization=${VLLM_GPU_UTIL} \ + policy.generation.vllm_cfg.skip_tokenizer_init=False \ + loss_fn.reference_policy_kl_penalty=${KL} \ + loss_fn.ratio_clip_min=${CLIP_MIN} \ + loss_fn.ratio_clip_max=${CLIP_MAX} \ + loss_fn.use_on_policy_kl_approximation=${USE_ON_POLICY_KL_APPROXIMATION} \ + loss_fn.use_importance_sampling_correction=${IMPORTANCE_SAMPLING_CORRECTION} \ + loss_fn.sequence_level_importance_ratios=${SEQ_LEVEL_IS} \ + loss_fn.token_level_loss=${TOKEN_LEVEL_LOSS} \ + loss_fn.force_on_policy_ratio=${FORCE_ON_POLICY_RATIO} \ + checkpointing.checkpoint_dir=${CHECKPOINT_DIR} \ + checkpointing.save_period=${SAVE_PERIOD} \ + checkpointing.keep_top_k=${KEEP_TOP_K} \ + ++checkpointing.metric_name=train:total_reward/mean \ + ++checkpointing.checkpoint_must_save_by=${MUST_SAVE_BY} \ + logger.wandb_enabled=True \ + logger.wandb.name=${WANDB_NAME} \ + logger.wandb.project=${WANDB_PROJ}" + +# Async non-colocated: generation cluster + clipping + agent turn/timeout knobs. +export COMMAND="${COMMAND} \ + policy.generation.colocated.resources.num_nodes=${NUM_GEN_NODES} \ + policy.generation.colocated.resources.gpus_per_node=${GPUS_PER_NODE} \ + grpo.advantage_clip_low=${ADVANTAGE_CLIP_LOW} \ + grpo.advantage_clip_high=${ADVANTAGE_CLIP_HIGH} \ + loss_fn.truncated_importance_sampling_ratio=${TIS_THRESHOLD} \ + env.nemo_gym.swe_agents_train.responses_api_agents.swe_agents.agent_max_turns=${AGENT_MAX_TURNS} \ + env.nemo_gym.swe_agents_train.responses_api_agents.swe_agents.swebench_agent_timeout=${AGENT_TIMEOUT} \ + env.nemo_gym.swe_agents_val.responses_api_agents.swe_agents.agent_max_turns=${AGENT_MAX_TURNS} \ + env.nemo_gym.swe_agents_val.responses_api_agents.swe_agents.swebench_agent_timeout=${AGENT_TIMEOUT}" + +# --------------------------------------------------------------------------- +# Mounts: the run tree at its own path (so ./examples/... resolves) and the +# gym source over the container's bundled copy. Export MOUNTS to replace, or +# EXTRA_MOUNTS to append (e.g. the filesystem holding MODEL_PATH/data). +# --------------------------------------------------------------------------- +GYM_CODE="${RUN_DIR}/3rdparty/Gym-workspace/Gym" +MOUNTS="${MOUNTS:-${RUN_DIR}:${RUN_DIR},${GYM_CODE}:/opt/nemo-rl/3rdparty/Gym-workspace/Gym}" +[ -n "${EXTRA_MOUNTS:-}" ] && MOUNTS="${MOUNTS},${EXTRA_MOUNTS}" + +mkdir -p "${CHECKPOINT_DIR}" "${BASE_LOG_DIR}" + +# ray.sub reads these from the environment. +export CONTAINER MOUNTS COMMAND SETUP_COMMAND GPUS_PER_NODE CPUS_PER_WORKER BASE_LOG_DIR +[ -n "${UV_CACHE_DIR_OVERRIDE:-}" ] && export UV_CACHE_DIR_OVERRIDE + +sbatch_args=( + --nodes="${NUM_NODES}" + --account="${ACCOUNT}" + --job-name="${EXP_NAME}" + --partition="${PARTITION}" + --time="${TIME}" + --gres=gpu:"${GPUS_PER_NODE}" + --exclusive + --dependency=singleton + --output="${BASE_LOG_DIR}/slurm-%j.out" +) +[ -n "${REAPER_COMMENT}" ] && sbatch_args+=(--comment="${REAPER_COMMENT}") +# shellcheck disable=SC2206 +[ -n "${SBATCH_EXTRA_ARGS:-}" ] && sbatch_args+=(${SBATCH_EXTRA_ARGS}) + +echo "==========================================" +echo "SWE async GRPO | Experiment: ${EXP_SUFFIX}" +echo "Entrypoint: ${ENTRYPOINT} (SC_MODE=${SC_MODE})" +echo "Run tree: ${RUN_DIR} @ ${RUN_COMMIT}" +echo "Account: ${ACCOUNT} / ${PARTITION}" +echo "Nodes: ${NUM_NODES} total (generation carves out ${NUM_GEN_NODES}) Time: ${TIME}" +echo "Container: ${CONTAINER}" +echo "Parallelism: TP=${TP}, EP=${EP}, CP=${CP}, PP=${PP}, vLLM_TP=${VLLM_TP}, pad=${MAKE_SEQ_DIVISIBLE_BY}" +echo "Training: PPS=${PPS}, GPP=${GPP}, GBS=${GBS}, LR=${LR}, seqlen=${SEQLEN}, max_steps=${MAX_NUM_STEPS:-}, min_prompt_groups=${MIN_PROMPT_GROUPS}" +echo "Streaming: over_sampling=${OVER_SAMPLING}, force_in_order=${FORCE_IN_ORDER}, age=${MAX_TRAJECTORY_AGE_STEPS}" +echo "Model: ${MODEL_PATH}" +echo "Checkpoint: ${CHECKPOINT_DIR}" +echo "WandB: ${WANDB_PROJ}/${WANDB_NAME}" +echo "==========================================" + +if [ "${DRY_RUN:-0}" = "1" ]; then + echo "[DRY_RUN] sbatch ${sbatch_args[*]} ${RUN_DIR}/ray.sub" + echo "[DRY_RUN] COMMAND (first 400 chars): ${COMMAND:0:400}..." + exit 0 +fi + +cd "${RUN_DIR}" +out=$(sbatch "${sbatch_args[@]}" "${RUN_DIR}/ray.sub") +echo "${out}" +job_id=$(echo "${out}" | grep -oE '[0-9]+' | head -1) +if [ -n "${job_id}" ]; then + echo "Job ID: ${job_id}" + echo "Monitor: squeue -j ${job_id}" + echo "Driver log: ${BASE_LOG_DIR}/${job_id}-logs/ray-driver.log" +fi diff --git a/nemo_rl/algorithms/advantage_estimator.py b/nemo_rl/algorithms/advantage_estimator.py index 491de0d9414..3167603967d 100644 --- a/nemo_rl/algorithms/advantage_estimator.py +++ b/nemo_rl/algorithms/advantage_estimator.py @@ -66,7 +66,7 @@ def __init__( self.use_leave_one_out_baseline = estimator_config.use_leave_one_out_baseline self.normalize_rewards = estimator_config.normalize_rewards - def compute_advantage(self, prompt_ids, rewards, mask, **kwargs): + def compute_advantage(self, prompt_ids, rewards, mask, valid_mask=None, **kwargs): """Compute GRPO advantages. Args: @@ -74,6 +74,11 @@ def compute_advantage(self, prompt_ids, rewards, mask, **kwargs): rewards: Tensor of shape [batch_size] containing reward for each sample. mask: Response token mask of shape [batch_size, seq_len], 1 for valid response tokens, 0 for padding. Used only for expanding advantages to token-level shape. + valid_mask: Optional tensor of shape [batch_size], 1.0 for samples whose + reward should participate in the per-prompt baseline/std. Token-capture + placeholder rows carry 0.0 (their sample_mask already excludes them + from the loss; excluding them here keeps siblings' baselines unbiased). + None keeps the legacy all-valid behavior. **kwargs: Additional arguments (unused). Returns: @@ -82,7 +87,7 @@ def compute_advantage(self, prompt_ids, rewards, mask, **kwargs): baseline, std = calculate_baseline_and_std_per_prompt( prompt_ids, rewards, - torch.ones_like(rewards), + torch.ones_like(rewards) if valid_mask is None else valid_mask.float(), leave_one_out_baseline=self.use_leave_one_out_baseline, ) advantages = (rewards - baseline).unsqueeze(-1) diff --git a/nemo_rl/algorithms/async_utils/replay_buffer.py b/nemo_rl/algorithms/async_utils/replay_buffer.py index 7d408d7b4fe..51828a9c445 100644 --- a/nemo_rl/algorithms/async_utils/replay_buffer.py +++ b/nemo_rl/algorithms/async_utils/replay_buffer.py @@ -13,7 +13,6 @@ # limitations under the License. import asyncio -import gc import statistics import threading as _threading import uuid @@ -22,17 +21,13 @@ from typing import Any, Iterable, Optional import ray -import torch from nemo_rl.algorithms.async_utils.interfaces import ReplayBufferProtocol from nemo_rl.data_plane import KVBatchMeta from nemo_rl.data_plane.schema import ROUTED_EXPERTS_FIELD -from nemo_rl.experience.interfaces import ( - NEMO_GYM_TASK_INDEX_KEY, - NEXT_NEMO_GYM_TASK_INDEX_KEY, - PromptGroupRecord, -) +from nemo_rl.experience.interfaces import PromptGroupRecord from nemo_rl.experience.payload import pack_payload, record_to_train_batch +from nemo_rl.experience.row_dump import maybe_dump_train_rows from nemo_rl.utils.r3_trace import trace_rollout_payload @@ -346,44 +341,6 @@ def state_dict(self) -> dict[str, Any]: "max_size": self.max_size, } - def save_to_path(self, path: str) -> int: - """Serialize inside the actor without materializing the buffer on the driver.""" - state = self.state_dict() - torch.save(state, path) - num_trajectories = len(state["trajectories"]) - del state - gc.collect() - return num_trajectories - - def load_from_path( - self, - path: str, - num_prompts_per_step: int | None = None, - current_training_step: int | None = None, - max_age_steps: int | None = None, - ) -> dict[str, int]: - """Restore inside the actor and return only compact coordination metadata.""" - state = torch.load(path, weights_only=False) - saved_task_indices = [ - int(trajectory[NEMO_GYM_TASK_INDEX_KEY]) - for trajectory in state.get("trajectories", []) - if trajectory.get(NEMO_GYM_TASK_INDEX_KEY) is not None - ] - next_task_index = max(saved_task_indices, default=-1) + 1 - num_trajectories = len(state["trajectories"]) - self.load_state_dict( - state, - num_prompts_per_step=num_prompts_per_step, - current_training_step=current_training_step, - max_age_steps=max_age_steps, - ) - del state - gc.collect() - return { - "num_trajectories": num_trajectories, - NEXT_NEMO_GYM_TASK_INDEX_KEY: next_task_index, - } - def load_state_dict( self, state: dict[str, Any], @@ -694,11 +651,16 @@ def __init__( partition_id: str, *, pad_value_dict: Mapping[str, int], + staging_partition_id: Optional[str] = None, require_routed_experts: bool = False, ): self._dp_client = dp_client self._partition_id = partition_id self._pad_value_dict = dict(pad_value_dict) + # Token-capture mode only (docs/design-docs/tq-gym-gate-authoritative.md): + # the staging partition whose per-call delta rows `remove` must clear + # alongside the canonical rows. None on the legacy path. + self._staging_partition_id = staging_partition_id self._require_routed_experts = require_routed_experts self.meta_list: list[Optional[KVBatchMeta]] = [] self.start_weight_list: list[int] = [] @@ -707,6 +669,9 @@ def __init__( self.target_step_list: list[Optional[int]] = [] self.ready_list: list[bool] = [] self._group_ids: list[str] = [] + # Parallel to the lists above; populated only in token-capture mode. + self._rollout_ids_list: list[Optional[list[str]]] = [] + self._staging_keys_list: list[Optional[list[str]]] = [] def reserve( self, @@ -714,6 +679,7 @@ def reserve( weight_version: int, target_step: Optional[int] = None, group_id: Optional[str] = None, + rollout_ids: Optional[list[str]] = None, ) -> str: """Append an unready slot tagged with weight_version. @@ -721,6 +687,9 @@ def reserve( weight_version: Weight version stamped on the slot. target_step: Training step this slot targets; only consulted by StalenessSampler.force_in_order. group_id: Per-group sample_id prefix; defaults to a fresh uuid4. + rollout_ids: Token-capture mode: the gate-registered rollout ids + this slot dispatched, recorded so cleanup can name what it + owns even before a receipt exists. Returns: group_id used by the matching commit. @@ -733,6 +702,10 @@ def reserve( self.target_step_list.append(target_step) self.ready_list.append(False) self._group_ids.append(group_id) + self._rollout_ids_list.append( + list(rollout_ids) if rollout_ids is not None else None + ) + self._staging_keys_list.append(None) return group_id async def commit( @@ -758,12 +731,12 @@ async def commit( ValueError: group_id has no live slot (removed or never reserved). RuntimeError: router replay is enabled but the payload has no routes. """ - # Precondition: reserve() must have registered this group_id. Raise - # before any side effects so a stray commit doesn't leak orphan DP rows. + # Check the slot is still live BEFORE writing: a slot evicted while + # its rollout was in flight must not orphan rows into the partition. if group_id not in self._group_ids: raise ValueError( - f"commit called with unknown group_id={group_id!r}; " - f"reserve() must precede commit() (or the slot was already removed)" + f"TQReplayBuffer.commit: group {group_id} has no live slot " + "(evicted or never reserved); nothing written" ) train_batch = record_to_train_batch(record, pad_value_dict=self._pad_value_dict) sample_ids, fields, tags = pack_payload( @@ -776,6 +749,13 @@ async def commit( "not produce that field. Check vLLM routed-expert capture and " "the async message-log flattening path." ) + maybe_dump_train_rows( + source="legacy_commit", + group_id=group_id, + sample_ids=list(sample_ids), + train_batch=train_batch, + weight_version=start_weight_version, + ) trace_rollout_payload(keys=sample_ids, data=train_batch) try: await self._call_dp( @@ -797,7 +777,15 @@ async def commit( tags=[dict(t) for t in tags], ) - idx = self._group_ids.index(group_id) + try: + idx = self._group_ids.index(group_id) + except ValueError: + # Evicted during the awaited write: un-write the rows so the + # partition holds nothing the buffer no longer tracks. + raise ValueError( + f"TQReplayBuffer.commit: group {group_id} was evicted " + "during the write; rows cleared" + ) from None self.meta_list[idx] = meta self.end_weight_list[idx] = end_weight_version self.ready_list[idx] = True @@ -839,9 +827,88 @@ async def remove_group(self, group_id: str, *, remove_in_dp: bool = False) -> in raise ValueError(f"unknown group_id={group_id!r}") from error return await self.remove([idx], remove_in_dp=remove_in_dp) + async def commit_finalized( + self, + group_id: str, + meta: KVBatchMeta, + group_min_wv: int, + group_max_wv: int, + *, + staging_keys: Optional[list[str]] = None, + ) -> KVBatchMeta: + """Mark a slot ready from finalizer output (token-capture mode). + + Unlike :meth:`commit`, the canonical rows are already in TQ — the + finalizer tensorized and put them — so this only fills the slot. + The slot's effective version is the group's OLDEST call version + (``group_min_wv``): staleness accounting stays conservative when a + rollout straddles a refit. + + Args: + group_id: group_id returned by the matching reserve call. + meta: KVBatchMeta the finalizer built over its published rows. + group_min_wv: Oldest weight version any call in the group used. + group_max_wv: Newest weight version any call in the group used. + staging_keys: The group's staged delta keys, recorded so + :meth:`remove` can clear the staging partition too. + + Raises: + ValueError: group_id has no live slot (removed or never reserved). + """ + try: + idx = self._group_ids.index(group_id) + except ValueError: + raise ValueError( + f"TQReplayBuffer.commit_finalized: group {group_id} has no " + "live slot (evicted or never reserved)" + ) from None + self.meta_list[idx] = meta + self.start_weight_list[idx] = group_min_wv + self.end_weight_list[idx] = group_max_wv + self.ready_list[idx] = True + self._staging_keys_list[idx] = ( + list(staging_keys) if staging_keys is not None else None + ) + return meta + + def abort(self, group_id: str) -> bool: + """Drop an unready slot whose dispatch failed or was cancelled. + + Token-capture mode; called from the failed dispatch path. + No DataPlane rows are cleared: an unready slot published no canonical + rows, and its staged deltas (keys unknown before a receipt) are swept + by the staging TTL backstop. + + Returns: + True when a slot was dropped; False when the group_id has no + live slot (already committed+consumed or never reserved). + """ + try: + idx = self._group_ids.index(group_id) + except ValueError: + return False + if self.ready_list[idx]: + return False + self._delete_slot(idx) + return True + + def _delete_slot(self, idx: int) -> None: + del self.meta_list[idx] + del self.start_weight_list[idx] + del self.end_weight_list[idx] + del self.target_step_list[idx] + del self.ready_list[idx] + del self._group_ids[idx] + del self._rollout_ids_list[idx] + del self._staging_keys_list[idx] + async def remove(self, idxs: list[int], remove_in_dp: bool) -> int: """Drop entries at the given indices and optionally clear them from DataPlane. + In token-capture mode (``staging_partition_id`` set), clearing a + group also clears its recorded staged delta rows, so eviction leaves + neither canonical nor staging bytes behind. + Args: idxs: Entry indices to drop. Must be within [0, size). remove_in_dp: If True, also clear the dropped rows from DataPlane. @@ -860,16 +927,15 @@ async def remove(self, idxs: list[int], remove_in_dp: bool) -> int: ) dropped_sample_ids: list[str] = [] + dropped_staging_keys: list[str] = [] for i in drop_idxs: meta = self.meta_list[i] if meta is not None: dropped_sample_ids.extend(meta.sample_ids) - del self.meta_list[i] - del self.start_weight_list[i] - del self.end_weight_list[i] - del self.target_step_list[i] - del self.ready_list[i] - del self._group_ids[i] + staging_keys = self._staging_keys_list[i] + if staging_keys: + dropped_staging_keys.extend(staging_keys) + self._delete_slot(i) if remove_in_dp: await self._call_dp( @@ -877,6 +943,12 @@ async def remove(self, idxs: list[int], remove_in_dp: bool) -> int: sample_ids=dropped_sample_ids, partition_id=self._partition_id, ) + if dropped_staging_keys and self._staging_partition_id is not None: + await self._call_dp( + "clear_samples", + sample_ids=dropped_staging_keys, + partition_id=self._staging_partition_id, + ) return len(drop_idxs) diff --git a/nemo_rl/algorithms/single_controller.py b/nemo_rl/algorithms/single_controller.py index 3a6a60e8f08..433b3408bb1 100644 --- a/nemo_rl/algorithms/single_controller.py +++ b/nemo_rl/algorithms/single_controller.py @@ -582,6 +582,8 @@ async def _train_pump(self) -> None: self._logger.log_metrics( timing_metrics, step=self._train_steps, prefix="timing/train" ) + if self._master_config.token_capture.enabled: + await self._log_gate_metrics() self._timer.reset() # min sample version refers to the version each consumed sample was @@ -694,9 +696,38 @@ async def _sync_weights( print(f" _sync_weights: sync done in {elapsed:.3f}s", flush=True) self._rollout_manager.set_weight_version(self._trainer_version) + if self._master_config.token_capture.enabled: + # Rotate the version vLLM workers stamp on captured model calls + # (per-call tagging; group staleness = min over the group's calls). + await asyncio.to_thread( + self._gen.set_rollout_weight_version, self._trainer_version + ) self._rollout_permitted.set() return aborted_stale_inflight_groups + async def _log_gate_metrics(self) -> None: + """Log the capture gate's cumulative § 8 counters + token_in_rate. + + Counters are cumulative over the run; ``gate/token_in_rate`` is the + cumulative marker-hit rate over all admitted model calls. Fetch + failures are logged and swallowed — metrics must never kill a step. + """ + try: + counters = await self._rollout_manager.gate_metrics() + except (RuntimeError, OSError) as error: + print(f"gate metrics fetch failed: {error}", flush=True) + return + if not counters: + return + calls = counters["token_in"] + sum( + v for k, v in counters.items() if k.startswith("fallback_") + ) + gate_metrics: dict[str, float] = {k: float(v) for k, v in counters.items()} + if calls: + gate_metrics["token_in_rate"] = counters["token_in"] / calls + self._logger.log_metrics(gate_metrics, step=self._train_steps, prefix="gate") + print(f"gate_metrics={gate_metrics}", flush=True) + async def _advantage_stage(self, meta: KVBatchMeta) -> KVBatchMeta: """Fetch advantage inputs, compute advantages, and write them back. @@ -752,6 +783,9 @@ async def _advantage_stage(self, meta: KVBatchMeta) -> KVBatchMeta: rewards=rewards, mask=mask, repeated_batch=repeated_batch, + # Real validity (token-capture placeholders carry sample_mask 0) + # instead of the hardwired all-ones — § 9.1, advantage_estimator. + valid_mask=sample_mask, **kwargs, ) response_advantages = torch.masked_select(advantages, mask.bool()) diff --git a/nemo_rl/algorithms/single_controller_utils/config.py b/nemo_rl/algorithms/single_controller_utils/config.py index 031b478f0ee..44cb67223f4 100644 --- a/nemo_rl/algorithms/single_controller_utils/config.py +++ b/nemo_rl/algorithms/single_controller_utils/config.py @@ -15,7 +15,7 @@ from __future__ import annotations from dataclasses import dataclass, field -from typing import Any, Optional +from typing import Any, Literal, Optional from pydantic import BaseModel, Field @@ -52,6 +52,55 @@ class AsyncRLConfig(BaseModel, extra="allow"): diagnostics: bool = False +class TokenCaptureConfig(BaseModel, extra="allow"): + """Gate-authoritative token capture (token-in/token-out via NeMo-Gym). + + Dormant by default: with ``enabled=False`` every legacy codepath behaves + exactly as before — no staging partition is registered, no gate is + installed, and rollouts ride the token-echo path. See + docs/design-docs/tq-gym-gate-authoritative.md. + """ + + enabled: bool = False + # TQ partition holding per-call staged token deltas (cleared by the + # finalizer; distinct from the canonical rollout partition). + staging_partition: str = "rollout_staging" + # A failed worker-side stage poisons the rollout; "continue" serves the + # completion and lets the finalizer emit a placeholder row, "abort" fails + # the whole rollout at the gate. + on_capture_failure: Literal["continue", "abort"] = "continue" + # "allow" trains groups whose calls span a refit (staleness accounted via + # group_min_wv); "reject" placeholders them. Strict modes beyond the MVP + # matrix raise NotImplementedError at setup. + mixed_weight_version_policy: Literal["allow", "reject"] = "allow" + # Drop the whole group when fewer than this fraction of its rollouts + # produced valid rows (None keeps every group). + min_valid_fraction_per_group: Optional[float] = None + # Gate-side cleanup backstops. + registration_ttl_s: float = 3600.0 + staging_ttl_s: float = 3600.0 + # Gym LineageIndex capacity (finding M: it holds each in-flight rollout's + # full cumulative token sequence, and eviction of a live rollout silently + # degrades token-in to fallbacks). None = derived at setup from the + # training config: rollouts ≈ 2 × max in-flight; tokens ≈ rollouts × max + # sequence length. Set explicitly for agentic workloads whose per-rollout + # call trees hold more than one context of tokens. + lineage_max_rollouts: Optional[int] = None + lineage_max_tokens: Optional[int] = None + # Bearer token for the gate's /ng-control/* routes (finding S). None = + # minted per run at setup; set explicitly only for multi-controller + # setups that must share one gate. + control_auth_token: Optional[str] = None + # Hard deadline per control-plane call (S5 finding: gate death must + # surface as a failed dispatch, not a silent retry stall). + control_timeout_s: float = 60.0 + # Directory for the Gym base capture layer the gate rides on (#2124-c1: + # the capture middleware only engages with a capture dir configured; the + # dir stays essentially empty on the gate path). None = derived at setup + # under the run's log dir. + capture_dir: Optional[str] = None + + class MasterConfig(BaseModel, extra="allow"): policy: PolicyConfig loss_fn: ClippedPGLossConfig @@ -63,6 +112,7 @@ class MasterConfig(BaseModel, extra="allow"): checkpointing: CheckpointingConfig data_plane: DataPlaneConfig async_rl: AsyncRLConfig + token_capture: TokenCaptureConfig = Field(default_factory=TokenCaptureConfig) def validate_sampler_buffer_capacity( diff --git a/nemo_rl/algorithms/single_controller_utils/setup.py b/nemo_rl/algorithms/single_controller_utils/setup.py index 96eff61231d..10e383c199b 100644 --- a/nemo_rl/algorithms/single_controller_utils/setup.py +++ b/nemo_rl/algorithms/single_controller_utils/setup.py @@ -21,6 +21,7 @@ from __future__ import annotations +import os import time from concurrent.futures import ThreadPoolExecutor from dataclasses import dataclass @@ -300,6 +301,12 @@ def _spinup_gym(master_config: MasterConfig, base_urls: list[str]) -> tuple[Any, enable_router_replay=enable_router_replay, routed_experts_dtype=routed_experts_dtype, use_fastokens=bool(policy_config["tokenizer"].get("use_fastokens")), + # Gate config rides into Gym's policy model server (§ 9.1). + token_capture=( + master_config.token_capture.model_dump() + if master_config.token_capture.enabled + else None + ), ) return actor, time.perf_counter() - t0 @@ -400,6 +407,67 @@ def setup_single_controller( "data.use_multiple_dataloader=True yet." ) + # Token capture: validate the MVP matrix loudly at setup (§ 6, § 10) and + # give capture-enabled vLLM workers a venv that carries nemo_gym (the + # worker hosts Gym's capture core + adapter in-process). + token_capture_cfg = master_config.token_capture + if token_capture_cfg.enabled: + if not _should_use_nemo_gym(master_config): + raise ValueError( + "token_capture.enabled requires the NeMo-Gym rollout path " + "(env.should_use_nemo_gym=true) — the gate lives in Gym's " + "policy model server" + ) + if generation_config["backend"] != "vllm": + raise NotImplementedError( + "token_capture.enabled supports the vllm backend only; got " + f"{generation_config['backend']!r}" + ) + if not generation_config["vllm_cfg"]["async_engine"]: + raise ValueError( + "token_capture.enabled requires " + "policy.generation.vllm_cfg.async_engine=true (the capture " + "host is the worker's in-process HTTP server)" + ) + from nemo_rl.distributed.ray_actor_environment_registry import ( + ACTOR_ENVIRONMENT_REGISTRY, + ) + from nemo_rl.distributed.virtual_cluster import PY_EXECUTABLES + + ACTOR_ENVIRONMENT_REGISTRY[ + "nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker" + ] = PY_EXECUTABLES.VLLM_GYM + + # Fill the derived gate-hosting fields (see TokenCaptureConfig): + # a per-run control-plane bearer token, the base capture dir the + # gate rides on, and LineageIndex capacity sized from the training + # config (finding M: eviction of a live rollout must not happen + # under normal operation). + if token_capture_cfg.control_auth_token is None: + # Deferred import: only needed on the capture path. + import secrets + + token_capture_cfg.control_auth_token = secrets.token_hex(32) + if token_capture_cfg.capture_dir is None: + token_capture_cfg.capture_dir = os.path.abspath( + os.path.join( + master_config.logger.get("log_dir") or "logs", + "gym_token_capture", + ) + ) + if token_capture_cfg.lineage_max_rollouts is None: + group_size = grpo_config.num_generations_per_prompt + in_flight = ( + master_config.async_rl.max_buffered_rollouts + + master_config.async_rl.max_inflight_prompts + ) * group_size + token_capture_cfg.lineage_max_rollouts = 2 * in_flight + if token_capture_cfg.lineage_max_tokens is None: + token_capture_cfg.lineage_max_tokens = ( + token_capture_cfg.lineage_max_rollouts + * int(master_config.policy["max_total_sequence_length"]) + ) + set_seed(grpo_config.seed) # ========================== @@ -560,6 +628,54 @@ def _build_generation_then_trainer( # Connect-only DP client; TQPolicy already bootstrapped the controller. dp_client = build_data_plane_client(dp_config, bootstrap=False) + # Token-capture mode: pre-register both rollout partitions from this + # single driver thread before any producer is live. TQ's controller + # registers unseen field names lazily inside update_production_status + # without a lock, so the first concurrent puts into an unregistered + # partition can race kv_retrieve_meta and kill the controller thread + # (see TQDataPlaneClient.register_partition). + token_capture_cfg = master_config.token_capture + if token_capture_cfg.enabled: + from nemo_rl.data_plane.schema import DP_TRAIN_FIELDS + from nemo_rl.data_plane.tq_token_sink import STAGING_FIELDS + + group_size = grpo_config.num_generations_per_prompt + num_rollout_samples = master_config.async_rl.max_buffered_rollouts * group_size + dp_client.register_partition( + partition_id=partition_id, + fields=list(DP_TRAIN_FIELDS), + num_samples=num_rollout_samples, + consumer_tasks=["prev_lp", "ref_lp", "train"], + grpo_group_size=group_size, + ) + dp_client.register_partition( + partition_id=token_capture_cfg.staging_partition, + fields=list(STAGING_FIELDS), + num_samples=num_rollout_samples, + consumer_tasks=["finalize"], + ) + # Host Gym's capture core in every vLLM DP leader (in-worker DP + # client + TQTokenSink + the single install_capture call), and give + # workers the initial weight version to stamp on captured calls. + try: + generation.setup_token_capture( + dp_config, token_capture_cfg.staging_partition + ) + except Exception as error: + if "No module named 'nemo_gym'" in str(error): + # Worker venvs are cached by actor class name + # (nemo_rl/utils/venvs.py), so a venv prebuilt before token + # capture predates the nemo_gym extra and is reused as-is. + raise RuntimeError( + "token_capture.enabled requires nemo_gym inside the vLLM " + "worker venv, but the cached worker venv predates it. " + "Rebuild worker venvs (NRL_FORCE_REBUILD_VENVS=true) or " + "delete $NEMO_RL_VENV_DIR/nemo_rl.models.generation.vllm." + "vllm_worker_async.VllmAsyncGenerationWorker and rerun." + ) from error + raise + generation.set_rollout_weight_version(0) + t0 = time.perf_counter() weight_synchronizer = create_weight_synchronizer( policy=trainer, @@ -587,7 +703,22 @@ def _build_generation_then_trainer( partition_id=partition_id, pad_value_dict={"token_ids": pad_id, "input_ids": pad_id}, require_routed_experts=router_replay_enabled(policy_config), + staging_partition_id=( + token_capture_cfg.staging_partition if token_capture_cfg.enabled else None + ), ) + finalizer = None + if token_capture_cfg.enabled: + from nemo_rl.experience.blackbox_finalizer import BlackboxFinalizer + + finalizer = BlackboxFinalizer( + dp_client, + partition_id=partition_id, + staging_partition=token_capture_cfg.staging_partition, + pad_token_id=pad_id, + mixed_weight_version_policy=token_capture_cfg.mixed_weight_version_policy, + min_valid_fraction_per_group=token_capture_cfg.min_valid_fraction_per_group, + ) rollout_manager = RolloutManager( tokenizer=tokenizer, task_to_env=env_handles, @@ -599,6 +730,7 @@ def _build_generation_then_trainer( use_nemo_gym=use_nemo_gym, mask_env_flagged_samples=should_mask_flagged_samples(master_config.env), tq_buffer=tq_buffer, + finalizer=finalizer, ) # Print setup timing metrics diff --git a/nemo_rl/data_plane/tq_token_sink.py b/nemo_rl/data_plane/tq_token_sink.py new file mode 100644 index 00000000000..0f83c1e9fc9 --- /dev/null +++ b/nemo_rl/data_plane/tq_token_sink.py @@ -0,0 +1,188 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""TransferQueue implementations of NeMo-Gym's token staging protocols. + +``TQTokenSink``/``TQTokenSource`` are NeMo-RL's providers for the +gate-authoritative capture design (docs/design-docs/tq-gym-gate-authoritative.md): +the sink is the worker-side write of one model call's token delta to the +``rollout_staging`` partition — the design's only heavy token hop — and the +source is the finalizer's read-back of those rows by staging key. This module +is the only hot-path file that knows tokens live in TQ; Gym sees opaque +staging keys. + +Each staged row carries three jagged columns (``token_ids_delta``, +``token_mask_delta``, ``generation_logprobs_delta``) plus scalar ``prev_len`` +and ``weight_version`` columns so a fetched row round-trips to a complete +``StagedCallSnapshot`` (parent pointers are lineage state and are rejoined +from the receipt manifest by the finalizer). Masks/logprobs are float32 on +the wire, matching ``compute_staging_digest``'s float32-bit-pattern scheme, +so the finalizer's digest recomputation over fetched values is byte-exact. +""" + +from __future__ import annotations + +from typing import Any, Optional + +import ray +import torch +from tensordict import TensorDict + +from nemo_gym.token_id_capture.staging.records import ( + StagedCallRecord, + StagedCallSnapshot, + StageResult, +) + +STAGING_FIELDS = [ + "token_ids_delta", + "token_mask_delta", + "generation_logprobs_delta", + "prev_len", + "weight_version", +] + + +def _call_dp(dp_client: Any, method_name: str, **kwargs: Any) -> Any: + """Call a DataPlaneClient method on a local client or a Ray actor handle.""" + method = getattr(dp_client, method_name) + remote = getattr(method, "remote", None) + if remote is not None: + return ray.get(remote(**kwargs)) + return method(**kwargs) + + +class TQTokenSink: + """Gym ``StagingSink`` over ``DataPlaneClient.put_samples``. + + ``stage`` is synchronous and returns only after TQ acknowledged the + write, so the capture layer's fail-closed ordering (bytes durable before + the model call is acked) holds by construction. Failures are reported in + the ``StageResult`` — the caller decides whether the rollout poisons or + aborts (``token_capture.on_capture_failure``). + """ + + def __init__(self, dp_client: Any, *, staging_partition: str) -> None: + self._dp_client = dp_client + self._staging_partition = staging_partition + + def stage(self, record: StagedCallRecord) -> StageResult: + key = record.staging_key + try: + fields = TensorDict( + { + "token_ids_delta": torch.tensor( + [record.token_ids_delta], dtype=torch.int64 + ), + "token_mask_delta": torch.tensor( + [record.token_mask_delta], dtype=torch.float32 + ), + "generation_logprobs_delta": torch.tensor( + [record.generation_logprobs_delta], dtype=torch.float32 + ), + "prev_len": torch.tensor([record.prev_len], dtype=torch.int64), + "weight_version": torch.tensor( + [record.weight_version], dtype=torch.int64 + ), + }, + batch_size=[1], + ) + tags = [ + { + "rollout_id": record.rollout_id, + "call_id": record.call_id, + "parent_call_id": record.parent_call_id, + "prev_len": record.prev_len, + "new_len": record.new_len, + "weight_version": record.weight_version, + "digest": record.digest, + "schema_version": record.schema_version, + } + ] + _call_dp( + self._dp_client, + "put_samples", + sample_ids=[key], + partition_id=self._staging_partition, + fields=fields, + tags=tags, + ) + except Exception as error: # noqa: BLE001 — any failure must poison, not crash serving + return StageResult( + ok=False, staging_key=key, error=f"{type(error).__name__}: {error}" + ) + return StageResult(ok=True, staging_key=key) + + def clear(self, staging_keys: list[str]) -> None: + """Drop staged rows (finalizer / eviction cleanup).""" + if not staging_keys: + return + _call_dp( + self._dp_client, + "clear_samples", + sample_ids=list(staging_keys), + partition_id=self._staging_partition, + ) + + +class TQTokenSource: + """Gym ``StagingSource`` over ``DataPlaneClient.get_samples``. + + Rows are fetched one key at a time (deltas are jagged across calls) in + the order requested. A missing or unreadable row raises ``KeyError`` per + the protocol — the finalizer maps that to a placeholder, never a silent + skip. + """ + + def __init__(self, dp_client: Any, *, staging_partition: str) -> None: + self._dp_client = dp_client + self._staging_partition = staging_partition + + def fetch(self, staging_keys: list[str]) -> list[StagedCallSnapshot]: + snapshots: list[StagedCallSnapshot] = [] + for key in staging_keys: + _, _, call_id = key.rpartition("/") + try: + row = _call_dp( + self._dp_client, + "get_samples", + sample_ids=[key], + partition_id=self._staging_partition, + select_fields=STAGING_FIELDS, + ) + snapshots.append(_row_to_snapshot(call_id, row)) + except KeyError: + raise + except Exception as error: # noqa: BLE001 — protocol maps any miss to KeyError + raise KeyError( + f"staged row {key!r} could not be fetched: {error}" + ) from error + return snapshots + + +def _row_to_snapshot(call_id: str, row: Any) -> StagedCallSnapshot: + def _leaf(name: str) -> torch.Tensor: + value = row[name] + tensor = value[0] if value.dim() > 1 or value.numel() > 1 else value + return tensor.reshape(-1) + + prev_len = int(_leaf("prev_len")[0].item()) + weight_version: Optional[int] = int(_leaf("weight_version")[0].item()) + return StagedCallSnapshot( + call_id=call_id, + prev_len=prev_len, + token_ids_delta=[int(t) for t in _leaf("token_ids_delta").tolist()], + token_mask_delta=[float(m) for m in _leaf("token_mask_delta").tolist()], + logprobs_delta=[float(p) for p in _leaf("generation_logprobs_delta").tolist()], + weight_version=weight_version, + ) diff --git a/nemo_rl/distributed/virtual_cluster.py b/nemo_rl/distributed/virtual_cluster.py index 96250b71802..15b32733a31 100644 --- a/nemo_rl/distributed/virtual_cluster.py +++ b/nemo_rl/distributed/virtual_cluster.py @@ -73,6 +73,10 @@ class PY_EXECUTABLES: # Use NeMo-Gym dependencies NEMO_GYM = f"uv run --locked --extra nemo_gym --directory {git_root}" + # vLLM worker hosting Gym's token capture (token_capture.enabled): the + # worker imports nemo_gym's dependency-free capture core + vLLM adapter. + VLLM_GYM = f"uv run --locked --extra vllm --extra nemo_gym --directory {git_root}" + # Use NeMo-RL direct dependencies and SGLang. SGLANG = f"uv run --locked --extra sglang --directory {git_root}" diff --git a/nemo_rl/environments/nemo_gym.py b/nemo_rl/environments/nemo_gym.py index 3eb9adbf700..57638ef9af7 100644 --- a/nemo_rl/environments/nemo_gym.py +++ b/nemo_rl/environments/nemo_gym.py @@ -11,6 +11,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +import asyncio import math import os import subprocess @@ -21,6 +22,7 @@ from pathlib import Path from typing import Any, Dict, List, NotRequired, Optional, TypedDict +import aiohttp import ray import torch from PIL import Image @@ -126,6 +128,19 @@ class NemoGymConfig(TypedDict): tokenizer_config: NotRequired[ Optional[TokenizerConfig] ] # For processor reconstruction inside the actor + # Gate-authoritative token capture (token_capture.enabled): the dumped + # TokenCaptureConfig. Turns on the gate in Gym's policy model server, + # switches run_rollouts to receipt mode, and adds the register/seal/fail + # control-plane helpers. None/absent = legacy token-echo path. + token_capture: NotRequired[Dict[str, Any] | None] + + +# Gym control-plane server name (the model server hosting the gate) and the +# opaque run-body key rollout ids ride on (Gym's ROLLOUT_ID_KEY_NAME): the +# agent derives the id from the run body and stamps /ng-rollout/ on every +# model call, so the TQ sample id IS the capture key end to end. +_POLICY_SERVER_NAME = "policy_model" +_NG_ROLLOUT_ID_BODY_KEY = "_ng_rollout_id" def _detect_invalid_tool_call_and_malformed_thinking( @@ -452,6 +467,79 @@ def _spinup(self) -> None: "port": self.head_server_port, } + self.rollout_max_attempts_to_avoid_lp_nan = initial_global_config_dict.pop( + "rollout_max_attempts_to_avoid_lp_nan", 1 + ) + + assert self.rollout_max_attempts_to_avoid_lp_nan >= 1, ( + "`rollout_max_attempts_to_avoid_lp_nan` must be at least 1" + ) + + # Gate-authoritative token capture: turn on the gate in the policy + # model server (via the policy_model global-config override block the + # env yamls already use) and disable the legacy token echo. Receipt + # mode is incompatible with re-dispatching a batch under the same + # rollout ids, so the NaN retry must be exactly 1 (create-only + # registration would fail the retry loudly anyway; fail at setup). + token_capture = self.cfg.get("token_capture") or None + self._token_capture_enabled = bool( + token_capture and token_capture.get("enabled") + ) + self._server_client = None + self._control_headers: Dict[str, str] = {} + self._control_timeout_s = 60.0 + if self._token_capture_enabled: + if self.rollout_max_attempts_to_avoid_lp_nan != 1: + raise ValueError( + "token_capture.enabled requires " + "rollout_max_attempts_to_avoid_lp_nan == 1: a NaN retry " + "would re-register create-only rollout ids at the gate" + ) + policy_overrides = ( + initial_global_config_dict.setdefault("policy_model", {}) + .setdefault("responses_api_models", {}) + .setdefault("vllm_model", {}) + ) + policy_overrides["return_token_id_information"] = False + policy_overrides["token_capture_gate"] = { + "enabled": True, + "registration_ttl_s": token_capture["registration_ttl_s"], + # Finding M: the LineageIndex holds each in-flight rollout's + # cumulative tokens; capacity is sized from the training + # config at setup, never left at the eval-sized defaults. + "lineage_max_rollouts": token_capture["lineage_max_rollouts"], + "lineage_max_tokens": token_capture["lineage_max_tokens"], + # Finding S: control routes are bearer-authed, + # default-required; the token is minted per run at setup. + "control_auth_token": token_capture["control_auth_token"], + } + # #2124-c1 workaround: the capture middleware mints model_call_id + # (and sets the capture context the gate keys on) only when the + # base token capture is active, which requires a capture dir. + # The base's own capture_tokens no-ops on the gate path (the + # worker strips token fields before responding), so the dir + # stays essentially empty. + initial_global_config_dict["token_id_capture_enabled"] = True + initial_global_config_dict["token_id_capture_dir"] = token_capture[ + "capture_dir" + ] + # Agents apply the /ng-rollout/ correlation prefix for token + # capture only when they opt in (the global enabled switch alone + # gates the infrastructure, not the prefix). In an RL training + # run every agent must correlate, and the SC cannot enumerate + # agent servers configured via config_paths — so flip the + # run-wide opt-in the Gym branch adds for exactly this case. + initial_global_config_dict["token_id_capture_all_agents"] = True + self._control_headers = { + "Authorization": f"Bearer {token_capture['control_auth_token']}" + } + # S5 chaos finding: Gym's shared request() retries connection + # errors indefinitely; a dead gate must surface as a failed + # dispatch (placeholders + TTL), not a silent stall. + self._control_timeout_s = float( + token_capture.get("control_timeout_s") or 60.0 + ) + self.rh = RunHelper() self.rh.start( global_config_dict_parser_config=GlobalConfigDictParserConfig( @@ -469,6 +557,65 @@ def _spinup(self) -> None: ) self.rch = RolloutCollectionHelper() + # ── gate control plane (token-capture mode) ───────────────────────────── + + def _control_client(self): + """Gym ServerClient resolving servers by name from the head server.""" + if self._server_client is None: + from nemo_gym.server_utils import ServerClient + + self._server_client = ServerClient.load_from_global_config( + self.head_server_config + ) + return self._server_client + + async def _control(self, method: str, path: str, **kwargs: Any) -> dict: + headers = {**kwargs.pop("headers", {}), **self._control_headers} + try: + response = await asyncio.wait_for( + self._control_client().request( + server_name=_POLICY_SERVER_NAME, + url_path=path, + method=method, + headers=headers, + **kwargs, + ), + timeout=self._control_timeout_s, + ) + except asyncio.TimeoutError: + raise RuntimeError( + f"gate control call {method} {path} exceeded " + f"{self._control_timeout_s}s (gate unreachable or stalled)" + ) from None + if response.status != 200: + raise RuntimeError( + f"gate control call {method} {path} failed: " + f"HTTP {response.status} {await response.text()}" + ) + return await response.json() + + async def register_rollouts(self, rollout_ids: list[str]) -> None: + """Create-only registration before dispatch (§ 3.1).""" + for rollout_id in rollout_ids: + await self._control("PUT", f"/ng-control/rollouts/{rollout_id}") + + async def fail_rollouts(self, rollout_ids: list[str], *, reason: str) -> None: + """Best-effort gate cleanup for a cancelled/failed dispatch (§ 7).""" + for rollout_id in rollout_ids: + try: + await self._control( + "POST", + f"/ng-control/rollouts/{rollout_id}/fail", + json={"reason": reason}, + ) + except (RuntimeError, OSError) as error: + # The registration TTL is the backstop when the gate is + # unreachable during teardown. + print(f"fail_rollout({rollout_id}) failed: {error}", flush=True) + + async def gate_metrics(self) -> dict: + return await self._control("GET", "/ng-control/metrics") + async def run_rollouts( self, nemo_gym_examples: list[dict], @@ -492,6 +639,14 @@ async def run_rollouts( # examples carry no `input_image` items (text-only case). encode_images_in_examples(nemo_gym_examples) + if self._token_capture_enabled: + # Receipt mode: register the ids riding each row's run body + # before dispatch; seal at completion. + rollout_ids = [ + example[_NG_ROLLOUT_ID_BODY_KEY] for example in nemo_gym_examples + ] + await self.register_rollouts(rollout_ids) + timer.start("_run_rollouts_total") nemo_gym_result_iterator = self.rch.run_examples( examples=nemo_gym_examples, head_server_config=self.head_server_config @@ -502,6 +657,15 @@ async def run_rollouts( with timer.time(label=f"{timer_prefix}/await_results"): try: nemo_gym_row, nemo_gym_result = await task + except aiohttp.ClientResponseError as e: + # aiohttp exceptions carry CIMultiDictProxy headers that + # Ray cannot pickle across the actor boundary, masking the + # real error with a TypeError; re-raise as a plain, + # picklable RuntimeError. + raise RuntimeError( + f"NemoGym rollout HTTP error: {e.status} " + f"{e.message} url={e.request_info.real_url}" + ) from None except Exception as error: if hasattr(error, "response_content"): print( @@ -512,15 +676,22 @@ async def run_rollouts( raise with timer.time(label=f"{timer_prefix}/postprocess_results"): - nemo_rl_result = self._postprocess_nemo_gym_to_nemo_rl_result( - nemo_gym_row, - nemo_gym_result, - tokenizer, - include_initial_multimodal_data=not deduplicate_multimodal_data, - ) - if _has_nan_generation_logprobs(nemo_rl_result): - raise RuntimeError("Generation logprobs contain NaN") - + if self._token_capture_enabled: + # Receipt mode: seal at the gate; token-free result. The + # canonical row is rebuilt by the finalizer, so no + # message_log walk (and no NaN check) applies here. + nemo_rl_result = await self._postprocess_receipt_mode( + nemo_gym_row, nemo_gym_result + ) + else: + nemo_rl_result = self._postprocess_nemo_gym_to_nemo_rl_result( + nemo_gym_row, + nemo_gym_result, + tokenizer, + include_initial_multimodal_data=not deduplicate_multimodal_data, + ) + if _has_nan_generation_logprobs(nemo_rl_result): + raise RuntimeError("Generation logprobs contain NaN") num_results += 1 timing_metrics = None if num_results == len(nemo_gym_examples): @@ -551,6 +722,40 @@ async def run_rollouts( yield nemo_gym_row["_rowidx"], nemo_rl_result, timing_metrics + async def _postprocess_receipt_mode( + self, nemo_gym_row: dict, nemo_gym_result: dict + ) -> dict: + """Seal the rollout and return a token-free result (§ 9.1). + + The legacy token walk (and its contiguity assert) does not run: the + gate owns lineage now, output items carry no token arrays, and the + canonical row is rebuilt by the finalizer from staged deltas. The + Ray return carries only the receipt (~100 B/call) beside the + agent-level result. + """ + assert isinstance(nemo_gym_result, dict), ( + f"Hit a non-successful response when querying NeMo Gym for rollouts: {nemo_gym_result}" + ) + rollout_id = nemo_gym_row[_NG_ROLLOUT_ID_BODY_KEY] + try: + receipt = await self._control( + "POST", + f"/ng-control/rollouts/{rollout_id}/seal", + json={"reward": float(nemo_gym_result.get("reward") or 0.0)}, + ) + except (RuntimeError, OSError) as error: + # An unsealable rollout finalizes as a placeholder; the gate's + # registration TTL sweeps its state. + print(f"seal({rollout_id}) failed: {error}", flush=True) + receipt = None + return { + "message_log": [], + "input_message_log": [], + "full_result": nemo_gym_result, + "rollout_id": rollout_id, + "receipt": receipt, + } + def _postprocess_nemo_gym_to_nemo_rl_result( self, nemo_gym_row: dict, @@ -928,6 +1133,7 @@ def spinup_nemo_gym_actor( enable_router_replay: bool, routed_experts_dtype: str, use_fastokens: bool, + token_capture: Optional[dict[str, Any]] = None, ) -> Any: """Spin up the NeMo-Gym actor against the given generation server URLs. @@ -977,6 +1183,7 @@ def spinup_nemo_gym_actor( routed_experts_dtype=routed_experts_dtype, use_fastokens=use_fastokens, initial_global_config_dict=nemo_gym_dict, + token_capture=token_capture, ) nemo_gym_py_exec = get_actor_python_env("nemo_rl.environments.nemo_gym.NemoGym") diff --git a/nemo_rl/experience/blackbox_finalizer.py b/nemo_rl/experience/blackbox_finalizer.py new file mode 100644 index 00000000000..b59d96c73a2 --- /dev/null +++ b/nemo_rl/experience/blackbox_finalizer.py @@ -0,0 +1,374 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Blackbox finalization: token-free receipts + staged deltas -> canonical rows. + +Orchestration only (docs/design-docs/tq-gym-gate-authoritative.md § 5, § 9.1): +per rollout, fetch the staged rows the receipt manifest names through the +``TokenSource``, re-verify them (digest recomputation over fetched values, +shape/mask/finite-logprob checks, length chaining, weight-version tag +equality), then delegate semantics to Gym's pure ``linearize`` +(``main_chain_only`` + ``terminal_hint``). Any rejection becomes a masked +placeholder row — the group always publishes exactly N rows so GRPO group +shape survives; validity folds into ``sample_mask`` (no new train field) and +placeholders copy ``prompt_ids_for_adv`` from a valid sibling so per-prompt +baselines stay well-formed. + +The finalizer is the only reader of the staging partition and clears a +group's staged rows after its canonical rows are durably published. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass, field +from typing import Any, Optional + +import torch + +from nemo_rl.data_plane import KVBatchMeta +from nemo_rl.data_plane.tq_token_sink import TQTokenSink, TQTokenSource +from nemo_rl.experience.payload import pack_payload +from nemo_rl.experience.row_dump import maybe_dump_train_rows + + +@dataclass(frozen=True) +class FinalizedRollout: + """One rollout's canonical row, or its rejection.""" + + rollout_id: str + valid: bool + rejection_reason: Optional[str] + token_ids: list[int] + token_mask: list[float] + logprobs: list[float] + prompt_len: int + reward: float + staging_keys: list[str] + min_wv: Optional[int] = None + max_wv: Optional[int] = None + + +@dataclass +class FinalizedGroup: + """What ``finalize_group`` hands back for ``commit_finalized``.""" + + meta: Optional[KVBatchMeta] + group_min_wv: int + group_max_wv: int + staging_keys: list[str] + metrics: dict[str, float] = field(default_factory=dict) + # True when min_valid_fraction_per_group rejected the whole group; the + # caller aborts the slot instead of committing it. + dropped: bool = False + + +class BlackboxFinalizer: + """Receipts -> verified rows -> N-row publish, off the generation hot path.""" + + def __init__( + self, + dp_client: Any, + *, + partition_id: str, + staging_partition: str, + pad_token_id: int, + mixed_weight_version_policy: str, + min_valid_fraction_per_group: Optional[float], + ) -> None: + self._dp_client = dp_client + self._partition_id = partition_id + self._pad_token_id = int(pad_token_id) + self._mixed_weight_version_policy = mixed_weight_version_policy + self._min_valid_fraction = min_valid_fraction_per_group + self._source = TQTokenSource(dp_client, staging_partition=staging_partition) + # The sink's clear() is the staging-partition delete; no staging + # writes happen here. + self._staging = TQTokenSink(dp_client, staging_partition=staging_partition) + + # ── per rollout ───────────────────────────────────────────────────────── + + def finalize_rollout( + self, rollout_id: str, receipt: Optional[dict[str, Any]], *, reward: float + ) -> FinalizedRollout: + """Verify one receipt against its staged rows and linearize the main chain. + + Never raises for rollout-level problems: every rejection returns an + invalid row whose reason feeds the metrics; the group publisher + substitutes a placeholder. + """ + # Deferred: nemo_gym is an optional extra absent in non-gym runs. + from nemo_gym.token_id_capture.staging.digest import compute_staging_digest + from nemo_gym.token_id_capture.staging.rebuild import RebuildError, linearize + from nemo_gym.token_id_capture.staging.records import RolloutReceipt + + def rejected(reason: str, staging_keys: list[str]) -> FinalizedRollout: + return FinalizedRollout( + rollout_id=rollout_id, + valid=False, + rejection_reason=reason, + token_ids=[], + token_mask=[], + logprobs=[], + prompt_len=0, + reward=reward, + staging_keys=staging_keys, + ) + + if receipt is None: + return rejected("missing_receipt", []) + try: + parsed = RolloutReceipt.model_validate(receipt) + except ValueError as error: + return rejected(f"invalid_receipt:{error}", []) + staging_keys = [record.staging_key for record in parsed.manifest] + if parsed.rollout_id != rollout_id: + return rejected(f"identity_mismatch:{parsed.rollout_id}", staging_keys) + if parsed.failure_reason is not None: + return rejected(f"rollout_failed:{parsed.failure_reason}", staging_keys) + if parsed.capture_poisoned: + return rejected("capture_poisoned", staging_keys) + if not parsed.manifest: + return rejected("empty_manifest", staging_keys) + + try: + snapshots = self._source.fetch(staging_keys) + except KeyError as error: + return rejected(f"missing_staging_row:{error}", staging_keys) + + for record, snapshot in zip(parsed.manifest, snapshots): + if not ( + len(snapshot.token_ids_delta) + == len(snapshot.token_mask_delta) + == len(snapshot.logprobs_delta) + ): + return rejected(f"misaligned_delta:{record.call_id}", staging_keys) + if any(m not in (0.0, 1.0) for m in snapshot.token_mask_delta): + return rejected(f"invalid_token_mask:{record.call_id}", staging_keys) + if any(not math.isfinite(p) for p in snapshot.logprobs_delta): + return rejected(f"non_finite_logprob:{record.call_id}", staging_keys) + if record.delta_len != len(snapshot.token_ids_delta) or ( + snapshot.prev_len + record.delta_len != record.cum_len + ): + return rejected(f"length_mismatch:{record.call_id}", staging_keys) + if snapshot.weight_version != record.weight_version: + return rejected( + f"weight_version_mismatch:{record.call_id}", staging_keys + ) + digest = compute_staging_digest( + rollout_id=rollout_id, + call_id=record.call_id, + prev_len=snapshot.prev_len, + token_ids_delta=snapshot.token_ids_delta, + token_mask_delta=snapshot.token_mask_delta, + logprobs_delta=snapshot.logprobs_delta, + ) + if digest != record.digest: + return rejected(f"digest_mismatch:{record.call_id}", staging_keys) + + # Parent pointers are lineage state, not storage state: rejoin from + # the manifest before rebuilding (the storage rows carry them too, + # but the receipt is authoritative). + rejoined = [ + snapshot.model_copy(update={"parent_call_id": record.parent_call_id}) + for snapshot, record in zip(snapshots, parsed.manifest) + ] + try: + row = linearize( + rollout_id, + rejoined, + parsed.manifest, + terminal_hint=parsed.terminal_call_id, + ) + except (RebuildError, NotImplementedError) as error: + return rejected(f"rebuild_failed:{error}", staging_keys) + + weight_versions = [record.weight_version for record in parsed.manifest] + min_wv, max_wv = min(weight_versions), max(weight_versions) + if self._mixed_weight_version_policy == "reject" and min_wv != max_wv: + return rejected(f"mixed_weight_versions:{min_wv}..{max_wv}", staging_keys) + + return FinalizedRollout( + rollout_id=rollout_id, + valid=True, + rejection_reason=None, + token_ids=row.token_ids, + token_mask=row.token_mask, + logprobs=row.logprobs, + prompt_len=row.prompt_len, + reward=reward, + staging_keys=staging_keys, + min_wv=min_wv, + max_wv=max_wv, + ) + + # ── per group ─────────────────────────────────────────────────────────── + + def finalize_group( + self, + group_id: str, + rollout_ids: list[str], + receipts: list[Optional[dict[str, Any]]], + rewards: list[float], + *, + fallback_weight_version: int, + ) -> FinalizedGroup: + """Publish exactly N canonical rows for one prompt group. + + Blocking (TQ round trips); run via ``asyncio.to_thread`` from the + dispatch task. ``fallback_weight_version`` stamps a group none of + whose rollouts produced a valid row (placeholder-only groups still + need a staleness tag). + """ + assert len(rollout_ids) == len(receipts) == len(rewards), ( + "rollout_ids, receipts, and rewards must be parallel" + ) + rows = [ + self.finalize_rollout(rollout_id, receipt, reward=reward) + for rollout_id, receipt, reward in zip(rollout_ids, receipts, rewards) + ] + valid_rows = [row for row in rows if row.valid] + staging_keys = [key for row in rows for key in row.staging_keys] + metrics = { + "finalize/invalid_row_rate": 1.0 - len(valid_rows) / len(rows), + "finalize/calls_per_rollout": ( + sum(len(row.staging_keys) for row in rows) / len(rows) + ), + } + for row in rows: + if not row.valid: + print( + f" finalize: rollout {row.rollout_id} rejected " + f"({row.rejection_reason}) — placeholder", + flush=True, + ) + + group_min_wv = min( + (r.min_wv for r in valid_rows), default=fallback_weight_version + ) + group_max_wv = max( + (r.max_wv for r in valid_rows), default=fallback_weight_version + ) + + valid_fraction = len(valid_rows) / len(rows) + if ( + self._min_valid_fraction is not None + and valid_fraction < self._min_valid_fraction + ): + self._clear_staging(staging_keys) + metrics["finalize/group_dropped"] = 1.0 + return FinalizedGroup( + meta=None, + group_min_wv=group_min_wv, + group_max_wv=group_max_wv, + staging_keys=[], + metrics=metrics, + dropped=True, + ) + + # Placeholders borrow a valid sibling's prompt ids so per-prompt + # baselines group correctly; an all-placeholder group uses a single + # pad token (its rows all carry sample_mask 0 and never train). + sibling_prompt = ( + valid_rows[0].token_ids[: valid_rows[0].prompt_len] if valid_rows else [] + ) or [self._pad_token_id] + + n = len(rows) + seq_lens = [max(1, len(row.token_ids)) for row in rows] + max_len = max(seq_lens) + input_ids = torch.full((n, max_len), self._pad_token_id, dtype=torch.int64) + token_mask = torch.zeros((n, max_len), dtype=torch.float32) + logprobs = torch.zeros((n, max_len), dtype=torch.float32) + prompt_ids_for_adv = torch.tensor([sibling_prompt] * n, dtype=torch.int64) + sample_mask = torch.zeros(n, dtype=torch.float32) + lengths = torch.tensor(seq_lens, dtype=torch.long) + rewards_t = torch.tensor([row.reward for row in rows], dtype=torch.float32) + for i, row in enumerate(rows): + if not row.valid: + continue + length = len(row.token_ids) + input_ids[i, :length] = torch.tensor(row.token_ids, dtype=torch.int64) + token_mask[i, :length] = torch.tensor(row.token_mask, dtype=torch.float32) + logprobs[i, :length] = torch.tensor(row.logprobs, dtype=torch.float32) + sample_mask[i] = 1.0 + + train_batch = { + "input_ids": input_ids, + "input_lengths": lengths, + "generation_logprobs": logprobs, + "token_mask": token_mask, + "sample_mask": sample_mask, + "prompt_ids_for_adv": prompt_ids_for_adv, + "total_reward": rewards_t, + } + sample_ids, fields, tags = pack_payload( + train_batch, weight_version=group_min_wv, group_id=group_id + ) + maybe_dump_train_rows( + source="finalizer", + group_id=group_id, + sample_ids=list(sample_ids), + train_batch=train_batch, + weight_version=group_min_wv, + ) + assert sample_ids == rollout_ids, ( + "canonical sample ids must equal the gate-registered rollout ids: " + f"{sample_ids} != {rollout_ids}" + ) + self._call_dp( + "put_samples", + sample_ids=sample_ids, + partition_id=self._partition_id, + fields=fields, + tags=tags, + ) + self._clear_staging(staging_keys) + + meta = KVBatchMeta( + partition_id=self._partition_id, + task_name="train", + sample_ids=list(sample_ids), + fields=list(fields.keys()), + sequence_lengths=[int(s) for s in lengths.tolist()], + tags=[dict(t) for t in tags], + ) + return FinalizedGroup( + meta=meta, + group_min_wv=group_min_wv, + group_max_wv=group_max_wv, + # Already cleared; nothing left for eviction to clear. + staging_keys=[], + metrics=metrics, + ) + + # ── internals ─────────────────────────────────────────────────────────── + + def _clear_staging(self, staging_keys: list[str]) -> None: + if not staging_keys: + return + try: + self._staging.clear(staging_keys) + except Exception as error: # noqa: BLE001 — cleanup must not fail the group; TTL sweeps leftovers + print( + f" finalize: staging clear failed ({error}); TTL will sweep", + flush=True, + ) + + def _call_dp(self, method_name: str, **kwargs: Any) -> Any: + import ray + + method = getattr(self._dp_client, method_name) + remote = getattr(method, "remote", None) + if remote is not None: + return ray.get(remote(**kwargs)) + return method(**kwargs) diff --git a/nemo_rl/experience/rollout_manager.py b/nemo_rl/experience/rollout_manager.py index 87c27954901..94f1b1c63cd 100644 --- a/nemo_rl/experience/rollout_manager.py +++ b/nemo_rl/experience/rollout_manager.py @@ -15,6 +15,7 @@ import asyncio import copy import json +import uuid from typing import Any, Optional import torch @@ -68,15 +69,21 @@ def __init__( self._max_rollout_turns = max_rollout_turns self._policy_generation = policy_generation - async def run_rollout(self, input_sample: DatumSpec) -> PromptGroupRecord: + async def run_rollout( + self, input_sample: DatumSpec, *, rollout_ids: Optional[list[str]] = None + ) -> PromptGroupRecord: """Run num_generations_per_prompt rollouts for one prompt. Args: input_sample: A single prompt (one DatumSpec entry). + rollout_ids: Unsupported here — token capture is NeMo-Gym only. Returns: PromptGroupRecord with num_generations_per_prompt completions. """ + assert rollout_ids is None, ( + "token capture (rollout_ids) is only supported on the NeMo-Gym path" + ) timer = Timer() timer_prefix = "timing/rollout" timer.start(f"{timer_prefix}/total") @@ -439,11 +446,17 @@ def __init__( self._validate_init_params() - async def run_rollout(self, input_sample: DatumSpec) -> PromptGroupRecord: + async def run_rollout( + self, input_sample: DatumSpec, *, rollout_ids: Optional[list[str]] = None + ) -> PromptGroupRecord: """Run num_generations_per_prompt rollouts for one prompt. Args: input_sample: A single prompt (one DatumSpec entry). + rollout_ids: Token-capture mode: gate-registered rollout ids, one + per generation, riding each row's run body as the opaque + ``_ng_rollout_id`` key (agents stamp /ng-rollout/ from it; + zero agent changes). Returns: PromptGroupRecord with num_generations_per_prompt completions. @@ -452,7 +465,7 @@ async def run_rollout(self, input_sample: DatumSpec) -> PromptGroupRecord: timer_prefix = "timing/rollout" timer.start(f"{timer_prefix}/total") - rollout_inputs = self._build_inputs(input_sample) + rollout_inputs = self._build_inputs(input_sample, rollout_ids=rollout_ids) completions, prompt_message_log, rollout_metrics = await self._run_rollouts( rollout_inputs, timer, timer_prefix ) @@ -483,7 +496,9 @@ def _validate_init_params(self) -> None: "Please set `max_rollout_turns` to 1." ) - def _build_inputs(self, input_sample: DatumSpec) -> list[dict]: + def _build_inputs( + self, input_sample: DatumSpec, *, rollout_ids: Optional[list[str]] = None + ) -> list[dict]: """Build N row dicts from input_sample, applying generation config params.""" # Build a template row from the input_sample's extra_env_info, applying generation params. template_row: dict = copy.deepcopy(input_sample["extra_env_info"]) # type: ignore @@ -504,10 +519,19 @@ def _build_inputs(self, input_sample: DatumSpec) -> list[dict]: ) # Build N rows with distinct rowidxs so run_rollouts can sort them correctly. + if rollout_ids is not None: + assert len(rollout_ids) == self._num_generations_per_prompt, ( + "token-capture rollout ids must be one per generation" + ) rows = [] for i in range(self._num_generations_per_prompt): row = copy.deepcopy(template_row) row["_rowidx"] = i + if rollout_ids is not None: + # Opaque run-body carrier (Gym's _ng_rollout_id key): the agent + # derives the id from the run body and stamps /ng-rollout/ + # on every model call, so the TQ sample id IS the capture key. + row["_ng_rollout_id"] = rollout_ids[i] rows.append(row) return rows @@ -564,6 +588,21 @@ async def _run_rollouts( def _result_to_completion(self, result: dict) -> Completion: """Convert one run_rollouts result dict into a Completion.""" + if "receipt" in result: + # Receipt mode (token capture): the result is token-free — the + # message_log is empty and the canonical row is rebuilt by the + # finalizer from staged deltas. The receipt and rollout id ride + # env_extras for the finalize step. + env_extras = dict(result["full_result"]) + env_extras["ng_receipt"] = result["receipt"] + env_extras["ng_rollout_id"] = result["rollout_id"] + return Completion( + message_log=result["message_log"], + env_extras=env_extras, + truncated=False, + reward=float(result["full_result"]["reward"]), + ) + # Tensorize token fields. _tensorize_by_key(result["message_log"], "token_ids") _tensorize_by_key( @@ -598,29 +637,54 @@ def _compute_rollout_metrics( """Aggregate per-sample and per-agent metrics.""" # Prepare lists of values for each metric. total_reward = [c.reward for c in completions] - turn_count = [ - sum(1 for m in c.message_log if m["role"] == "user") for c in completions - ] - # token metrics - total_tokens = [ - sum(len(m["token_ids"]) for m in c.message_log) for c in completions - ] - assistant_tokens = [ - sum(len(m["token_ids"]) for m in c.message_log if m["role"] == "assistant") - for c in completions - ] - # max_gen_tokens_per_turn: Diagnostic for long single generations - max_gen_tokens_per_turn = [ - max( - ( + receipt_mode = bool(completions) and "ng_receipt" in completions[0].env_extras + if receipt_mode: + # Token-free receipts: token accounting comes from the manifest + # (cum_len of the deepest chain; delta sums as the generation + # proxy) instead of a message_log walk. + manifests = [ + ((c.env_extras.get("ng_receipt") or {}).get("manifest") or []) + for c in completions + ] + turn_count = [len(m) for m in manifests] + total_tokens = [ + max((entry["cum_len"] for entry in m), default=0) for m in manifests + ] + assistant_tokens = [ + sum(entry["delta_len"] for entry in m) for m in manifests + ] + max_gen_tokens_per_turn = [ + max((entry["delta_len"] for entry in m), default=0) for m in manifests + ] + else: + turn_count = [ + sum(1 for m in c.message_log if m["role"] == "user") + for c in completions + ] + # token metrics + total_tokens = [ + sum(len(m["token_ids"]) for m in c.message_log) for c in completions + ] + assistant_tokens = [ + sum( len(m["token_ids"]) for m in c.message_log if m["role"] == "assistant" - ), - default=0, - ) - for c in completions - ] + ) + for c in completions + ] + # max_gen_tokens_per_turn: Diagnostic for long single generations + max_gen_tokens_per_turn = [ + max( + ( + len(m["token_ids"]) + for m in c.message_log + if m["role"] == "assistant" + ), + default=0, + ) + for c in completions + ] # truncated metrics truncated = [c.truncated for c in completions] @@ -644,8 +708,12 @@ def _compute_rollout_metrics( "truncation_rate": sum(truncated) / n, } - # Agent-level metrics. - agent_extras = [c.env_extras for c in completions] + # Agent-level metrics. Receipts are lineage records, not agent + # results — keep them (and their manifests) out of the logged table. + agent_extras = [ + {k: v for k, v in c.env_extras.items() if k not in ("ng_receipt",)} + for c in completions + ] for key in agent_extras[0].keys(): values = [ float(r[key]) # type: ignore @@ -683,10 +751,15 @@ def __init__( use_nemo_gym: bool = False, mask_env_flagged_samples: bool = True, tq_buffer: Optional[TQReplayBuffer] = None, + finalizer: Optional[Any] = None, ) -> None: assert num_generations_per_prompt >= 1, ( "num_generations_per_prompt must be >= 1" ) + if finalizer is not None: + assert use_nemo_gym, ( + "token capture (finalizer) is only supported on the NeMo-Gym path" + ) if not use_nemo_gym: rollout_cls = AsyncRolloutImpl @@ -713,6 +786,10 @@ def __init__( self._tokenizer = tokenizer self._num_generations_per_prompt = num_generations_per_prompt self._tq_buffer = tq_buffer + self._finalizer = finalizer + # The NeMo-Gym env handle doubles as the gate control-plane proxy + # (gate_metrics / fail_rollouts) on the capture path. + self._env_handles = task_to_env self._weight_version: int = 0 def set_weight_version(self, version: int) -> None: @@ -723,8 +800,26 @@ def set_weight_version(self, version: int) -> None: """ self._weight_version = int(version) - async def run_rollout(self, input_sample: DatumSpec) -> PromptGroupRecord: - return await self._impl.run_rollout(input_sample) + async def gate_metrics(self) -> Optional[dict[str, int]]: + """Fetch the capture gate's § 8 counters, or None off the capture path. + + Returns: + Cumulative gate counters (token_in, fallback_*, capture_failed, + registered/sealed/failed/expired) from ``/ng-control/metrics``, + or None when no NemoGym env handle is wired. + """ + env = self._env_handles.get("nemo_gym") if self._env_handles else None + if env is None: + return None + return await env.gate_metrics.remote() + + async def run_rollout( + self, input_sample: DatumSpec, *, rollout_ids: Optional[list[str]] = None + ) -> PromptGroupRecord: + if rollout_ids is None: + # Legacy path: keep the impl call signature byte-identical. + return await self._impl.run_rollout(input_sample) + return await self._impl.run_rollout(input_sample, rollout_ids=rollout_ids) async def generate_and_push( self, @@ -744,6 +839,9 @@ async def generate_and_push( assert self._tq_buffer is not None, ( "generate_and_push requires tq_buffer to be set at __init__" ) + if self._finalizer is not None: + await self._generate_and_finalize(input_sample, target_step=target_step) + return start_version = self._weight_version group_id = self._tq_buffer.reserve( weight_version=start_version, target_step=target_step @@ -777,3 +875,66 @@ async def generate_and_push( flush=True, ) raise + + async def _generate_and_finalize( + self, input_sample: DatumSpec, *, target_step: Optional[int] = None + ) -> None: + """Token-capture dispatch: receipts in, canonical rows via the finalizer. + + Mints the group's rollout ids up front — sample ids and gate-registered + rollout ids are the same strings (``{group_id}_g{i}``) — reserves the + slot with them so cleanup can name what it owns before a receipt + exists, and commits via ``commit_finalized`` with the group's + min/max call weight versions. + """ + start_version = self._weight_version + group_id = str(uuid.uuid4()) + rollout_ids = [ + f"{group_id}_g{i}" for i in range(self._num_generations_per_prompt) + ] + self._tq_buffer.reserve( + weight_version=start_version, + target_step=target_step, + group_id=group_id, + rollout_ids=rollout_ids, + ) + try: + record = await self.run_rollout(input_sample, rollout_ids=rollout_ids) + receipts = [c.env_extras.get("ng_receipt") for c in record.completions] + rewards = [float(c.reward) for c in record.completions] + finalized = await asyncio.to_thread( + self._finalizer.finalize_group, + group_id, + rollout_ids, + receipts, + rewards, + fallback_weight_version=start_version, + ) + record.rollout_metrics.update(finalized.metrics) + if finalized.dropped: + # min_valid_fraction_per_group rejected the group: nothing + # was published, so drop the slot like a failed dispatch. + self._tq_buffer.abort(group_id) + raise RuntimeError( + f"token capture: group {group_id} dropped " + "(min_valid_fraction_per_group)" + ) + await self._tq_buffer.commit_finalized( + group_id, + finalized.meta, + finalized.group_min_wv, + finalized.group_max_wv, + staging_keys=finalized.staging_keys, + ) + except BaseException: + self._tq_buffer.abort(group_id) + # Best-effort gate cleanup: no receipt will ever seal these ids. + nemo_gym_env = self._env_handles.get("nemo_gym") + if nemo_gym_env is not None: + try: + await nemo_gym_env.fail_rollouts.remote( + rollout_ids, reason="dispatch_failed" + ) + except Exception as error: # noqa: BLE001 — TTL is the backstop + print(f"fail_rollouts({group_id}) failed: {error}", flush=True) + raise diff --git a/nemo_rl/experience/row_dump.py b/nemo_rl/experience/row_dump.py new file mode 100644 index 00000000000..12b831a6e11 --- /dev/null +++ b/nemo_rl/experience/row_dump.py @@ -0,0 +1,77 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Env-gated dump of canonical training rows at their TQ publish sites. + +Set ``NRL_SC_DUMP_TRAIN_ROWS=`` to append one JSON line per training row +whenever a group is published to the ``rollout_data`` partition — from the +legacy ``TQReplayBuffer.commit`` path and from the token-capture +``BlackboxFinalizer`` publish. Off (no I/O, no imports of the payload) unless +the env var is set. Used by the S5 legacy-vs-capture offline row diff. +""" + +import json +import os +import threading +from collections.abc import Mapping +from typing import Any, Optional + +import torch + +_DUMP_ENV_VAR = "NRL_SC_DUMP_TRAIN_ROWS" +# The finalizer publishes from a worker thread (asyncio.to_thread) while the +# legacy path publishes from the SC event loop; serialize appends. +_G_WRITE_LOCK = threading.Lock() + + +def _row_value(tensor: torch.Tensor, row: int) -> Any: + value = tensor[row] + if value.dim() == 0: + return value.item() + return value.tolist() + + +def maybe_dump_train_rows( + *, + source: str, + group_id: str, + sample_ids: list[str], + train_batch: Mapping[str, torch.Tensor], + weight_version: Optional[int], +) -> None: + """Append each row of a published group to the dump file, if enabled. + + Args: + source: Publish site tag (``"legacy_commit"`` or ``"finalizer"``). + group_id: Prompt-group id the rows belong to. + sample_ids: Canonical per-row sample ids (``{group_id}_g{i}``). + train_batch: Column tensors as passed to ``pack_payload``. + weight_version: Weight version stamped on the rows' tags. + """ + dump_dir = os.environ.get(_DUMP_ENV_VAR) + if not dump_dir: + return + os.makedirs(dump_dir, exist_ok=True) + path = os.path.join(dump_dir, f"train_rows_{source}.jsonl") + lines = [] + for i, sample_id in enumerate(sample_ids): + record = { + "source": source, + "group_id": group_id, + "sample_id": sample_id, + "weight_version": weight_version, + **{name: _row_value(tensor, i) for name, tensor in train_batch.items()}, + } + lines.append(json.dumps(record)) + with _G_WRITE_LOCK, open(path, "a") as f: + f.write("\n".join(lines) + "\n") diff --git a/nemo_rl/models/generation/vllm/vllm_generation.py b/nemo_rl/models/generation/vllm/vllm_generation.py index 888e2539ce4..2740aa5ea60 100644 --- a/nemo_rl/models/generation/vllm/vllm_generation.py +++ b/nemo_rl/models/generation/vllm/vllm_generation.py @@ -533,6 +533,37 @@ def _post_init(self): results = ray.get(futures) return results + def setup_token_capture( + self, dp_cfg: dict[str, Any], staging_partition: str + ) -> None: + """Install gate-authoritative token capture in every DP-leader worker. + + Called once at setup when ``token_capture.enabled``; each async worker + builds its in-worker data-plane client + TQTokenSink and makes the + single Gym ``install_capture`` call (see + docs/design-docs/tq-gym-gate-authoritative.md § 9.1). + """ + assert self.cfg["vllm_cfg"]["async_engine"], ( + "token capture requires the async vLLM engine (the capture host " + "is the worker's in-process HTTP server)" + ) + futures = self.worker_group.run_all_workers_single_data( + "setup_token_capture", + dp_cfg=dp_cfg, + staging_partition=staging_partition, + run_rank_0_only_axes=["tensor_parallel", "pipeline_parallel"], + ) + ray.get(futures) + + def set_rollout_weight_version(self, version: int) -> None: + """Rotate the weight version workers stamp on captured model calls.""" + futures = self.worker_group.run_all_workers_single_data( + "set_rollout_weight_version", + version=version, + run_rank_0_only_axes=["tensor_parallel", "pipeline_parallel"], + ) + ray.get(futures) + def _get_raw_spec_counters(self) -> dict[str | tuple[str, int], float]: """Collect raw spec decode counters from workers.""" futures = self.worker_group.run_all_workers_single_data( diff --git a/nemo_rl/models/generation/vllm/vllm_worker_async.py b/nemo_rl/models/generation/vllm/vllm_worker_async.py index eec501eb609..9ad9428f11f 100644 --- a/nemo_rl/models/generation/vllm/vllm_worker_async.py +++ b/nemo_rl/models/generation/vllm/vllm_worker_async.py @@ -16,6 +16,7 @@ import copy import gc import logging +import os import threading import time import uuid @@ -100,6 +101,17 @@ def __init__( self.base_url = None self.http_server = None + # Gate-authoritative token capture (dormant until the + # setup_token_capture fan-out runs; see + # docs/design-docs/tq-gym-gate-authoritative.md § 9.1). The weight + # version is stamped per model call at begin_call time and rotated by + # the set_rollout_weight_version fan-out from the SC's _sync_weights. + self.token_capture = None + self._rollout_weight_version = 0 + # In-flight captured calls keyed by id(request): (ActiveCall, the + # exact engine prompt ids recorded at preprocess time). + self._capture_calls: dict[int, tuple[Any, list[int]]] = {} + super().__init__( config, bundle_indices, @@ -345,8 +357,100 @@ async def get_reserved_url(self) -> Optional[str]: async def report_dp_openai_server_base_url(self) -> Optional[str]: return self.base_url + def install_token_capture(self, capture: Any) -> None: + """Gym's ``install_capture`` seam (the ``CaptureHost`` contract).""" + self.token_capture = capture + + async def setup_token_capture( + self, dp_cfg: dict[str, Any], staging_partition: str + ) -> bool: + """Host gate-authoritative token capture in this worker. + + Fan-out target (token_capture.enabled only): builds the in-worker + data-plane client and TQTokenSink, then makes the single + ``install_capture`` call wiring Gym's engine-blind capture core + + vLLM adapter into this worker. Returns whether capture was installed + (False on non-model-owner ranks, which serve no HTTP). + """ + if not self.is_model_owner: + return False + # Deferred: nemo_gym is an optional extra absent in non-gym runs. + from nemo_gym.token_id_capture.adapters.vllm import VLLMCaptureAdapter + from nemo_gym.token_id_capture.staging import install_capture + + from nemo_rl.data_plane import build_data_plane_client + from nemo_rl.data_plane.tq_token_sink import TQTokenSink + + dp_client = build_data_plane_client(dp_cfg, bootstrap=False) + sink = TQTokenSink(dp_client, staging_partition=staging_partition) + install_capture( + self, + sink=sink, + weight_version_fn=lambda: self._rollout_weight_version, + adapter=VLLMCaptureAdapter(), + ) + return True + + async def set_rollout_weight_version(self, version: int) -> None: + """Rotate the weight version stamped on subsequent captured calls.""" + self._rollout_weight_version = int(version) + + def _begin_request_capture(self, request: Any, prompt_token_ids: list[int]) -> None: + """Admit one gate-forwarded call into the capture layer. + + Called from preprocess_chat once the exact engine prompt is known + (post-splice in token-in mode, full render in text mode). No-op + unless capture is installed and the request carries the gate's + ``ng_capture`` context. + """ + capture = self.token_capture + context = getattr(request, "ng_capture", None) + if capture is None or not context: + return + call = capture.begin_call( + rollout_id=context["rollout_id"], + call_id=context["call_id"], + parent_call_id=context.get("parent_call_id"), + prev_len=int(context.get("prev_len") or 0), + mode=context.get("mode") or "text", + stream=bool(getattr(request, "stream", False)), + ) + self._capture_calls[id(request)] = (call, list(prompt_token_ids)) + + def _finish_request_capture(self, request: Any, content: dict) -> dict: + """Stage the finished call and ride its coords on the response. + + Fail-closed (§ 3.5): the sink write happens inside complete_call — + the coords exist only after the bytes are durable, and any capture + failure degrades to capture_failed coords without breaking the + completion. Token ids and logprobs are stripped: the staged delta is + the only token store on this path, so the worker->gate hop carries + text + delta ids + coords only (§ 3.2). + """ + state = self._capture_calls.pop(id(request), None) + if state is None: + return content + call, prompt_token_ids = state + payload = dict(content) + # vLLM's OpenAI response carries no prompt ids; the adapter reads the + # preprocess-time engine prompt off the payload (see + # nemo_gym.token_id_capture.adapters.vllm.extract_prompt_ids). + payload["prompt_token_ids"] = prompt_token_ids + coords = self.token_capture.complete_call_from_response(call, payload) + for choice in content.get("choices") or []: + choice.pop("logprobs", None) + content["ng_commit_coords"] = coords.model_dump() + return content + + def _abort_request_capture(self, request: Any, *, reason: str) -> None: + """Drop the in-flight capture state for a request that errored.""" + state = self._capture_calls.pop(id(request), None) + if state is not None and self.token_capture is not None: + self.token_capture.fail_call(state[0], reason=reason) + # ruff: noqa def _setup_vllm_openai_api_server(self, app: FastAPI) -> FastAPI: + worker_self = self from copy import deepcopy from logging import Filter as LoggingFilter from logging import LogRecord @@ -514,6 +618,11 @@ async def preprocess_chat( actual_request_max_tokens, res[1][0]["prompt_token_ids"], ) + # Token capture, text mode: the full render is the exact + # engine prompt. + worker_self._begin_request_capture( + request, res[1][0]["prompt_token_ids"] + ) return res last_assistant_message_idx = None @@ -570,6 +679,10 @@ async def preprocess_chat( final_prompt_token_ids, ) + # Token capture, token-in mode: the spliced prompt is the + # exact engine prompt. + worker_self._begin_request_capture(request, final_prompt_token_ids) + return res ######################################## @@ -581,6 +694,9 @@ class NeMoRLChatCompletionRequest( NeMoRLOpenAIChatRequestMixin, ChatCompletionRequest ): required_prefix_token_ids: Optional[List[int]] = None + # Gate-authoritative token capture: the call identity the gate + # attaches (rollout_id, call_id, parent_call_id, prev_len, mode). + ng_capture: Optional[dict[str, Any]] = None # vLLM 0.25 routes both /v1/chat/completions and /tokenize through # OnlineRenderer.preprocess_chat, so the prefix-token override @@ -745,6 +861,7 @@ async def create_chat_completion( # max_model_len during tokenization, instead of returning an # ErrorResponse. Convert to HTTP 400 so the Gym proxy can # detect context-length overflow and handle it gracefully. + worker_self._abort_request_capture(request, reason="context_length") return JSONResponse( content={ "error": { @@ -755,19 +872,26 @@ async def create_chat_completion( }, status_code=400, ) + except BaseException: + worker_self._abort_request_capture(request, reason="engine_error") + raise if isinstance(generator, ErrorResponse): + worker_self._abort_request_capture(request, reason="error_response") return JSONResponse( content=generator.model_dump(), status_code=generator.error.code ) elif isinstance(generator, ChatCompletionResponse): - return JSONResponse( - content=model_dump_chat_response_with_dynamic_message_fields( - generator - ) + content = model_dump_chat_response_with_dynamic_message_fields( + generator ) + # Token capture: stage the delta and ride the coords on the + # response; strips logprobs/ids (no-op when capture is off). + content = worker_self._finish_request_capture(request, content) + return JSONResponse(content=content) + worker_self._abort_request_capture(request, reason="streaming_response") return StreamingResponse(content=generator, media_type="text/event-stream") ######################################## @@ -899,6 +1023,13 @@ def _setup_vllm_server(self) -> "tuple[threading.Thread, str, uvicorn.Server]": base_url = f"http://{node_ip}:{free_port}/v1" print(f"Starting server on {base_url}") + byte_dir = os.environ.get("NRL_HTTP_BYTES_DIR") + if byte_dir: + # Perf-measurement tooling only (see nemo_rl/utils/http_byte_counter.py). + from nemo_rl.utils.http_byte_counter import HttpByteCounterMiddleware + + app = HttpByteCounterMiddleware(app, "vllm_worker", byte_dir) # type: ignore[assignment] + config = uvicorn.Config( app, host="0.0.0.0", diff --git a/nemo_rl/utils/http_byte_counter.py b/nemo_rl/utils/http_byte_counter.py new file mode 100644 index 00000000000..7038d35b053 --- /dev/null +++ b/nemo_rl/utils/http_byte_counter.py @@ -0,0 +1,82 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Env-gated per-route HTTP byte counter (``NRL_HTTP_BYTES_DIR``). + +Measurement tooling for the token-capture perf comparison: sums request-body +and response-body bytes per path on the vLLM worker's in-process HTTP server +and periodically flushes an aggregate JSON. Mirrors Gym's +``HttpByteCounterMiddleware`` (separate copy: worker venvs cannot assume +``nemo_gym`` is installed on the legacy path). Never installed unless the env +var is set. +""" + +import json +import os +from typing import Any, Awaitable, Callable + +Scope = dict[str, Any] +Message = dict[str, Any] +Receive = Callable[[], Awaitable[Message]] +Send = Callable[[Message], Awaitable[None]] + + +class HttpByteCounterMiddleware: + """Pure ASGI wrapper counting per-path request/response body bytes.""" + + FLUSH_EVERY = 25 + + def __init__(self, app: Any, server_name: str, out_dir: str) -> None: + self.app = app + self.out_path = os.path.join(out_dir, f"{server_name}_{os.getpid()}.json") + os.makedirs(out_dir, exist_ok=True) + self.counts: dict[str, list[int]] = {} + self._events = 0 + + def _flush(self) -> None: + with open(self.out_path, "w") as f: + json.dump( + { + path: { + "requests": c[0], + "req_bytes": c[1], + "resp_bytes": c[2], + } + for path, c in self.counts.items() + }, + f, + ) + + async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: + if scope["type"] != "http": + return await self.app(scope, receive, send) + entry = self.counts.setdefault(scope["path"], [0, 0, 0]) + entry[0] += 1 + + async def counting_receive() -> Message: + message = await receive() + if message["type"] == "http.request": + entry[1] += len(message.get("body", b"")) + return message + + async def counting_send(message: Message) -> None: + if message["type"] == "http.response.body": + entry[2] += len(message.get("body", b"")) + await send(message) + + try: + await self.app(scope, counting_receive, counting_send) + finally: + self._events += 1 + if self._events % self.FLUSH_EVERY == 0: + self._flush() diff --git a/pyrefly.toml b/pyrefly.toml index e0c35ebf542..877c6a1b80c 100644 --- a/pyrefly.toml +++ b/pyrefly.toml @@ -9,6 +9,7 @@ replace-imports-with-any = [ "tensorrt_llm.*", "vllm.*", "math_verify.*", + "nemo_gym.*", "sympy.*", "torchdata.*", "torchaudio.*", @@ -123,6 +124,7 @@ project-includes = [ "nemo_rl/data_plane/observability.py", "nemo_rl/data_plane/preshard.py", "nemo_rl/data_plane/schema.py", + "nemo_rl/data_plane/tq_token_sink.py", "nemo_rl/data_plane/worker_mixin.py", "nemo_rl/distributed/__init__.py", "nemo_rl/distributed/collectives.py", @@ -143,6 +145,8 @@ project-includes = [ "nemo_rl/evals/__init__.py", "nemo_rl/evals/answer_parsing.py", "nemo_rl/experience/__init__.py", + "nemo_rl/experience/blackbox_finalizer.py", + "nemo_rl/experience/row_dump.py", "nemo_rl/experience/interfaces.py", "nemo_rl/experience/metric_utils.py", "nemo_rl/experience/payload.py", @@ -214,6 +218,7 @@ project-includes = [ "nemo_rl/utils/config.py", "nemo_rl/utils/fastokens.py", "nemo_rl/utils/grad_norm.py", + "nemo_rl/utils/http_byte_counter.py", "nemo_rl/utils/multimodal_payload_metrics.py", "nemo_rl/utils/native_checkpoint.py", "nemo_rl/utils/nsys.py", diff --git a/swe/EXPERIMENT_LAUNCH.md b/swe/EXPERIMENT_LAUNCH.md new file mode 100644 index 00000000000..f9f3e9801f4 --- /dev/null +++ b/swe/EXPERIMENT_LAUNCH.md @@ -0,0 +1,129 @@ +# Experiment Launch Plan — SWE Token-Capture A/B (2026-07-28) + +Concrete launch record for the run described generically in `SWE_RUN.md`. +Everything here is pinned to what will actually execute; update this file if +any pin changes before submission. **Status: READY — not yet submitted.** + +## 1. Hypothesis and expected outcome + +The gate-authoritative token-capture pipeline +(`docs/design-docs/tq-gym-gate-authoritative.md`) reduces HTTP bytes per +trained token and wall-clock step time vs the legacy token-echo path on a +real multi-turn agentic workload, at equal training quality. + +| Expectation | Basis | +|---|---| +| Bytes/token reduction ≥ 35.6 % (the single-turn floor), toward/beyond −46.9 % (sync-prototype multi-turn) | S5 instrumented A/B; legacy pays per-turn history re-echo (~quadratic in turns), capture stays O(generated tokens) | +| `exposed_generation` and `total_step_time` at or below legacy | S5 saw −16.8 % / −9.0 % single-turn; expect relative timing win to compress on a 30B MoE (GPU decode dominates) while the bytes win grows | +| `gate/token_in_rate` → 1.0 for continuation calls | First real measurement — OpenHands marker survival is the biggest unknown (risk #1); a low rate is a correctness-preserving perf bug, not a training bug | +| Reward / KL curves in the same band on both arms | S5: identical rewards 10/10, same KL band | + +## 2. Exact pins + +| What | Value | +|---|---| +| RL repo | branch `yukih/sc-entrypoint` @ `e71055650` (local-only; no remote has it) | +| Gym fork (submodule) | branch `tq-gate-capture` @ `e3b3eac6` (= upstream #2124 head `32b555f04` + S1–S3 capture commits + S5 byte-counter middleware; local-only) | +| Code snapshot | `code_snapshots/sc-swe-capture` @ `e710556` (immutable run tree; both arms run from it) | +| Container | `nemo-rl:sc-swe-baked.sqsh` (biguo; nightly-062526 + baked venvs — **stale vs branch lock**, hence forced venv rebuild below) | +| Model | Qwen3-30B-A3B-Thinking-2507, from SWE1 `step_230_hf` checkpoint (bihu, run dc3m70us lineage) | +| Train data | R2E-Gym subset jsonl (sdevare), 4,518 samples | +| Site config | `test_assets/SWE/grpo_qwen3_30b_async_swe.yaml` (real sandbox-image paths; snapshot copy is authoritative at run time) | +| Derived arm configs | `test_assets/SWE/derived_configs/grpo_swe_ab_{legacy,capture}.yaml` — regenerated at each submit by `make_capture_config.py` | + +Verified diff between derived arm configs: **only** `token_capture.enabled: +true` on the capture arm; `rollout_max_attempts_to_avoid_lp_nan: 1` pinned on +**both** arms (capture hard-errors without it; pinning both removes the +NaN-retry confound). + +## 3. Compute shape (per job; one job per arm) + +- 16 nodes × 8 GPUs (128 GPUs), SLURM `batch` partition, account + `coreai_dlalgo_genai`, 4 h wall time, `--exclusive`, + `--dependency=singleton` (arms queue back-to-back on comparable + allocations), idle-GPU-reaper exemption (180 min) in the job comment. +- 8 training nodes: Megatron-Core, TP=4 EP=8 CP=4 PP=2, pad 32. +- 8 generation nodes: async vLLM, TP=2, non-colocated. +- Single-controller entrypoint (`SC_MODE=1`, + `examples/run_grpo_single_controller.py`) + TransferQueue data plane. +- Sequence budget 131,072; GBS=64 (8 prompts/step × 8 generations/prompt); + LR 1e-6; staleness `age=1`, `force_in_order=true`, `over_sampling=false`. +- Agent: NeMo-Gym OpenHands, per-instance Singularity sandboxes (`.sif` + under igitman/sdevare lustre trees), up to ~100–200 turns/rollout. + +## 4. Environment posture (identical on both arms) + +- `NRL_FORCE_REBUILD_VENVS=true` — container venvs predate the branch's + `uv.lock`; capture arm additionally needs the unbaked `VLLM_GYM` worker + venv. First node-local build may take ~30 min (reaper exemption covers it). +- `GYM_VENV_DIR=/tmp/nemo_gym_venvs` — baked `/opt/gym_venvs` deps predate + the Gym fork's floors (aiohttp bump). +- `BYTES=1` (capture-arm measurement run): `NG_HTTP_BYTES_DIR` + + `NRL_HTTP_BYTES_DIR` → per-server JSON under `http_bytes//`; + forwarding into the job env added in `e71055650`. +- Secrets: `profiles/env.sh` (mode 600) sourced by the site wrapper; wrapper + fails loudly if `WANDB_API_KEY`/`HF_TOKEN` are unset. + +## 5. Run sequence and gates + +All submissions via (from repo root): + +```bash +SITE_WRAPPER=$PWD/test_assets/SWE/grpo_swe_tests.sh MAX_NUM_STEPS= ARM= [BYTES=1] bash swe/launch_swe_ab.sh +``` + +| # | Run | Purpose | Gate to proceed | +|---|---|---|---| +| 1 | `ARM=legacy MAX_NUM_STEPS=3` | Flag-off smoke: branch + Gym pin reproduce the known-good legacy SWE run | Job completes 3 steps; reward/KL sane; no venv/import failures | +| 2 | `ARM=legacy MAX_NUM_STEPS=25` | Measurement baseline | Completes ≥ 20 steps cleanly | +| 3 | `ARM=capture MAX_NUM_STEPS=25 BYTES=1` | Measurement + mechanism + byte accounting | — | +| 4 | (optional) repeat pair ×2 | Variance bars; S5 assessment puts run-to-run variance at ±5–10 % | 1 pair = directional, 3 pairs = quotable | + +W&B: project `pthombre-swe-capture-ab`, runs named +`swe-ab--`. Checkpoints: `/results//`. + +## 6. Analysis plan + +- Medians over steps 5..N (skip warm-up): `timing/train/total_step_time`, + `timing/train/exposed_generation`, `timing/train/valid_tokens_per_sec_per_gpu`. +- Mechanism: `gate/token_in_rate` (the headline first-time measurement), + `gate/fallback_{no_marker,fingerprint_miss,unknown_marker}` — sustained + `fingerprint_miss` ⇒ OpenHands rewrites history (marker-survival risk #1; + report back to the design doc §11). +- Health: `gate/capture_failed`, `train/global_valid_seqs` == GBS both arms; + `finalize/*` latency (if it dominates dispatch → argues for H5 finalizer pool). +- Equivalence: overlay `train/reward` + `train/gen_kl_error` across arms. +- Bytes: `python swe/aggregate_perf.py ` + → total bytes, bytes/trained-token, per-hop table. +- Results recorded in + `docs/design-docs/tq-gym-gate-authoritative-implementation-log.md` (post-MVP + perf report per design § 10). + +## 7. Known risks (accepted, watched) + +- **Gate death = silent stall** (S5 chaos finding; H1 fix queued): control + plane retries a dead gate unboundedly → job hangs, doesn't fail. + Supervise via `squeue` + driver log in + `code_snapshots/sc-swe-capture/logs/slurm/`. +- **Gate buffer growth uncapped** (H5): ids-only delta forests; at 131k × + in-flight concurrency raw ids are ~100s MB but Python overhead is real — + watch policy-model server RSS. +- **OpenHands marker survival unknown** — measured, not assumed + (`token_in_rate`); low rate degrades perf, never correctness. +- **Secrets in job env**: the launcher embeds `WANDB_API_KEY`/HF token in + the sbatch command (visible in SLURM logs; pre-existing behavior). + Rotate tokens if logs are shared. +- Snapshot is immutable but the **derived configs are re-generated per + submit** — do not edit the site yaml between arms of a pair. + +## 8. Pre-flight checklist (all verified 2026-07-28) + +- [x] S1–S5 + launcher committed (RL `e71055650`, Gym `e3b3eac6`) +- [x] Snapshot `sc-swe-capture` @ `e710556`: S5 files, `examples/swe_bench/` + launcher, site yaml, Gym gate + byte middleware present +- [x] Dry-run both arms end-to-end (`DRY_RUN=1`): correct account, snapshot + paths, singleton, reaper comment; derived-config diff = capture flag only +- [x] Container / model / data / sandbox images readable by pthombre +- [x] SLURM account `coreai_dlalgo_genai` valid for user (nemotron_sw_post is not) +- [x] Secrets profile in place; wrapper verified with tokens stripped from env +- [ ] SUBMIT (awaiting explicit go) diff --git a/swe/SWE_RUN.md b/swe/SWE_RUN.md new file mode 100644 index 00000000000..335eff4e6db --- /dev/null +++ b/swe/SWE_RUN.md @@ -0,0 +1,193 @@ +# SWE Run — Token-Capture vs Legacy Perf A/B at Scale + +Runbook for measuring the performance of the gate-authoritative token-capture +pipeline (`token_capture.enabled=true`, design: +`docs/design-docs/tq-gym-gate-authoritative.md`) against the legacy token-echo +path on a real workload: **async GRPO on SWE-bench** (Qwen3-30B-A3B thinking, +NeMo-Gym OpenHands agent, 16 nodes). This is the multi-turn run the 2-GPU S5 +experiments could not provide — SWE agents run up to 100–200 turns per +rollout, which is exactly where the legacy echo pays its per-turn token +re-transmission and where capture's `token_in_rate` and prefix serving are +finally exercised for real. + +Small-scale S5 results this run extends (implementation log, +`docs/design-docs/tq-gym-gate-authoritative-implementation-log.md` § S5): +−35.6 % HTTP bytes/trained token, −9 % step time, −16.8 % generation time on +a *single-turn* workload — the floor. The sync prototype measured −46.9 % +bytes/token multi-turn. + +--- + +## 1. The stack being launched + +``` +swe/launch_swe_ab.sh (this folder: picks the arm, derives the capture yaml) + └─ site wrapper e.g. .../nemo_rl-sc-test/test_assets/SWE/grpo_swe_tests.sh + (account, container, secrets, lustre paths — user-specific, gitignored) + └─ snapshot launcher /examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh + (env-agnostic: parallelism, async_rl/data_plane overrides, sbatch) + └─ ray.sub → SLURM (16 nodes: 8 train TP4/EP8/CP4/PP2 + 8 gen vLLM TP2, + seqlen 131072, SC_MODE=1 single-controller + TQ) +``` + +Jobs run from an immutable **code snapshot** +(`test_assets/SWE/sync_code.sh`). The snapshot rsyncs `3rdparty/` wholesale, +so the **local-only Gym fork commits ride along** — the open gitlink/CI +question does not block this experiment. + +## 2. The A/B design + +One independent variable. Everything else — snapshot, container, data, model, +parallelism, async_rl knobs, node count — identical between arms. + +| | Arm A: `legacy` | Arm B: `capture` | +|---|---|---| +| Config | site yaml as-is | site yaml + `token_capture.enabled: true` (derived by `make_capture_config.py`) | +| Token path | gate string-parses ids + `/tokenize` per call; ids/logprobs echoed through agent messages every turn; tokens ride the Ray return | gate custodies lineage; worker stages delta ids+logprobs to TQ once; markers + token-free receipts | +| Rows into training | `TQReplayBuffer.commit` tensorize | `BlackboxFinalizer` verify → publish | + +Both arms use the launcher's defaults, which sit inside the capture MVP +matrix: `batch_selection_strategy=staleness_window`, age 1, +`mixed_weight_version_policy=allow` (default), async vLLM engine, +non-colocated generation. `make_capture_config.py` also pins +`rollout_max_attempts_to_avoid_lp_nan: 1` — capture setup hard-errors +otherwise (NaN-retry would re-register create-only rollout ids); it is set on +**both** arms' derived configs so retry behavior is not a confound. + +### Run protocol (perf reading) + +1. **Flag-off smoke first** (`MAX_NUM_STEPS=3`): proves the branch + Gym pin + reproduce the legacy SWE run before anything is compared. The standing + dormant-by-default discipline. +2. **Measurement runs**: `MAX_NUM_STEPS=20`–`30` per arm (long enough that + medians beat setup noise; short enough to iterate). `--dependency=singleton` + is already in the launcher — submit both arms back-to-back under the same + job name family so they land on comparable allocations. +3. **Repeats**: 1 pair = directional; 3 pairs = quotable (the S5 assessment: + run-to-run variance on this stack is plausibly ±5–10 %). + +## 3. Launching + +### Prerequisites (once) + +- The branch state you want measured is **committed** (RL repo and Gym + submodule) — snapshots are cut from a checkout. +- A checkout of `yukih/sc-entrypoint` (with the `tq-gate-capture` submodule + state) on lustre to snapshot from, e.g. clone + check out, or reuse an + existing sc-test workspace updated to this branch. +- A personal site wrapper (copy + `test_assets/SWE/grpo_swe_tests.sh` and edit ACCOUNT / CONTAINER / + secrets-profile / cache / W&B paths for your user). +- Snapshot it: + +```bash +cd # repo root, branch checked out +bash test_assets/SWE/sync_code.sh sc-swe-capture +``` + +### Submitting the arms + +```bash +# from this repo's swe/ folder +SITE_WRAPPER=/lustre/.../your_grpo_swe_tests.sh \ + MAX_NUM_STEPS=3 ARM=legacy bash swe/launch_swe_ab.sh # flag-off smoke + +SITE_WRAPPER=... MAX_NUM_STEPS=25 ARM=legacy bash swe/launch_swe_ab.sh +SITE_WRAPPER=... MAX_NUM_STEPS=25 ARM=capture bash swe/launch_swe_ab.sh + +# optional: HTTP byte accounting (writes per-server JSON to lustre) +SITE_WRAPPER=... MAX_NUM_STEPS=25 ARM=capture BYTES=1 bash swe/launch_swe_ab.sh +``` + +`launch_swe_ab.sh` does four things: derives the arm's config +(`make_capture_config.py` for the capture arm), stamps +`EXP_SUFFIX=swe-ab--...` so W&B separates the arms, forces the venv +posture below, and `exec`s your site wrapper (all launcher knobs still pass +through the environment: `TP`, `PPS`, `OVER_SAMPLING`, `DRY_RUN=1`, …). + +### Environment posture (why the wrapper forces these) + +- **`NRL_FORCE_REBUILD_VENVS=true` on BOTH arms.** The baked container's + `/opt/ray_venvs` predate this branch's `uv.lock` (S1 regenerated it), and + the capture arm additionally needs the `VLLM_GYM` worker venv + (`--extra vllm --extra nemo_gym`) that no image bakes yet. Forcing rebuild + on both arms keeps setup cost out of the A/B. Pre-warm `LUSTRE_UV_CACHE` + once (see the launcher header) or the first node-local build is ~30 min — + the 180-min idle-GPU reaper exemption in the site wrapper covers it. +- **`GYM_VENV_DIR=/opt/gym_venvs` (image-baked) on BOTH arms.** A node-local + `/tmp` rebuild does NOT work multi-node: the venv build runs only on the + NemoGym actor's node while Gym spawns servers cluster-wide (job 14542017 + failed on exactly this). The baked venvs are dependency-compatible with the + fork (verified: `swe_agents`/`vllm_model` requirements unchanged vs the old + pin; only the core aiohttp CVE floor moved, irrelevant at runtime), and the + editable install serves the fork's *code* on every node. If a future fork + commit does change server deps, rebake via + `test_assets/SWE/prebuild_gym_venvs.sh`. +- **Pre-warm `LUSTRE_UV_CACHE` for the branch's `uv.lock`** (single process, + inside the job container — the source builds need `nvcc`): without it the + 16-node venv rebuild exceeds the idle-GPU reaper's *de facto* 60-minute + kill (observed twice; the 180-min exemption comment is valid per the + exemption guide but not honored beyond 60 — escalate to + @job-reaper-support with jobs 14516316/14521677 if 180 is ever needed). + +## 4. What to compare (the perf read) + +All of these land in W&B (both arms) — the capture-arm `gate/*` block comes +from the per-step SC gate-metrics logging added in S5. + +**Headline perf (medians over steps 5..N, skipping warm-up):** + +| Metric (W&B key) | Expectation | +|---|---| +| `timing/train/total_step_time` | capture ≤ legacy; S5 saw −9 % single-turn | +| `timing/train/exposed_generation` | the mechanism lives here (no `/tokenize` round-trip, ~40 % smaller per-call payloads, no echo growth); S5 saw −16.8 % | +| `timing/train/valid_tokens_per_sec_per_gpu` | inverse of the above | + +**Capture-arm health / mechanism metrics:** + +| Metric | What it tells you | +|---|---| +| `gate/token_in_rate` | THE number this run finally measures. High (→1.0 for continuations) = OpenHands echoes markers faithfully and exact-prefix serving works at depth. Low = fallbacks (see next row) — runs stay *correct* but pay full re-renders. | +| `gate/fallback_no_marker` / `fallback_fingerprint_miss` / `fallback_unknown_marker` | Which § 3.3 fallback is firing if token_in_rate is low. `fingerprint_miss` at scale ⇒ the OpenHands history pipeline rewrites messages (report back — this is marker-survival risk #1 for agent frameworks). | +| `gate/capture_failed`, `train/global_valid_seqs` | staging failures → placeholder rows. valid_seqs should equal GBS on both arms. | +| `finalize/*` (rollout metrics) | finalizer latency rides the dispatch task (MVP placement) — SWE's 131k-token multi-call rollouts make this worth watching; if it dominates dispatch, that argues for the H5 finalizer pool. | + +**Training equivalence (guards the perf claim):** overlay `train/reward` +curves and `train/gen_kl_error` between arms — same band = the perf delta is +not bought with training drift. (S5: identical rewards 10/10, same KL band.) + +**Optional byte accounting (`BYTES=1`):** env-gated ASGI counters +(`NG_HTTP_BYTES_DIR` on every Gym server — the SWE config runs +`num_workers: 1`, which the counter supports — and `NRL_HTTP_BYTES_DIR` on +the vLLM workers) write per-server, per-route JSON to +`/http_bytes/`. Aggregate with `swe/aggregate_perf.py + ` → total bytes, bytes per trained token, per-hop +table. Expect the multi-turn reduction to land between the S5 floor +(−35.6 %) and beyond the prototype's −46.9 % as turn count grows. + +## 5. Known risks at this scale (accepted, watched) + +- **Gate death = silent stall, not loud failure** (S5 chaos finding, fix + queued for H1): Gym's control-plane client retries a dead gate unboundedly, + so if the policy-model server dies mid-run the job hangs rather than + failing. Supervise; `squeue` + driver log tell you which. +- **Gate buffer growth is uncapped** (H5 adds caps): per-rollout delta + forests, ids only. At seqlen 131k × concurrency 768 the raw ids are modest + (~100s of MB), but Python-object overhead is real — watch the policy-model + server RSS. +- **Marker survival through OpenHands** is the biggest unknown — it uses the + same message carrier the legacy token echo uses (so it *should* survive + wherever legacy works today), but `token_in_rate` is the proof either way, + and a low rate is a correctness-preserving perf bug, not a training bug. +- The idle-GPU reaper exemption and `checkpoint_must_save_by` are already + handled by the site wrapper / launcher. + +## 6. Files in this folder + +| File | Purpose | +|---|---| +| `SWE_RUN.md` | this runbook | +| `EXPERIMENT_LAUNCH.md` | pinned launch record for the 2026-07-28 A/B (exact commits, snapshot, gates, checklist) | +| `launch_swe_ab.sh` | arm selector: derives config, names the run, forces venv posture, delegates to your site wrapper | +| `make_capture_config.py` | derives the arm configs from the site yaml (`token_capture.enabled` + NaN-retry pin) | +| `aggregate_perf.py` | offline aggregation: per-hop HTTP bytes, bytes/trained-token, timing medians, token_in_rate | diff --git a/swe/aggregate_perf.py b/swe/aggregate_perf.py new file mode 100644 index 00000000000..c5490a60ebd --- /dev/null +++ b/swe/aggregate_perf.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python3 +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Aggregate the instrumented legacy-vs-capture perf pair. + +Usage: python aggregate_perf.py + +Each run dir: http_bytes/*.json (per-server per-route byte counters), +metrics.json (TB dump), train_rows_*.jsonl (row dump for token counts). +""" + +import glob +import json +import os +import sys + + +def load_bytes(run_dir): + per_server = {} + for path in glob.glob(os.path.join(run_dir, "http_bytes", "*.json")): + name = os.path.basename(path).rsplit("_", 1)[0] + data = json.load(open(path)) + agg = per_server.setdefault(name, {}) + for route, c in data.items(): + e = agg.setdefault(route, {"requests": 0, "req_bytes": 0, "resp_bytes": 0}) + for k in e: + e[k] += c[k] + return per_server + + +def total_trained_tokens(run_dir): + total = 0 + for path in glob.glob(os.path.join(run_dir, "train_rows_*.jsonl")): + for line in open(path): + if line.strip(): + total += json.loads(line)["input_lengths"] + return total + + +def timing(run_dir): + d = json.load(open(os.path.join(run_dir, "metrics.json"))) + out = {} + for key in ( + "timing/train/total_step_time", + "timing/train/exposed_generation", + "timing/train/policy_training", + "timing/train/weight_sync", + "timing/train/valid_tokens_per_sec_per_gpu", + "train/gen_kl_error", + "train/reward", + "gate/token_in_rate", + "gate/token_in", + "gate/fallback_no_marker", + "gate/fallback_fingerprint_miss", + ): + s = d.get(key) + if isinstance(s, dict) and s: + vals = [v for _, v in sorted(s.items(), key=lambda kv: int(kv[0]))] + out[key] = vals + return out + + +def main(): + legacy_dir, capture_dir = sys.argv[1], sys.argv[2] + report = {} + for tag, run_dir in (("legacy", legacy_dir), ("capture", capture_dir)): + servers = load_bytes(run_dir) + toks = total_trained_tokens(run_dir) + grand = {"requests": 0, "req_bytes": 0, "resp_bytes": 0} + print(f"\n===== {tag} =====") + for name, routes in sorted(servers.items()): + s = {"requests": 0, "req_bytes": 0, "resp_bytes": 0} + for route, c in routes.items(): + for k in s: + s[k] += c[k] + for k in grand: + grand[k] += s[k] + print(f"{name:32s} req={s['requests']:6d} in={s['req_bytes']/1e6:8.2f} MB out={s['resp_bytes']/1e6:8.2f} MB") + for route, c in sorted(routes.items(), key=lambda kv: -(kv[1]["req_bytes"] + kv[1]["resp_bytes"]))[:4]: + print(f" {route:40s} n={c['requests']:5d} in={c['req_bytes']/1e6:7.2f} MB out={c['resp_bytes']/1e6:7.2f} MB") + total_bytes = grand["req_bytes"] + grand["resp_bytes"] + print(f"{'TOTAL':32s} req={grand['requests']:6d} bytes={total_bytes/1e6:.2f} MB trained_tokens={toks} bytes/token={total_bytes/max(toks,1):.1f}") + report[tag] = {"total_bytes": total_bytes, "tokens": toks, "timing": timing(run_dir)} + + lt, ct = report["legacy"], report["capture"] + print("\n===== comparison =====") + bt_l = lt["total_bytes"] / max(lt["tokens"], 1) + bt_c = ct["total_bytes"] / max(ct["tokens"], 1) + print(f"HTTP bytes/trained token: legacy {bt_l:.1f} -> capture {bt_c:.1f} ({(bt_c/bt_l-1)*100:+.1f}%)") + for key in ("timing/train/total_step_time", "timing/train/exposed_generation", "timing/train/valid_tokens_per_sec_per_gpu"): + lv, cv = lt["timing"].get(key), ct["timing"].get(key) + if lv and cv: + import statistics + ml, mc = statistics.median(lv), statistics.median(cv) + print(f"{key}: legacy median {ml:.3f} -> capture median {mc:.3f} ({(mc/ml-1)*100:+.1f}%)") + tir = ct["timing"].get("gate/token_in_rate") + if tir: + print(f"capture token_in_rate (cumulative, final): {tir[-1]:.3f}") + + +if __name__ == "__main__": + main() diff --git a/swe/launch_swe_ab.sh b/swe/launch_swe_ab.sh new file mode 100755 index 00000000000..aff2fc04f88 --- /dev/null +++ b/swe/launch_swe_ab.sh @@ -0,0 +1,75 @@ +#!/bin/bash +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= +# SWE token-capture A/B arm launcher — see swe/SWE_RUN.md. +# +# Picks the arm, derives its config, names the run, forces the venv posture, +# then delegates to YOUR site wrapper (account/container/secrets live there). +# All launcher knobs (TP, PPS, MAX_NUM_STEPS, DRY_RUN=1, ...) pass through. +# +# Required: +# ARM=legacy|capture +# SITE_WRAPPER=/path/to/your grpo_swe_tests.sh-style wrapper +# Optional: +# BYTES=1 enable per-hop HTTP byte counters (JSON under the +# checkpoint dir; aggregate with swe/aggregate_perf.py) +# BASE_CONFIG site yaml to derive from (default: the wrapper's default, +# discovered via CONFIG_FILE after the wrapper sources it — +# set explicitly if your wrapper computes CONFIG_FILE late) +# ============================================================================= +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ARM="${ARM:?set ARM=legacy|capture}" +SITE_WRAPPER="${SITE_WRAPPER:?set SITE_WRAPPER=/path/to/your site wrapper}" +[[ "${ARM}" == "legacy" || "${ARM}" == "capture" ]] || { echo "ARM must be legacy|capture" >&2; exit 1; } + +# ---- Derive the arm's config from the site yaml ----------------------------- +# Default BASE_CONFIG: the yaml sitting next to the site wrapper (the +# grpo_swe_tests.sh convention). Override for other layouts. +BASE_CONFIG="${BASE_CONFIG:-$(dirname "${SITE_WRAPPER}")/grpo_qwen3_30b_async_swe.yaml}" +[ -f "${BASE_CONFIG}" ] || { echo "BASE_CONFIG not found: ${BASE_CONFIG}" >&2; exit 1; } +DERIVED_DIR="${DERIVED_DIR:-$(dirname "${BASE_CONFIG}")/derived_configs}" +mkdir -p "${DERIVED_DIR}" +export CONFIG_FILE="${DERIVED_DIR}/grpo_swe_ab_${ARM}.yaml" +python3 "${SCRIPT_DIR}/make_capture_config.py" "${BASE_CONFIG}" "${CONFIG_FILE}" "${ARM}" + +# ---- Run naming (W&B separates the arms) ------------------------------------ +export EXP_SUFFIX="${EXP_SUFFIX:-swe-ab-${ARM}-$(date +%m%d%H%M)}" + +# ---- Venv posture: identical on both arms (see SWE_RUN.md § 3) -------------- +# Baked /opt/ray_venvs predate this branch's lock; the capture arm needs the +# unbaked VLLM_GYM worker venv. Forcing rebuild on both arms keeps setup cost +# out of the A/B. +export NRL_FORCE_REBUILD_VENVS=true +# Gym venvs: use the image-baked /opt/gym_venvs (present on EVERY node). +# A node-local GYM_VENV_DIR=/tmp/... does NOT work multi-node: the venv build +# runs only on the NemoGym actor's node while Gym spawns servers cluster-wide +# (learned from job 14542017). The fork's only dep-floor change vs the baked +# venvs is the aiohttp CVE bump (verified: swe_agents/vllm_model requirements +# unchanged), and editable installs serve the fork's *code* either way. +export GYM_VENV_DIR="${GYM_VENV_DIR:-/opt/gym_venvs}" + +# ---- Optional per-hop HTTP byte accounting ---------------------------------- +if [ "${BYTES:-0}" = "1" ]; then + BYTES_DIR="${BYTES_DIR:-${CHECKPOINT_ROOT:-$(dirname "${SITE_WRAPPER}")/../..}/http_bytes/${EXP_SUFFIX}}" + mkdir -p "${BYTES_DIR}" + export NG_HTTP_BYTES_DIR="${BYTES_DIR}" + export NRL_HTTP_BYTES_DIR="${BYTES_DIR}" + echo "[launch_swe_ab] byte counters on -> ${BYTES_DIR}" +fi + +echo "[launch_swe_ab] arm=${ARM} config=${CONFIG_FILE} exp=${EXP_SUFFIX}" +exec bash "${SITE_WRAPPER}" "$@" diff --git a/swe/make_capture_config.py b/swe/make_capture_config.py new file mode 100644 index 00000000000..5880a736655 --- /dev/null +++ b/swe/make_capture_config.py @@ -0,0 +1,50 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Derive the SWE A/B arm configs from a site yaml (see swe/SWE_RUN.md). + +Usage: + python swe/make_capture_config.py legacy|capture + +Both arms get `env.nemo_gym.rollout_max_attempts_to_avoid_lp_nan: 1` (the +capture arm hard-errors without it; pinning it on both keeps NaN-retry +behavior out of the A/B). The capture arm additionally gets +`token_capture.enabled: true` — the gate config injection into the Gym +policy-model server happens in code (`environments/nemo_gym.py:_spinup`), +so no other yaml change is needed. +""" + +import sys + +import yaml + + +def main() -> None: + src, dst, arm = sys.argv[1], sys.argv[2], sys.argv[3] + assert arm in ("legacy", "capture"), f"arm must be legacy|capture, got {arm}" + with open(src) as f: + config = yaml.safe_load(f) + + config.setdefault("env", {}).setdefault("nemo_gym", {})[ + "rollout_max_attempts_to_avoid_lp_nan" + ] = 1 + if arm == "capture": + config.setdefault("token_capture", {})["enabled"] = True + + with open(dst, "w") as f: + yaml.safe_dump(config, f, sort_keys=False) + print(f"[make_capture_config] arm={arm}: {src} -> {dst}") + + +if __name__ == "__main__": + main() diff --git a/tests/functional/L1_Functional_Tests_SingleController.sh b/tests/functional/L1_Functional_Tests_SingleController.sh index 2a6fad63eea..201ac4cf02a 100755 --- a/tests/functional/L1_Functional_Tests_SingleController.sh +++ b/tests/functional/L1_Functional_Tests_SingleController.sh @@ -36,6 +36,9 @@ run_test() { run_test fast uv run --no-sync bash ./tests/functional/grpo_dp_single_controller.sh run_test fast uv run --no-sync bash ./tests/functional/grpo_async_gym_single_controller.sh +# Token-capture (gate-authoritative) path: same SC+Gym smoke with the gate +# custodying token lineage and the finalizer publishing training rows. +run_test uv run --no-sync bash ./tests/functional/grpo_async_gym_single_controller.sh ++token_capture.enabled=true cd ${PROJECT_ROOT}/tests if compgen -G ".coverage*" > /dev/null; then diff --git a/tests/unit/algorithms/test_advantage_validity.py b/tests/unit/algorithms/test_advantage_validity.py new file mode 100644 index 00000000000..64c1bc829b9 --- /dev/null +++ b/tests/unit/algorithms/test_advantage_validity.py @@ -0,0 +1,54 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""S4: validity-aware GRPO baseline (token-capture placeholder rows).""" + +import torch + +from nemo_rl.algorithms.advantage_estimator import GRPOAdvantageEstimator + + +def _estimator(**overrides) -> GRPOAdvantageEstimator: + config = {"use_leave_one_out_baseline": False, "normalize_rewards": False} + config.update(overrides) + return GRPOAdvantageEstimator(config, loss_config=None) + + +def test_invalid_rows_do_not_bias_the_baseline(): + prompt_ids = torch.zeros( + 4, 3, dtype=torch.long + ) # one shared prompt (2D, as prompt_ids_for_adv) + # The last row is a token-capture placeholder: reward 0, sample_mask 0. + rewards = torch.tensor([1.0, 3.0, 2.0, 0.0]) + valid_mask = torch.tensor([1.0, 1.0, 1.0, 0.0]) + mask = torch.ones(4, 5) + + adv = _estimator().compute_advantage( + prompt_ids, rewards, mask, valid_mask=valid_mask + ) + # Baseline over valid rows only: mean(1,3,2) = 2 (placeholder's 0 excluded). + assert torch.allclose(adv[0], torch.full((5,), -1.0)) + assert torch.allclose(adv[1], torch.full((5,), 1.0)) + assert torch.allclose(adv[2], torch.full((5,), 0.0)) + + +def test_none_valid_mask_keeps_legacy_all_valid_behavior(): + prompt_ids = torch.zeros(2, 3, dtype=torch.long) + rewards = torch.tensor([1.0, 3.0]) + mask = torch.ones(2, 3) + legacy = _estimator().compute_advantage(prompt_ids, rewards, mask) + explicit = _estimator().compute_advantage( + prompt_ids, rewards, mask, valid_mask=torch.ones(2) + ) + assert torch.equal(legacy, explicit) diff --git a/tests/unit/data_plane/test_blackbox_finalizer.py b/tests/unit/data_plane/test_blackbox_finalizer.py new file mode 100644 index 00000000000..58ecdae854c --- /dev/null +++ b/tests/unit/data_plane/test_blackbox_finalizer.py @@ -0,0 +1,247 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""S4: BlackboxFinalizer against a live TQ simple backend. + +Drives the S1 golden call sequences end to end: stage the fixture's delta +rows via TQTokenSink, hand the fixture receipt to the finalizer, and require +the published canonical rows to match the fixture's frozen training row. +Every rejection path (missing rows, digest corruption, poisoned receipts) +must yield a masked placeholder — always N rows — and the group publisher's +min/max weight versions and staging cleanup must hold. + +Marked nemo_gym (run with ``--nemo-gym-only``): the finalizer delegates +rebuild semantics to Gym's staging package. +""" + +from __future__ import annotations + +import pytest +import torch + +nemo_gym = pytest.importorskip("nemo_gym.token_id_capture.staging") + +from nemo_gym.token_id_capture.staging.conformance.kit import ( # noqa: E402 + build_fixture_artifacts, + f32, + load_fixture, +) + +from nemo_rl.data_plane.tq_token_sink import ( # noqa: E402 + STAGING_FIELDS, + TQTokenSink, +) +from nemo_rl.experience.blackbox_finalizer import BlackboxFinalizer # noqa: E402 + +pytestmark = pytest.mark.nemo_gym + +STAGING_PARTITION = "rollout_staging_fin_test" +CANONICAL_PARTITION = "rollout_data_fin_test" +PAD = 0 + + +@pytest.fixture() +def partitions(tq_client): + tq_client.register_partition( + partition_id=STAGING_PARTITION, + fields=list(STAGING_FIELDS), + num_samples=64, + consumer_tasks=["finalize"], + ) + tq_client.register_partition( + partition_id=CANONICAL_PARTITION, + fields=[ + "input_ids", + "input_lengths", + "generation_logprobs", + "token_mask", + "sample_mask", + "prompt_ids_for_adv", + "total_reward", + ], + num_samples=64, + consumer_tasks=["train"], + ) + yield + tq_client.clear_samples(sample_ids=None, partition_id=STAGING_PARTITION) + tq_client.clear_samples(sample_ids=None, partition_id=CANONICAL_PARTITION) + + +def _finalizer(tq_client, **overrides) -> BlackboxFinalizer: + kwargs = dict( + partition_id=CANONICAL_PARTITION, + staging_partition=STAGING_PARTITION, + pad_token_id=PAD, + mixed_weight_version_policy="allow", + min_valid_fraction_per_group=None, + ) + kwargs.update(overrides) + return BlackboxFinalizer(tq_client, **kwargs) + + +def _stage_fixture(tq_client, name: str, *, rollout_id: str | None = None): + """Stage one golden fixture's rows (optionally re-keyed to rollout_id) + and return (receipt_dict, expected LinearizedRow).""" + fixture = load_fixture(name) + if rollout_id is not None: + fixture = dict(fixture) + fixture["rollout_id"] = rollout_id + records, _, receipt, row = build_fixture_artifacts(fixture) + sink = TQTokenSink(tq_client, staging_partition=STAGING_PARTITION) + for record in records: + assert sink.stage(record).ok + return receipt.model_dump(), row + + +def test_finalize_rollout_reproduces_the_golden_row(tq_client, partitions): + receipt, expected = _stage_fixture(tq_client, "worked_example") + finalizer = _finalizer(tq_client) + row = finalizer.finalize_rollout("g7_r0", receipt, reward=1.0) + assert row.valid, row.rejection_reason + assert row.token_ids == expected.token_ids + assert row.token_mask == [f32(m) for m in expected.token_mask] + assert row.logprobs == [f32(p) for p in expected.logprobs] + assert row.prompt_len == expected.prompt_len + # The worked example spans a single weight version (wv 4 throughout). + assert (row.min_wv, row.max_wv) == (4, 4) + + +def test_finalize_rollout_rejections(tq_client, partitions): + finalizer = _finalizer(tq_client) + assert ( + finalizer.finalize_rollout("r", None, reward=0.0).rejection_reason + == "missing_receipt" + ) + + receipt, _ = _stage_fixture(tq_client, "single_call", rollout_id="rej_a") + poisoned = dict(receipt, capture_poisoned=True) + assert ( + finalizer.finalize_rollout("rej_a", poisoned, reward=0.0).rejection_reason + == "capture_poisoned" + ) + empty = dict(receipt, manifest=[], terminal_call_id=None) + assert ( + finalizer.finalize_rollout("rej_a", empty, reward=0.0).rejection_reason + == "empty_manifest" + ) + wrong_identity = finalizer.finalize_rollout("someone_else", receipt, reward=0.0) + assert (wrong_identity.rejection_reason or "").startswith("identity_mismatch") + + # A manifest naming rows that were never staged. + ghost = dict(receipt) + ghost["manifest"] = [ + {**entry, "staging_key": "ghost/row"} for entry in receipt["manifest"] + ] + missing = finalizer.finalize_rollout("rej_a", ghost, reward=0.0) + assert (missing.rejection_reason or "").startswith("missing_staging_row") + + # Digest corruption: break the manifest digest so recomputation misses. + corrupted = dict(receipt) + corrupted["manifest"] = [ + {**entry, "digest": "0" * 64} for entry in receipt["manifest"] + ] + bad = finalizer.finalize_rollout("rej_a", corrupted, reward=0.0) + assert (bad.rejection_reason or "").startswith("digest_mismatch") + + +def test_mixed_weight_version_policy_reject(tq_client, partitions): + receipt, _ = _stage_fixture(tq_client, "mixed_weight_versions", rollout_id="mix_r0") + receipt["rollout_id"] = "mix_r0" + allow_row = _finalizer(tq_client).finalize_rollout("mix_r0", receipt, reward=0.0) + assert allow_row.valid + assert allow_row.min_wv < allow_row.max_wv + reject_row = _finalizer( + tq_client, mixed_weight_version_policy="reject" + ).finalize_rollout("mix_r0", receipt, reward=0.0) + assert (reject_row.rejection_reason or "").startswith("mixed_weight_versions") + + +def _fetch_rows(tq_client, sample_ids): + return tq_client.get_samples( + sample_ids=sample_ids, + partition_id=CANONICAL_PARTITION, + select_fields=[ + "input_ids", + "input_lengths", + "generation_logprobs", + "token_mask", + "sample_mask", + "prompt_ids_for_adv", + "total_reward", + ], + ) + + +def test_finalize_group_publishes_n_rows_with_placeholder(tq_client, partitions): + group_id = "grp1" + receipt, expected = _stage_fixture( + tq_client, "worked_example", rollout_id=f"{group_id}_g0" + ) + receipt["rollout_id"] = f"{group_id}_g0" + rollout_ids = [f"{group_id}_g0", f"{group_id}_g1"] + + finalizer = _finalizer(tq_client) + finalized = finalizer.finalize_group( + group_id, + rollout_ids, + [receipt, None], # second rollout lost its receipt -> placeholder + [1.0, 0.0], + fallback_weight_version=9, + ) + assert not finalized.dropped + assert finalized.meta is not None + assert finalized.meta.sample_ids == rollout_ids + # Group staleness comes from the valid rollout's calls (wv 4), not the fallback. + assert (finalized.group_min_wv, finalized.group_max_wv) == (4, 4) + assert finalized.metrics["finalize/invalid_row_rate"] == 0.5 + + rows = _fetch_rows(tq_client, rollout_ids) + sample_mask = torch.as_tensor(rows["sample_mask"]).flatten() + assert sample_mask.tolist() == [1.0, 0.0] + valid_len = len(expected.token_ids) + input_ids = torch.as_tensor(rows["input_ids"][0]).flatten() + assert input_ids[:valid_len].tolist() == expected.token_ids + # Placeholder borrows the valid sibling's prompt for baseline grouping. + prompt = expected.token_ids[: expected.prompt_len] + adv_prompt_valid = torch.as_tensor(rows["prompt_ids_for_adv"][0]).flatten() + adv_prompt_placeholder = torch.as_tensor(rows["prompt_ids_for_adv"][1]).flatten() + assert adv_prompt_valid.tolist() == prompt + assert adv_prompt_placeholder.tolist() == prompt + placeholder_mask = torch.as_tensor(rows["token_mask"][1]).flatten() + assert placeholder_mask.sum().item() == 0.0 + rewards = torch.as_tensor(rows["total_reward"]).flatten() + assert rewards.tolist() == [1.0, 0.0] + + # The finalizer cleared its staged rows after publishing. + with pytest.raises(KeyError): + finalizer._source.fetch([receipt["manifest"][0]["staging_key"]]) + + +def test_finalize_group_min_valid_fraction_drops(tq_client, partitions): + group_id = "grp2" + rollout_ids = [f"{group_id}_g0", f"{group_id}_g1"] + finalizer = _finalizer(tq_client, min_valid_fraction_per_group=0.5) + finalized = finalizer.finalize_group( + group_id, + rollout_ids, + [None, None], + [0.0, 0.0], + fallback_weight_version=3, + ) + assert finalized.dropped + assert finalized.meta is None + assert (finalized.group_min_wv, finalized.group_max_wv) == (3, 3) + with pytest.raises((KeyError, RuntimeError, ValueError)): + rows = _fetch_rows(tq_client, rollout_ids) + assert not rows # nothing published diff --git a/tests/unit/data_plane/test_tq_token_sink.py b/tests/unit/data_plane/test_tq_token_sink.py new file mode 100644 index 00000000000..d543756a084 --- /dev/null +++ b/tests/unit/data_plane/test_tq_token_sink.py @@ -0,0 +1,111 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""TQTokenSink / TQTokenSource against a live TQ backend. + +Runs NeMo-Gym's installable conformance kit (golden call sequences → +byte-exact digests, manifests, and linearized rows) over the TransferQueue +implementations — the framework-CI half of the § 3.0 contract — plus the +protocol edges the kit does not cover (missing keys, stage failure shape). +""" + +from __future__ import annotations + +import pytest + +nemo_gym = pytest.importorskip("nemo_gym.token_id_capture.staging") + +from nemo_gym.token_id_capture.staging.conformance import ( # noqa: E402 + build_fixture_artifacts, + fixture_names, + load_fixture, + run_sink_source_conformance, +) +from nemo_gym.token_id_capture.staging.protocols import ( # noqa: E402 + StagingSink as TokenSinkProtocol, +) +from nemo_gym.token_id_capture.staging.protocols import ( # noqa: E402 + StagingSource as TokenSourceProtocol, +) + +from nemo_rl.data_plane.tq_token_sink import ( # noqa: E402 + STAGING_FIELDS, + TQTokenSink, + TQTokenSource, +) + +STAGING_PARTITION = "rollout_staging_test" + +pytestmark = pytest.mark.nemo_gym + + +@pytest.fixture() +def staging_partition(tq_client): + tq_client.register_partition( + partition_id=STAGING_PARTITION, + fields=list(STAGING_FIELDS), + num_samples=64, + consumer_tasks=["finalize"], + ) + yield STAGING_PARTITION + tq_client.clear_samples(sample_ids=None, partition_id=STAGING_PARTITION) + + +def test_implementations_satisfy_protocols(tq_client, staging_partition): + sink = TQTokenSink(tq_client, staging_partition=staging_partition) + source = TQTokenSource(tq_client, staging_partition=staging_partition) + assert isinstance(sink, TokenSinkProtocol) + assert isinstance(source, TokenSourceProtocol) + + +@pytest.mark.parametrize( + "fixture_name", ["worked_example", "single_call", "mixed_weight_versions"] +) +def test_tq_sink_source_passes_conformance(tq_client, staging_partition, fixture_name): + assert fixture_name in fixture_names() + sink = TQTokenSink(tq_client, staging_partition=staging_partition) + source = TQTokenSource(tq_client, staging_partition=staging_partition) + run_sink_source_conformance(load_fixture(fixture_name), sink, source) + + +def test_fetch_missing_key_raises_keyerror(tq_client, staging_partition): + source = TQTokenSource(tq_client, staging_partition=staging_partition) + with pytest.raises(KeyError): + source.fetch(["ghost_rollout/ghost_call"]) + + +def test_stage_failure_reports_not_raises(staging_partition): + class ExplodingClient: + def put_samples(self, **kwargs): + raise RuntimeError("controller down") + + sink = TQTokenSink(ExplodingClient(), staging_partition=staging_partition) + records, _, _, _ = build_fixture_artifacts(load_fixture("single_call")) + result = sink.stage(records[0]) + assert not result.ok + assert result.staging_key == records[0].staging_key + assert "controller down" in (result.error or "") + + +def test_sink_clear_drops_rows(tq_client, staging_partition): + sink = TQTokenSink(tq_client, staging_partition=staging_partition) + source = TQTokenSource(tq_client, staging_partition=staging_partition) + records, _, _, _ = build_fixture_artifacts(load_fixture("single_call")) + for record in records: + assert sink.stage(record).ok + keys = [record.staging_key for record in records] + assert len(source.fetch(keys)) == len(keys) + sink.clear(keys) + with pytest.raises(KeyError): + source.fetch(keys) diff --git a/tests/unit/experience/test_rollout_manager.py b/tests/unit/experience/test_rollout_manager.py index 30909aa856b..e80f0d3d223 100644 --- a/tests/unit/experience/test_rollout_manager.py +++ b/tests/unit/experience/test_rollout_manager.py @@ -77,6 +77,7 @@ def __init__(self) -> None: self.reserve_calls: list[int] = [] # weight_versions passed to reserve self.commit_calls: list[tuple[str, object, int, int]] = [] self.remove_calls: list[str] = [] + self.abort_calls: list[str] = [] # reserve(weight_version=X) -> group_id; commit fills the slot. self._slots: list[str] = [] @@ -86,13 +87,22 @@ def reserve( weight_version: int, target_step: int | None = None, group_id: str | None = None, + rollout_ids: list[str] | None = None, ) -> str: + del target_step, rollout_ids if group_id is None: group_id = str(uuid.uuid4()) self.reserve_calls.append(weight_version) self._slots.append(group_id) return group_id + def abort(self, group_id: str) -> bool: + self.abort_calls.append(group_id) + if group_id in self._slots: + self._slots.remove(group_id) + return True + return False + async def commit( self, group_id: str, @@ -132,6 +142,8 @@ def _make_manager(buffer: _FakeBuffer, impl: _FakeImpl) -> RolloutManager: mgr._tokenizer = None mgr._num_generations_per_prompt = 1 mgr._tq_buffer = buffer + mgr._finalizer = None + mgr._env_handles = {} mgr._weight_version = 0 return mgr @@ -294,6 +306,8 @@ async def _second_run(_sample): second_mgr._tokenizer = None second_mgr._num_generations_per_prompt = 1 second_mgr._tq_buffer = buf + second_mgr._finalizer = None + second_mgr._env_handles = {} second_mgr._weight_version = 0 async def _drive(): @@ -319,6 +333,39 @@ def test_requires_tq_buffer(self): with pytest.raises(AssertionError, match="tq_buffer"): _run(mgr.generate_and_push({"prompt": "p"})) + def test_failed_rollout_aborts_reserved_slot(self): + """A dispatch that raises must not leave a phantom unready slot.""" + + async def _boom(_input_sample): + raise RuntimeError("rollout exploded") + + buf = _FakeBuffer() + mgr = _make_manager(buf, _FakeImpl(on_run=_boom)) + + with pytest.raises(RuntimeError, match="rollout exploded"): + _run(mgr.generate_and_push({"prompt": "p"})) + + assert len(buf.reserve_calls) == 1 + assert buf.commit_calls == [] + assert len(buf.remove_calls) == 1 + assert buf._slots == [] # the reserved slot was dropped + + def test_failed_commit_aborts_reserved_slot(self): + """Commit failures (e.g. evicted slot) also abort the reservation.""" + + class _CommitBoomBuffer(_FakeBuffer): + async def commit( + self, group_id, record, start_weight_version, end_weight_version + ): + raise ValueError("no live slot") + + buf = _CommitBoomBuffer() + mgr = _make_manager(buf, _FakeImpl()) + + with pytest.raises(ValueError, match="no live slot"): + _run(mgr.generate_and_push({"prompt": "p"})) + assert len(buf.remove_calls) == 1 + # --------------------------------------------------------------------------- # Tests for RolloutManager @@ -942,3 +989,168 @@ def _last_assistant_token_ids(msg_log): assert orig_val == pytest.approx(new_val), ( f"rollout_metrics[{key!r}] mismatch — original {orig_val}, manager {new_val}" ) + + +class _FakeFinalizedGroup: + def __init__(self, *, dropped=False): + self.meta = None if dropped else "meta-sentinel" + self.group_min_wv = 3 + self.group_max_wv = 4 + self.staging_keys = [] + self.metrics = {"finalize/invalid_row_rate": 0.0} + self.dropped = dropped + + +class _FakeFinalizer: + def __init__(self, *, dropped=False): + self.calls: list[tuple] = [] + self._dropped = dropped + + def finalize_group( + self, group_id, rollout_ids, receipts, rewards, *, fallback_weight_version + ): + self.calls.append( + (group_id, rollout_ids, receipts, rewards, fallback_weight_version) + ) + return _FakeFinalizedGroup(dropped=self._dropped) + + +class _FakeCaptureBuffer(_FakeBuffer): + def __init__(self): + super().__init__() + self.reserve_rollout_ids: list[list[str] | None] = [] + self.commit_finalized_calls: list[tuple] = [] + + def reserve( + self, *, weight_version, target_step=None, group_id=None, rollout_ids=None + ): + self.reserve_rollout_ids.append(rollout_ids) + return super().reserve( + weight_version=weight_version, + target_step=target_step, + group_id=group_id, + rollout_ids=rollout_ids, + ) + + async def commit_finalized( + self, group_id, meta, group_min_wv, group_max_wv, *, staging_keys=None + ): + self.commit_finalized_calls.append( + (group_id, meta, group_min_wv, group_max_wv, staging_keys) + ) + return meta + + +class _FakeGymEnvHandle: + """NemoGym actor stand-in exposing fail_rollouts.remote.""" + + def __init__(self): + self.failed: list[tuple[list[str], str]] = [] + outer = self + + class _FailRollouts: + def remote(self, rollout_ids, reason): + outer.failed.append((list(rollout_ids), reason)) + + async def _done(): + return None + + return _done() + + self.fail_rollouts = _FailRollouts() + + +def _receipt_record(rollout_ids, receipts): + completions = [ + Completion( + message_log=[], + env_extras={"reward": 0.5, "ng_receipt": receipt, "ng_rollout_id": rid}, + truncated=False, + reward=0.5, + ) + for rid, receipt in zip(rollout_ids, receipts) + ] + return PromptGroupRecord( + prompt_idx=0, + prompt=[], + extra_env_info={}, + metadata={"task_name": "nemo_gym"}, + completions=completions, + rollout_metrics={}, + ) + + +def _make_capture_manager(buf, finalizer, *, on_run=None, num_generations=2): + mgr = object.__new__(RolloutManager) + mgr._tokenizer = None + mgr._num_generations_per_prompt = num_generations + mgr._tq_buffer = buf + mgr._finalizer = finalizer + mgr._env_handles = {"nemo_gym": _FakeGymEnvHandle()} + mgr._weight_version = 7 + + class _CaptureImpl: + def __init__(self): + self.seen_rollout_ids = None + + async def run_rollout(self, _sample, *, rollout_ids=None): + self.seen_rollout_ids = rollout_ids + if on_run is not None: + await on_run(_sample) + return _receipt_record( + rollout_ids, [{"rollout_id": rid} for rid in rollout_ids] + ) + + mgr._impl = _CaptureImpl() + return mgr + + +class TestGenerateAndFinalizeFlow: + def test_mints_ids_finalizes_and_commits(self): + buf = _FakeCaptureBuffer() + finalizer = _FakeFinalizer() + mgr = _make_capture_manager(buf, finalizer) + + _run(mgr.generate_and_push({"prompt": "p"}, target_step=5)) + + # Rollout ids were minted from the reserved group id and threaded + # end to end: reserve -> impl -> finalizer. + (group_id,) = buf._slots + expected_ids = [f"{group_id}_g0", f"{group_id}_g1"] + assert buf.reserve_rollout_ids == [expected_ids] + assert mgr._impl.seen_rollout_ids == expected_ids + (fin_group_id, fin_ids, receipts, rewards, fallback_wv) = finalizer.calls[0] + assert fin_group_id == group_id + assert fin_ids == expected_ids + assert [r["rollout_id"] for r in receipts] == expected_ids + assert rewards == [0.5, 0.5] + assert fallback_wv == 7 + # commit_finalized carried the group's min/max call versions. + assert buf.commit_finalized_calls == [(group_id, "meta-sentinel", 3, 4, [])] + # The legacy commit path was not used and nothing failed at the gate. + assert buf.commit_calls == [] + assert mgr._env_handles["nemo_gym"].failed == [] + + def test_dropped_group_aborts_slot(self): + buf = _FakeCaptureBuffer() + mgr = _make_capture_manager(buf, _FakeFinalizer(dropped=True)) + with pytest.raises(RuntimeError, match="min_valid_fraction"): + _run(mgr.generate_and_push({"prompt": "p"})) + assert buf.commit_finalized_calls == [] + assert len(buf.abort_calls) >= 1 + + def test_failed_dispatch_aborts_and_fails_gate_rollouts(self): + buf = _FakeCaptureBuffer() + + async def _boom(_sample): + raise RuntimeError("rollout exploded") + + mgr = _make_capture_manager(buf, _FakeFinalizer(), on_run=_boom) + with pytest.raises(RuntimeError, match="rollout exploded"): + _run(mgr.generate_and_push({"prompt": "p"})) + assert buf.commit_finalized_calls == [] + assert len(buf.abort_calls) == 1 + (failed_ids, reason) = mgr._env_handles["nemo_gym"].failed[0] + (group_id,) = [buf.abort_calls[0]] + assert failed_ids == [f"{group_id}_g0", f"{group_id}_g1"] + assert reason == "dispatch_failed" diff --git a/tests/unit/models/generation/test_vllm_token_capture_hosting.py b/tests/unit/models/generation/test_vllm_token_capture_hosting.py new file mode 100644 index 00000000000..65111211205 --- /dev/null +++ b/tests/unit/models/generation/test_vllm_token_capture_hosting.py @@ -0,0 +1,312 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""S2 worker hosting: install_capture wiring, fan-outs, version stamping. + +Marked nemo_gym (run with ``--nemo-gym-only``): the hosting seam imports +Gym's capture core. No engine or GPU is needed — the worker methods are +driven unbound against light fakes, and the VllmGeneration fan-outs against +a mock worker group. +""" + +from __future__ import annotations + +import asyncio +from types import SimpleNamespace +from unittest.mock import MagicMock + +import pytest + +nemo_gym = pytest.importorskip("nemo_gym.token_id_capture.staging") + +from nemo_gym.token_id_capture.staging.capture import ( # noqa: E402 + RolloutTokenCapture, +) +from nemo_gym.token_id_capture.staging.records import ( # noqa: E402 + StagedCallRecord, + StageResult, +) + +from nemo_rl.models.generation.vllm.vllm_generation import VllmGeneration # noqa: E402 +from nemo_rl.models.generation.vllm.vllm_worker_async import ( # noqa: E402 + VllmAsyncGenerationWorkerImpl, +) + +pytestmark = pytest.mark.nemo_gym + + +class _MemorySink: + def __init__(self) -> None: + self.records: list[StagedCallRecord] = [] + + def stage(self, record: StagedCallRecord) -> StageResult: + self.records.append(record) + return StageResult(ok=True, staging_key=record.staging_key) + + +def _fake_worker(*, is_model_owner: bool = True) -> SimpleNamespace: + """The attribute surface setup_token_capture touches, minus the engine.""" + worker = SimpleNamespace( + is_model_owner=is_model_owner, + token_capture=None, + _rollout_weight_version=0, + ) + worker.install_token_capture = lambda capture: setattr( + worker, "token_capture", capture + ) + return worker + + +def test_setup_token_capture_installs_capture_with_vllm_adapter(monkeypatch): + sink = _MemorySink() + monkeypatch.setattr( + "nemo_rl.data_plane.build_data_plane_client", + lambda dp_cfg, bootstrap: MagicMock(name="dp_client"), + ) + monkeypatch.setattr( + "nemo_rl.data_plane.tq_token_sink.TQTokenSink", + lambda dp_client, *, staging_partition: sink, + ) + worker = _fake_worker() + + installed = asyncio.run( + VllmAsyncGenerationWorkerImpl.setup_token_capture( + worker, dp_cfg={"backend": "simple"}, staging_partition="rollout_staging" + ) + ) + + assert installed is True + assert isinstance(worker.token_capture, RolloutTokenCapture) + assert worker.token_capture.adapter is not None + # The adapter is the vLLM one (prefix ids enter via the worker's field). + payload = worker.token_capture.adapter.enter_prefix({}, [1, 2]) + assert payload["required_prefix_token_ids"] == [1, 2] + + +def test_setup_token_capture_skips_non_model_owners(monkeypatch): + worker = _fake_worker(is_model_owner=False) + installed = asyncio.run( + VllmAsyncGenerationWorkerImpl.setup_token_capture( + worker, dp_cfg={}, staging_partition="rollout_staging" + ) + ) + assert installed is False + assert worker.token_capture is None + + +def test_weight_version_is_stamped_from_worker_state(monkeypatch): + """The install closure reads _rollout_weight_version live: a + set_rollout_weight_version between calls changes the stamp.""" + sink = _MemorySink() + monkeypatch.setattr( + "nemo_rl.data_plane.build_data_plane_client", + lambda dp_cfg, bootstrap: MagicMock(), + ) + monkeypatch.setattr( + "nemo_rl.data_plane.tq_token_sink.TQTokenSink", + lambda dp_client, *, staging_partition: sink, + ) + worker = _fake_worker() + asyncio.run( + VllmAsyncGenerationWorkerImpl.setup_token_capture( + worker, dp_cfg={}, staging_partition="rollout_staging" + ) + ) + + asyncio.run(VllmAsyncGenerationWorkerImpl.set_rollout_weight_version(worker, 4)) + first = worker.token_capture.begin_call(rollout_id="r", call_id="c1", mode="text") + asyncio.run(VllmAsyncGenerationWorkerImpl.set_rollout_weight_version(worker, 5)) + second = worker.token_capture.begin_call(rollout_id="r", call_id="c2", mode="text") + + assert (first.weight_version, second.weight_version) == (4, 5) + + coords = worker.token_capture.complete_call( + first, prompt_token_ids=[1], generated_token_ids=[2], generated_logprobs=[-0.1] + ) + assert coords.weight_version == 4 + assert sink.records[0].weight_version == 4 + + +def _generation_with_mock_group(*, async_engine: bool = True) -> VllmGeneration: + gen = object.__new__(VllmGeneration) + gen.cfg = {"vllm_cfg": {"async_engine": async_engine}} + gen.worker_group = MagicMock() + gen.worker_group.run_all_workers_single_data.return_value = [] + return gen + + +def test_generation_setup_token_capture_fans_out(monkeypatch): + gen = _generation_with_mock_group() + gen.setup_token_capture({"backend": "simple"}, "rollout_staging") + gen.worker_group.run_all_workers_single_data.assert_called_once_with( + "setup_token_capture", + dp_cfg={"backend": "simple"}, + staging_partition="rollout_staging", + run_rank_0_only_axes=["tensor_parallel", "pipeline_parallel"], + ) + + +def test_generation_setup_token_capture_requires_async_engine(): + gen = _generation_with_mock_group(async_engine=False) + with pytest.raises(AssertionError, match="async vLLM engine"): + gen.setup_token_capture({}, "rollout_staging") + + +def test_generation_set_rollout_weight_version_fans_out(): + gen = _generation_with_mock_group() + gen.set_rollout_weight_version(7) + gen.worker_group.run_all_workers_single_data.assert_called_once_with( + "set_rollout_weight_version", + version=7, + run_rank_0_only_axes=["tensor_parallel", "pipeline_parallel"], + ) + + +# --------------------------------------------------------------------------- +# S4: the request-path hookup (begin -> finish/abort around a served call) +# --------------------------------------------------------------------------- + + +class _FakeRequest(SimpleNamespace): + pass + + +def _worker_with_capture(sink: _MemorySink): + from nemo_gym.token_id_capture.adapters.vllm import VLLMCaptureAdapter + + worker = _fake_worker() + worker._capture_calls = {} + worker.token_capture = RolloutTokenCapture( + sink=sink, + weight_version_fn=lambda: worker._rollout_weight_version, + adapter=VLLMCaptureAdapter(), + ) + return worker + + +def _served_content(gen_ids, logprobs): + return { + "choices": [ + { + "index": 0, + "message": {"role": "assistant", "content": "x"}, + "logprobs": { + "content": [ + {"token": f"token_id:{t}", "logprob": lp} + for t, lp in zip(gen_ids, logprobs) + ] + }, + } + ] + } + + +def test_request_capture_round_trip_stages_and_rides_coords(): + sink = _MemorySink() + worker = _worker_with_capture(sink) + request = _FakeRequest( + ng_capture={ + "rollout_id": "r0", + "call_id": "c1", + "parent_call_id": None, + "prev_len": 0, + "mode": "text", + }, + stream=False, + ) + VllmAsyncGenerationWorkerImpl._begin_request_capture(worker, request, [10, 11, 12]) + content = _served_content([13, 14], [-0.1, -0.2]) + content = VllmAsyncGenerationWorkerImpl._finish_request_capture( + worker, request, content + ) + # Bytes were staged before the coords existed (fail-closed ordering). + assert len(sink.records) == 1 + assert sink.records[0].token_ids_delta == [10, 11, 12, 13, 14] + coords = content["ng_commit_coords"] + assert coords["disposition"] == "staged" + assert (coords["delta_len"], coords["cum_len"]) == (5, 5) + # Logprobs never transit worker -> gate; state map is drained. + assert ( + "logprobs" not in content["choices"][0] + or content["choices"][0]["logprobs"] is None + ) + assert worker._capture_calls == {} + + +def test_request_capture_token_in_prev_len_chains(): + sink = _MemorySink() + worker = _worker_with_capture(sink) + request = _FakeRequest( + ng_capture={ + "rollout_id": "r0", + "call_id": "c2", + "parent_call_id": "c1", + "prev_len": 3, + "mode": "token_in", + }, + stream=False, + ) + spliced_prompt = [10, 11, 12, 20, 21] # exact prefix + fresh suffix + VllmAsyncGenerationWorkerImpl._begin_request_capture( + worker, request, spliced_prompt + ) + content = VllmAsyncGenerationWorkerImpl._finish_request_capture( + worker, request, _served_content([22], [-0.5]) + ) + coords = content["ng_commit_coords"] + assert coords["parent_call_id"] == "c1" + assert (coords["delta_len"], coords["cum_len"]) == (3, 6) + assert sink.records[0].token_ids_delta == [20, 21, 22] + + +def test_request_capture_is_a_noop_without_context_or_capture(): + sink = _MemorySink() + worker = _worker_with_capture(sink) + plain = _FakeRequest(stream=False) # no ng_capture attribute + VllmAsyncGenerationWorkerImpl._begin_request_capture(worker, plain, [1, 2]) + content = { + "choices": [{"message": {"role": "assistant"}, "logprobs": {"content": []}}] + } + out = VllmAsyncGenerationWorkerImpl._finish_request_capture( + worker, plain, dict(content) + ) + assert "ng_commit_coords" not in out + assert out["choices"][0]["logprobs"] is not None # untouched off the capture path + assert sink.records == [] + + +def test_request_capture_abort_fails_the_call_and_drains_state(): + sink = _MemorySink() + worker = _worker_with_capture(sink) + request = _FakeRequest( + ng_capture={ + "rollout_id": "r0", + "call_id": "c1", + "parent_call_id": None, + "prev_len": 0, + "mode": "text", + }, + stream=False, + ) + VllmAsyncGenerationWorkerImpl._begin_request_capture(worker, request, [1, 2]) + VllmAsyncGenerationWorkerImpl._abort_request_capture( + worker, request, reason="engine_error" + ) + assert worker._capture_calls == {} + assert sink.records == [] + # A late finish after abort is a no-op (state already drained). + out = VllmAsyncGenerationWorkerImpl._finish_request_capture( + worker, request, _served_content([3], [-0.1]) + ) + assert "ng_commit_coords" not in out diff --git a/tests/unit/single_controller/test_single_controller.py b/tests/unit/single_controller/test_single_controller.py index 91eeb884105..343f18a1baa 100644 --- a/tests/unit/single_controller/test_single_controller.py +++ b/tests/unit/single_controller/test_single_controller.py @@ -200,7 +200,9 @@ def test_sync_weights_honors_recompute_kv_cache_config( ctrl._inflight_by_group_id = {} # env={} -> _should_use_nemo_gym is False, so _sync_weights takes the native # abort path (empty registry -> no-op) instead of the gym gate. - ctrl._master_config = SimpleNamespace(env={}) + ctrl._master_config = SimpleNamespace( + env={}, token_capture=SimpleNamespace(enabled=False) + ) asyncio.run(ctrl._sync_weights()) @@ -229,7 +231,9 @@ def test_sync_weights_calibrates_and_forwards_fp8_kv_scales() -> None: ctrl._inflight_by_group_id = {} # env={} -> _should_use_nemo_gym is False, so _sync_weights takes the native # abort path (empty registry -> no-op) instead of the gym gate. - ctrl._master_config = SimpleNamespace(env={}) + ctrl._master_config = SimpleNamespace( + env={}, token_capture=SimpleNamespace(enabled=False) + ) calibration_data = BatchedDataDict( { "input_ids": torch.tensor([[1, 2]]), diff --git a/tests/unit/single_controller/test_single_controller_setup.py b/tests/unit/single_controller/test_single_controller_setup.py index 1fb7bef41f6..df2864a1f3f 100644 --- a/tests/unit/single_controller/test_single_controller_setup.py +++ b/tests/unit/single_controller/test_single_controller_setup.py @@ -427,6 +427,7 @@ def test_nemo_gym_wires_env_handle(self, patched_factories): enable_router_replay=False, routed_experts_dtype="int16", use_fastokens=False, + token_capture=None, ) assert actor_args.env_handles["nemo_gym"] is fake_gym_actor diff --git a/tests/unit/single_controller/test_tq_replay_buffer.py b/tests/unit/single_controller/test_tq_replay_buffer.py index 3cc29fd2540..8f61e2bb462 100644 --- a/tests/unit/single_controller/test_tq_replay_buffer.py +++ b/tests/unit/single_controller/test_tq_replay_buffer.py @@ -401,3 +401,209 @@ def test_size_and_len(self): _run(buf.remove([0], remove_in_dp=True)) assert buf.size() == 1 assert len(buf) == 1 + + +class MultiPartitionFakeDataPlaneClient(FakeDataPlaneClient): + """Fake DP client that tracks rows per partition (token-capture mode).""" + + def __init__(self) -> None: + super().__init__(partition_id="rollout_data") + self.rows_by_partition: dict[str, dict[str, Any]] = {} + self.clear_calls_by_partition: list[tuple[str, list[str]]] = [] + + def put_samples(self, sample_ids, partition_id, fields=None, tags=None): + bucket = self.rows_by_partition.setdefault(partition_id, {}) + for i, sid in enumerate(sample_ids): + bucket[sid] = {"tag": dict(tags[i]) if tags is not None else {}} + return KVBatchMeta( + partition_id=partition_id, + task_name=None, + sample_ids=list(sample_ids), + fields=None, + tags=[dict(t) for t in tags] if tags is not None else None, + ) + + def clear_samples(self, sample_ids, partition_id): + ids = list(sample_ids) if sample_ids is not None else [] + self.clear_calls_by_partition.append((partition_id, ids)) + bucket = self.rows_by_partition.setdefault(partition_id, {}) + for sid in ids: + bucket.pop(sid, None) + + +class TestTQReplayBufferTokenCaptureMode: + """commit_finalized / abort / rollout_ids / staging-aware remove. + + All of these are uncalled on the legacy (token_capture.enabled=false) + path; the existing test classes above are the legacy-invariance guard. + """ + + def _make_capture_buffer(self, dp) -> TQReplayBuffer: + return TQReplayBuffer( + dp, + partition_id="rollout_data", + pad_value_dict={"token_ids": 0}, + staging_partition_id="rollout_staging", + ) + + def test_reserve_records_rollout_ids(self): + buf = self._make_capture_buffer(MultiPartitionFakeDataPlaneClient()) + buf.reserve(weight_version=1, rollout_ids=["g0_g0", "g0_g1"]) + assert buf._rollout_ids_list == [["g0_g0", "g0_g1"]] + # Legacy reserve records None. + buf.reserve(weight_version=1) + assert buf._rollout_ids_list[1] is None + + def test_commit_finalized_fills_slot_with_group_min_wv(self): + dp = MultiPartitionFakeDataPlaneClient() + buf = self._make_capture_buffer(dp) + group_id = buf.reserve(weight_version=4, rollout_ids=["r0", "r1"]) + # The finalizer published its own rows; commit_finalized only fills the slot. + meta = KVBatchMeta( + partition_id="rollout_data", + task_name=None, + sample_ids=[f"{group_id}_g0", f"{group_id}_g1"], + fields=None, + ) + _run( + buf.commit_finalized( + group_id, + meta, + group_min_wv=3, + group_max_wv=5, + staging_keys=["r0/c1", "r0/c2", "r1/c1"], + ) + ) + assert buf.ready_list == [True] + assert buf.start_weight_list == [3] # oldest call version, not reserve-time 4 + assert buf.end_weight_list == [5] + assert buf.meta_list[0] is meta + assert buf._staging_keys_list == [["r0/c1", "r0/c2", "r1/c1"]] + # No tensorize/put happened here. + assert dp.rows_by_partition.get("rollout_data") is None + + def test_commit_finalized_raises_for_evicted_slot(self): + buf = self._make_capture_buffer(MultiPartitionFakeDataPlaneClient()) + meta = KVBatchMeta( + partition_id="rollout_data", task_name=None, sample_ids=[], fields=None + ) + with pytest.raises(ValueError, match="no live slot"): + _run(buf.commit_finalized("ghost", meta, group_min_wv=0, group_max_wv=0)) + + def test_abort_drops_unready_slot_only(self): + dp = MultiPartitionFakeDataPlaneClient() + buf = self._make_capture_buffer(dp) + gid_unready = buf.reserve(weight_version=1) + gid_ready = buf.reserve(weight_version=1) + _run( + buf.commit( + gid_ready, + _make_record(), + start_weight_version=1, + end_weight_version=1, + ) + ) + assert buf.abort(gid_unready) is True + assert buf.size() == 1 + # Ready slots and unknown ids are not abortable. + assert buf.abort(gid_ready) is False + assert buf.abort("ghost") is False + assert buf.size() == 1 + + def test_remove_clears_staging_rows_alongside_canonical(self): + dp = MultiPartitionFakeDataPlaneClient() + buf = self._make_capture_buffer(dp) + group_id = buf.reserve(weight_version=1, rollout_ids=["r0"]) + meta = KVBatchMeta( + partition_id="rollout_data", + task_name=None, + sample_ids=[f"{group_id}_g0"], + fields=None, + ) + _run( + buf.commit_finalized( + group_id, + meta, + group_min_wv=1, + group_max_wv=1, + staging_keys=["r0/c1", "r0/c2"], + ) + ) + n = _run(buf.remove([0], remove_in_dp=True)) + assert n == 1 + assert ("rollout_data", [f"{group_id}_g0"]) in dp.clear_calls_by_partition + assert ("rollout_staging", ["r0/c1", "r0/c2"]) in dp.clear_calls_by_partition + + def test_remove_without_staging_partition_skips_staging_clear(self): + dp = MultiPartitionFakeDataPlaneClient() + buf = TQReplayBuffer( + dp, partition_id="rollout_data", pad_value_dict={"token_ids": 0} + ) + group_id = buf.reserve(weight_version=1) + meta = KVBatchMeta( + partition_id="rollout_data", + task_name=None, + sample_ids=[f"{group_id}_g0"], + fields=None, + ) + _run(buf.commit_finalized(group_id, meta, group_min_wv=1, group_max_wv=1)) + _run(buf.remove([0], remove_in_dp=True)) + partitions_cleared = {p for p, _ in dp.clear_calls_by_partition} + assert partitions_cleared == {"rollout_data"} + + +class TestTQReplayBufferEvictedCommit: + def test_commit_on_evicted_slot_writes_nothing(self): + """The pre-write check: an evicted group must not orphan rows.""" + dp = FakeDataPlaneClient() + buf = _make_buffer(dp) + group_id = buf.reserve(weight_version=1) + _run(buf.remove([0], remove_in_dp=False)) + + with pytest.raises(ValueError, match="no live slot"): + _run( + buf.commit( + group_id, + _make_record(), + start_weight_version=1, + end_weight_version=1, + ) + ) + assert dp.put_calls == [] + assert dp.depth() == 0 + + def test_commit_evicted_during_write_unwrites_rows(self): + """Eviction interleaving with the awaited put must clear the rows.""" + + class EvictDuringPut(FakeDataPlaneClient): + def __init__(self): + super().__init__() + self.buf: TQReplayBuffer | None = None + + async def put_samples( + self, sample_ids, partition_id, fields=None, tags=None + ): + result = FakeDataPlaneClient.put_samples( + self, sample_ids, partition_id, fields=fields, tags=tags + ) + # Simulate the sampler evicting the slot mid-write. + await self.buf.remove([0], remove_in_dp=False) + return result + + dp = EvictDuringPut() + buf = _make_buffer(dp) + dp.buf = buf + group_id = buf.reserve(weight_version=1) + + with pytest.raises(ValueError, match="evicted during"): + _run( + buf.commit( + group_id, + _make_record(), + start_weight_version=1, + end_weight_version=1, + ) + ) + # The written rows were un-written. + assert dp.depth() == 0 + assert len(dp.clear_calls) == 1 From 8651ace39f913557ac61e82220fc8156830bd0ea Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Sun, 2 Aug 2026 00:21:55 -0700 Subject: [PATCH 04/52] feat(swe): capture-arm nano launcher with the full capture posture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Encodes everything the capture leg needed by hand on job 5764598: hydra +append for the yaml-less token_capture key, the lp-NaN pin, driver-side nemo_gym PYTHONPATH + orjson (via new NRL_DRIVER_PYTHONPATH / NRL_DRIVER_PIP_INSTALL launcher knobs), and a spec-correct VLLM_GYM worker venv rebuild (venv caching is not spec-aware — PR #3456 known issue). Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre --- swe_nano_sc_capture_interactive.sh | 49 ++++++++++++++++++++++++++++++ ultra_launch.sh | 2 ++ 2 files changed, 51 insertions(+) create mode 100755 swe_nano_sc_capture_interactive.sh diff --git a/swe_nano_sc_capture_interactive.sh b/swe_nano_sc_capture_interactive.sh new file mode 100755 index 00000000000..1c639e537cd --- /dev/null +++ b/swe_nano_sc_capture_interactive.sh @@ -0,0 +1,49 @@ +#!/bin/bash +# ============================================================================= +# swe_nano_sc_capture_interactive.sh — the CAPTURE arm of the nano SWE TQ A/B: +# swe_nano_sc_interactive.sh + gate-authoritative token capture enabled. +# +# Same 6-node SingleController shape as swe_nano_sc_interactive.sh; adds the +# posture the capture path needs (learned on job 5764598, 2026-08-01): +# +token_capture.enabled=true hydra APPEND — no yaml in the nano chain +# defines the key (pydantic default only) +# +env.nemo_gym.rollout_max_attempts_to_avoid_lp_nan=1 +# capture hard-errors otherwise; pin on the +# legacy arm too when running an A/B pair +# NRL_DRIVER_PYTHONPATH driver imports nemo_gym staging records; +# the baked driver venv has no nemo_gym +# NRL_DRIVER_PIP_INSTALL=orjson Gym's token_id_capture/__init__ eagerly +# imports consumer->store->orjson (purity +# gap; PR #2278 feedback) +# VllmAsyncGenerationWorker in NRL_FORCE_REBUILD_VENVS_LIST +# the capture leg needs the VLLM_GYM venv +# (--extra nemo_gym); venv caching is not +# spec-aware (PR #3456 known issue), so a +# legacy-leg venv would be silently reused +# +# Usage (same contract as the other nano launchers): +# bash swe_nano_sc_capture_interactive.sh [hydra overrides...] +# ============================================================================= +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +set -a +# shellcheck disable=SC1091 +source "${HERE}/swe_nano.env" + +EXP_NAME="${SC_EXP_NAME:-nano-swe-sc-tq-capture}" +NRL_ENTRYPOINT="${CODE_DIR}/examples/run_grpo_single_controller.py" +CONFIG_PATH="${CODE_DIR}/examples/configs/ultra/nano_swe_teacher_sc.yaml" +RESULTS_DIR="${WORKSPACE_DIR}/results/${EXP_NAME}" +BASE_LOG_DIR="${WORKSPACE_DIR}/ray_logs/${EXP_NAME}" + +NRL_FORCE_REBUILD_VENVS_LIST="nemo_rl.environments.nemo_gym.NemoGym,nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker" +NRL_DRIVER_PYTHONPATH="/opt/nemo-rl/3rdparty/Gym-workspace/Gym" +NRL_DRIVER_PIP_INSTALL="orjson" +set +a + +INTERACTIVE=1 DRY_RUN=0 INTERACTIVE_WAIT="${INTERACTIVE_WAIT:-1}" \ + bash "${HERE}/ultra_launch.sh" \ + +token_capture.enabled=true \ + +env.nemo_gym.rollout_max_attempts_to_avoid_lp_nan=1 \ + "$@" diff --git a/ultra_launch.sh b/ultra_launch.sh index b37b2180bae..bf9c5a3d819 100755 --- a/ultra_launch.sh +++ b/ultra_launch.sh @@ -677,8 +677,10 @@ export SETUP_COMMAND # per-run overrides: cluster shape, paths, judge endpoints, logging. # ============================================================================= TRAIN_CMD="cd ${CODE_ROOT} && date ; \ +${NRL_DRIVER_PIP_INSTALL:+uv pip install --python /opt/nemo_rl_venv/bin/python ${NRL_DRIVER_PIP_INSTALL} ; }\ ${GENRM_RUNTIME_SETUP}\ ${VLLM_ENV_SOURCE}\ +${NRL_DRIVER_PYTHONPATH:+PYTHONPATH=${NRL_DRIVER_PYTHONPATH} }\ OMP_NUM_THREADS=16 \ RAY_DEDUP_LOGS=1 \ WANDB_INIT_TIMEOUT=300 \ From b9ee7f74e2dda7076d381938c1f77f495b28d2c2 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Tue, 4 Aug 2026 22:39:02 -0700 Subject: [PATCH 05/52] feat(swe): batch capture launcher (swe_nano_sc_capture.sh) Batch counterpart of swe_nano_sc_capture_interactive.sh; capture env block survives swe_nano.env sourcing (which unconditionally overwrites NRL_FORCE_REBUILD_VENVS_LIST). Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre --- swe_nano_sc_capture.sh | 48 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 swe_nano_sc_capture.sh diff --git a/swe_nano_sc_capture.sh b/swe_nano_sc_capture.sh new file mode 100644 index 00000000000..53d3a0ba069 --- /dev/null +++ b/swe_nano_sc_capture.sh @@ -0,0 +1,48 @@ +#!/bin/bash +# ============================================================================= +# swe_nano_sc_capture.sh — BATCH launch of the nano SWE SingleController + +# TransferQueue recipe with gate-authoritative token capture enabled. +# +# Batch counterpart of swe_nano_sc_capture_interactive.sh, exactly as +# swe_nano_sc.sh is the batch counterpart of swe_nano_sc_interactive.sh: +# same capture env block (driver PYTHONPATH + orjson, vllm worker venv +# rebuild) and the same two hydra appends; ray.sub runs the driver directly. +# +# Run from a NETWORKED shell at the repo root: +# DRY_RUN=0 SC_EXP_NAME= bash swe_nano_sc_capture.sh [extra overrides] +# Capture-canonical fingerprints: export NG_TIC_FP_CANONICAL=1 (recommended). +# ============================================================================= +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +_DRY_RUN_IN="${DRY_RUN:-}" +_WALLTIME_IN="${WALLTIME:-}" + +set -a +# shellcheck disable=SC1091 +source "${HERE}/swe_nano.env" + +EXP_NAME="${SC_EXP_NAME:-nano-swe-sc-tq-capture}" +NRL_ENTRYPOINT="${CODE_DIR}/examples/run_grpo_single_controller.py" +CONFIG_PATH="${CODE_DIR}/examples/configs/ultra/nano_swe_teacher_sc.yaml" +RESULTS_DIR="${WORKSPACE_DIR}/results/${EXP_NAME}" +BASE_LOG_DIR="${WORKSPACE_DIR}/ray_logs/${EXP_NAME}" + +NRL_FORCE_REBUILD_VENVS_LIST="nemo_rl.environments.nemo_gym.NemoGym,nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker" +NRL_DRIVER_PYTHONPATH="/opt/nemo-rl/3rdparty/Gym-workspace/Gym" +NRL_DRIVER_PIP_INSTALL="orjson" + +# --- Per-call latency breakdown (CALL_TIMING=0 to disable) -------------------- +if [ "${CALL_TIMING:-1}" = "1" ]; then + NRL_CALL_TIMING_DIR="${NRL_CALL_TIMING_DIR:-${WORKSPACE_DIR}/call_timing/${EXP_NAME}}" + NG_CALL_TIMING_DIR="${NG_CALL_TIMING_DIR:-${NRL_CALL_TIMING_DIR}}" + mkdir -p "${NRL_CALL_TIMING_DIR}" +fi +[ -n "${_DRY_RUN_IN}" ] && DRY_RUN="${_DRY_RUN_IN}" +[ -n "${_WALLTIME_IN}" ] && WALLTIME="${_WALLTIME_IN}" +set +a + +bash "${HERE}/ultra_launch.sh" \ + +token_capture.enabled=true \ + +env.nemo_gym.rollout_max_attempts_to_avoid_lp_nan=1 \ + "$@" From a2a37bd4e8e1b7fbb2c8b983ff869578d23edb54 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 5 Aug 2026 08:21:12 -0700 Subject: [PATCH 06/52] docs(swe): token-capture recipe guide (nano SWE) Companion to nano-swe-transferqueue.md: launch posture for the gate-authoritative token-capture arm, why each capture env knob exists, the data-flow, engagement checks (token_in_rate, empty_manifest, staging PUT_DATA), measured A/B results, and known limits. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre --- docs/guides/nano-swe-token-capture.md | 173 ++++++++++++++++++++++++++ docs/index.md | 8 ++ 2 files changed, 181 insertions(+) create mode 100644 docs/guides/nano-swe-token-capture.md diff --git a/docs/guides/nano-swe-token-capture.md b/docs/guides/nano-swe-token-capture.md new file mode 100644 index 00000000000..dbbaa975c97 --- /dev/null +++ b/docs/guides/nano-swe-token-capture.md @@ -0,0 +1,173 @@ +# Nano SWE RL with Gate-Authoritative Token Capture + +A reproducible 6-node recipe that runs agentic SWE RL on +Nemotron-3-Nano-30B-A3B with **exact-token capture**: the vLLM worker stages +each model call's token delta durably into the TransferQueue data plane, the +Gym gate serves verified prefix token ids back on every follow-up call +(token-in), and the trainer consumes rows rebuilt from staged deltas — no +token echo over HTTP, no re-tokenization of agent history. + +It builds directly on the [Nano SWE TransferQueue +recipe](nano-swe-transferqueue.md); read that first for the cluster shape, +`swe_nano.env` setup, and SingleController constraints. This guide covers only +what capture adds. Code under test: RL PR #3456 + Gym PR #2278 (plus the +attribution and canonical-fingerprint fixes staged for #2278). + +## Verified result + +Measured across four legacy-vs-capture A/B pairs on 6× GB200 nodes +(15-step pair: jobs 5845431/5845433; three 12-step pairs: 5860015-22): + +| | legacy (token echo) | capture (token-in) | +|---|---|---| +| Total step time (3 pairs, steps 1-12) | 7,582 ± 292 s | 7,590 ± 220 s (**+0.1 %**) | +| exposed_generation | 5,896 ± 291 s | 5,884 ± 229 s (−0.2 %) | +| HTTP bytes / 15-step run | 69.96 GB | 21.11 GB (**−70 %**) | +| worker preprocess CPU | 17-19 ks (2 renders/call) | 8-10 ks (1/call) | +| gate event-loop lag p99 | 22 ms | 4-5 ms | +| token_in_rate | n/a | **0.9999** | +| token_mult_prob_error max (3 runs) | 121 / 1.8e9 / 301 | **1.13 / 1.13 / 2.87** | +| capture staging cost | — | ~39 ms/call (tq_put p50 38 ms; 0.8 % of engine) | + +Wall-clock parity, −70 % bytes, half the worker CPU, and — decisively — +clean training rows: the legacy path's re-tokenization drift produces +pathological probability-ratio errors at 20k+-token contexts; exact-token +rows do not. + +## Quick start + +Batch, one command from a networked shell at the repo root: + +```bash +DRY_RUN=0 SC_EXP_NAME=my-capture-run NG_TIC_FP_CANONICAL=1 \ + WALLTIME=3:59:00 bash swe_nano_sc_capture.sh +``` + +Interactive (allocate once, iterate on the driver by hand — see the +TransferQueue guide for the attach/run-cmd workflow): + +```bash +NG_TIC_FP_CANONICAL=1 bash swe_nano_sc_capture_interactive.sh +``` + +The legacy comparison arm is the plain `swe_nano_sc.sh` / +`swe_nano_sc_interactive.sh` from the TransferQueue recipe. Batch and +interactive build byte-for-byte the same driver command. + +Two flags matter: + +- `NG_TIC_FP_CANONICAL=1` — canonical text fingerprints in the gate. + **Without it token-in silently degrades to ~0**: reasoning models echo + history with `` blocks stripped, the gate's fingerprint of the served + turn never matches, every call falls back to text mode, and the run develops + a +38 % generated-token divergence. With it, `token_in_rate ≈ 0.9999`. +- `WALLTIME` must be a Slurm time string (`3:59:00`). `3h` fails with + `sbatch: error: Invalid --time specification` printed at the very *end* of + the launch output — easy to miss. Budget ~1 h 40 of setup (venv rebuild + + checkpoint load) before the first step. + +## What the capture launcher adds, and why + +Every line of the capture posture in `swe_nano_sc_capture.sh` exists because +its absence crashed a run: + +| Setting | Without it | +|---|---| +| `+token_capture.enabled=true` (hydra append) | Capture never engages. It is a `+` append because no yaml in the nano chain defines the key; a plain override is rejected. | +| `NRL_DRIVER_PYTHONPATH=/opt/nemo-rl/3rdparty/Gym-workspace/Gym` | Driver `ModuleNotFoundError: nemo_gym` — the driver imports the staging record schema, and the baked driver venv has no nemo_gym. | +| `NRL_DRIVER_PIP_INSTALL=orjson` | Driver `ModuleNotFoundError: orjson` — Gym's `token_id_capture/__init__` eagerly imports the store. | +| `VllmAsyncGenerationWorker` in `NRL_FORCE_REBUILD_VENVS_LIST` | Worker `ModuleNotFoundError: orjson` — venv caching is spec-unaware and silently reuses the legacy arm's venv (RL #3456 known issue). | +| capture env set *after* sourcing `swe_nano.env` | `swe_nano.env` exports `NRL_FORCE_REBUILD_VENVS_LIST` unconditionally and clobbers an env-prefix value. | + +## How the data flows + +``` +agent (nv-OpenHands) gate (vllm_model, gate mode) + /run body carries ng_rollout_id ───► registers rollout, admits calls + fingerprints history → resolves + parent → sends exact prefix ids + │ required_prefix_token_ids + ▼ +vLLM worker: splices prefix verbatim, renders only the new tail, + generates, then STAGES the delta (ids+mask+logprobs) to TransferQueue + (synchronous tq_put — durable before the call is acked) and returns + token-light CommitCoords on the response + │ coords (≈4 B/token) + ▼ +gate ingests coords into lineage; at /run end returns a token-free +RolloutReceipt (manifest of call_ids + staging keys) to the trainer + ▼ +finalizer: fetches staged deltas by key, digest-verifies, rebuilds the +exact training row, clears the staged rows +``` + +The heavy bytes (token arrays, logprobs) move exactly once, worker→TQ, +node-locally. The gate and the `/run` response stay token-light. + +## Verifying capture is really engaged + +Config echo is not evidence. Check, in order: + +1. **Gate metrics in the SC step output** (grade from the SC worker + `.out` under `-logs/ray/session_*/logs/worker-*-.out` — the + driver log's actor-stdout forwarding is not reliable): + + ``` + gate_metrics={'registered': 128.0, 'token_in': 7042.0, + 'fallback_no_match': 0.0, 'capture_failed': 0.0, ...} + ``` + + `token_in / (token_in + fallback_*)` should be ≥ 0.99. A rate near 0 with + large `fallback_no_match` means canonical fingerprints are off (see above). + `unattributed_calls` > 0 means the harness is not forwarding + `ng_rollout_id` (attribution fix missing). + +2. **No finalize rejections.** `finalize: ... rejected (empty_manifest)` on + every rollout means calls are reaching the worker without gate context — + the run generates but nothing is trainable. + +3. **TQ staging traffic**: `PUT_DATA` on the staging partition per model call + (tens of thousands per run), not just per training batch. + +4. **Training equivalence**: `token_mult_prob_error` should sit near 1.0 + (max ≲ 3). `gen_kl_error` ~0.004 matches the legacy arm. + +## Perf instrumentation (optional) + +`CALL_TIMING=1` (default in these launchers) sets `NRL_CALL_TIMING_DIR` / +`NG_CALL_TIMING_DIR` and emits per-call JSONL segments from every server: +worker engine/preprocess/splice, `tq_put`, gate prepare/ingest, per-path HTTP +timings, event-loop lag. Aggregate with `swe/aggregate_call_timing.py`. +`cached_tokens` on each `worker_engine` record reports vLLM prefix-cache hits +(~94 % of prompt tokens on this workload). All probes are env-gated and +dormant unless the dir is set. + +## Known limits + +- **Grade runs from the SC worker `.out` or W&B, never the driver log** — + Ray's driver-log stdout forwarding dropped entire actors in testing (runs + looked stalled while training normally). +- **Receipt-mode W&B rollout metrics are not yet comparable**: + `gen_tokens_per_sample` counts the carried prompt tail and + `truncation_rate` is constant on the capture arm. Compare token counts via + the call-timing JSONL (engine `usage`) instead. +- **`global_valid_toks` runs ~15 % lower on capture** at equal work — + masking-semantics difference under investigation; do not quote + convergence-per-wallclock until resolved. +- **Router replay (R3)**: capture stages routed experts as TQ extras beside + the token delta (`routed_experts_delta`), keeping the multi-MB arrays off + HTTP — but vLLM 0.20.0's engine crashes with + `enable_return_routed_experts=true` on this model (`IndexError` in + `sample_tokens`); blocked on an engine fix. +- **Weight-version mixing** across spliced chains at a refit boundary is + guarded conservatively; a per-row `weight_version` check in the finalizer + is the hardening item. + +## Related + +- [Nano SWE with TransferQueue](nano-swe-transferqueue.md) — base recipe, + cluster shape, SingleController constraints +- [Router Replay](router-replay.md) — R3 background and trainer-side replay +- `nemo_rl/data_plane/tq_token_sink.py` — the staging sink/source over TQ +- `nemo_rl/experience/blackbox_finalizer.py` — receipt → training row +- Gym `nemo_gym/token_id_capture/staging/records.py` — the wire schema diff --git a/docs/index.md b/docs/index.md index ff0d1ad1718..257949f12f8 100644 --- a/docs/index.md +++ b/docs/index.md @@ -149,6 +149,13 @@ Train Qwen3-30B-A3B-Thinking into a SWE agent with a pivot stage plus end-to-end Six-node SWE RL smoke on Nemotron-3-Nano-30B-A3B with rollouts flowing through the TransferQueue data plane via SingleController. ::: +:::{grid-item-card} {octicon}`shield-check` Nano SWE with Token Capture +:link: guides/nano-swe-token-capture +:link-type: doc + +Exact-token SWE RL: the worker stages each call's token delta into TransferQueue and the gate serves verified prefixes back — no token echo, no re-tokenization. +::: + :::{grid-item-card} {octicon}`plus-circle` Adding New Models :link: adding-new-models :link-type: doc @@ -305,6 +312,7 @@ guides/cispo.md guides/prorlv2.md guides/swe-rl-qwen3.md guides/nano-swe-transferqueue.md +guides/nano-swe-token-capture.md guides/grpo.md guides/ppo.md guides/grpo-deepscaler.md From 09312c5cfc2a3e486066d1c26a73443852543bf2 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Thu, 6 Aug 2026 13:49:22 -0700 Subject: [PATCH 07/52] docs(swe): restructure token-capture guide as a run recipe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Match nano-swe-transferqueue.md's shape: how to trigger a capture run and a holistic view of the implementation (data flow + component map). Experiment/perf material moved out — it lives in the campaign reports, not the recipe. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre --- docs/guides/nano-swe-token-capture.md | 208 +++++++++++++------------- 1 file changed, 102 insertions(+), 106 deletions(-) diff --git a/docs/guides/nano-swe-token-capture.md b/docs/guides/nano-swe-token-capture.md index dbbaa975c97..974f48a8bed 100644 --- a/docs/guides/nano-swe-token-capture.md +++ b/docs/guides/nano-swe-token-capture.md @@ -1,38 +1,75 @@ # Nano SWE RL with Gate-Authoritative Token Capture A reproducible 6-node recipe that runs agentic SWE RL on -Nemotron-3-Nano-30B-A3B with **exact-token capture**: the vLLM worker stages -each model call's token delta durably into the TransferQueue data plane, the -Gym gate serves verified prefix token ids back on every follow-up call -(token-in), and the trainer consumes rows rebuilt from staged deltas — no -token echo over HTTP, no re-tokenization of agent history. +Nemotron-3-Nano-30B-A3B with **exact-token capture** enabled: the vLLM worker +stages each model call's token delta durably into the TransferQueue data +plane, the Gym gate serves verified prefix token ids back on every follow-up +call (token-in), and the trainer consumes rows rebuilt from the staged deltas. +No token echo over HTTP, no re-tokenization of agent history — the tokens the +engine sampled are byte-for-byte the tokens the trainer sees. It builds directly on the [Nano SWE TransferQueue recipe](nano-swe-transferqueue.md); read that first for the cluster shape, -`swe_nano.env` setup, and SingleController constraints. This guide covers only -what capture adds. Code under test: RL PR #3456 + Gym PR #2278 (plus the -attribution and canonical-fingerprint fixes staged for #2278). +`swe_nano.env` setup, and the SingleController constraints. This guide covers +only what token capture adds. ## Verified result -Measured across four legacy-vs-capture A/B pairs on 6× GB200 nodes -(15-step pair: jobs 5845431/5845433; three 12-step pairs: 5860015-22): - -| | legacy (token echo) | capture (token-in) | -|---|---|---| -| Total step time (3 pairs, steps 1-12) | 7,582 ± 292 s | 7,590 ± 220 s (**+0.1 %**) | -| exposed_generation | 5,896 ± 291 s | 5,884 ± 229 s (−0.2 %) | -| HTTP bytes / 15-step run | 69.96 GB | 21.11 GB (**−70 %**) | -| worker preprocess CPU | 17-19 ks (2 renders/call) | 8-10 ks (1/call) | -| gate event-loop lag p99 | 22 ms | 4-5 ms | -| token_in_rate | n/a | **0.9999** | -| token_mult_prob_error max (3 runs) | 121 / 1.8e9 / 301 | **1.13 / 1.13 / 2.87** | -| capture staging cost | — | ~39 ms/call (tq_put p50 38 ms; 0.8 % of engine) | - -Wall-clock parity, −70 % bytes, half the worker CPU, and — decisively — -clean training rows: the legacy path's re-tokenization drift produces -pathological probability-ratio errors at 20k+-token contexts; exact-token -rows do not. +Run on 6 GB200 NVL72 nodes: + +| | | +|---|---| +| Entrypoint | `examples/run_grpo_single_controller.py` | +| Config | `examples/configs/ultra/nano_swe_teacher_sc.yaml` | +| Model | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` | +| Shape | train 4 nodes (TP2·PP2·CP4, EP1) / gen 2 nodes (vLLM TP2 → 4 engines) | +| Capture flags | `+token_capture.enabled=true` (hydra append), `NG_TIC_FP_CANONICAL=1` | +| Outcome | 15/15 steps, `token_in_rate ≈ 0.9999`, zero `empty_manifest`, zero `capture_failed`, `valid_seqs = GBS` | +| Launched via | `swe_nano_sc_capture.sh` (batch) / `swe_nano_sc_capture_interactive.sh` (attach + iterate) | + +## How it works + +Legacy SWE runs recover training tokens by echoing token ids in every model +response and re-tokenizing the agent's rendered history each turn. Both are +lossy for a reasoning model — the chat template strips `` content from +history, and re-tokenizing an assistant turn can split differently than the +tokens the model actually sampled. Token capture replaces that path: + +``` +agent (nv-OpenHands) gate (vllm_model, gate mode) + /run body carries ng_rollout_id ───► registers the rollout, admits calls, + fingerprints incoming history → + resolves the parent call → sends the + parent's exact prefix token ids + │ required_prefix_token_ids + ▼ +vLLM worker: splices the prefix verbatim, renders only the new tail, + generates, then STAGES the call's token delta (ids + mask + logprobs) + to TransferQueue — a synchronous tq_put, durable before the call is + acked — and returns token-light CommitCoords on the response + │ coords (≈4 B/token) + ▼ +gate ingests coords into its lineage index; when the rollout ends it +returns a token-free RolloutReceipt (a manifest of call_ids and staging +keys) on the /run response + ▼ +finalizer (trainer side): fetches the staged deltas by key, verifies +digests, rebuilds the exact training row, publishes it, clears staging +``` + +The heavy bytes (token arrays, logprobs) move exactly once, worker→TQ, +node-locally. The gate hop and the `/run` response stay token-light. + +The pieces, by repo: + +| Component | Where | +|---|---| +| Gate mode, prefix serving, lineage | Gym `responses_api_models/vllm_model/app.py` + `nemo_gym/token_id_capture/gate.py` | +| Rollout attribution (`ng_rollout_id`) | Gym capture middleware + `swe_agents` harness forwarding | +| Wire schema (deltas, coords, receipts) | Gym `nemo_gym/token_id_capture/staging/records.py` | +| Worker-side capture + prefix splice | `nemo_rl/models/generation/vllm/vllm_worker_async.py` | +| Staging sink/source over TransferQueue | `nemo_rl/data_plane/tq_token_sink.py` | +| Receipt → training row | `nemo_rl/experience/blackbox_finalizer.py` | ## Quick start @@ -43,74 +80,47 @@ DRY_RUN=0 SC_EXP_NAME=my-capture-run NG_TIC_FP_CANONICAL=1 \ WALLTIME=3:59:00 bash swe_nano_sc_capture.sh ``` -Interactive (allocate once, iterate on the driver by hand — see the -TransferQueue guide for the attach/run-cmd workflow): +Without `DRY_RUN=0` it prints the resolved driver command and exits, which is +worth doing once. Interactive mode — allocate once, keep Ray up, run the +driver by hand, edit, re-run — follows the same attach/run-cmd workflow as +the TransferQueue recipe: ```bash NG_TIC_FP_CANONICAL=1 bash swe_nano_sc_capture_interactive.sh ``` -The legacy comparison arm is the plain `swe_nano_sc.sh` / -`swe_nano_sc_interactive.sh` from the TransferQueue recipe. Batch and -interactive build byte-for-byte the same driver command. +The non-capture baseline remains `swe_nano_sc.sh` / +`swe_nano_sc_interactive.sh`. Batch and interactive build byte-for-byte the +same driver command. -Two flags matter: +`WALLTIME` must be a Slurm time string (`3:59:00`). A bare `3h` fails with +`sbatch: error: Invalid --time specification` printed at the very *end* of +the launch output — easy to miss. Budget ~1 h 40 of setup (venv rebuild + +checkpoint load) before the first step. -- `NG_TIC_FP_CANONICAL=1` — canonical text fingerprints in the gate. - **Without it token-in silently degrades to ~0**: reasoning models echo - history with `` blocks stripped, the gate's fingerprint of the served - turn never matches, every call falls back to text mode, and the run develops - a +38 % generated-token divergence. With it, `token_in_rate ≈ 0.9999`. -- `WALLTIME` must be a Slurm time string (`3:59:00`). `3h` fails with - `sbatch: error: Invalid --time specification` printed at the very *end* of - the launch output — easy to miss. Budget ~1 h 40 of setup (venv rebuild + - checkpoint load) before the first step. +## What the capture launchers add, and why -## What the capture launcher adds, and why - -Every line of the capture posture in `swe_nano_sc_capture.sh` exists because -its absence crashed a run: +Every line of the capture posture in `swe_nano_sc_capture*.sh` exists because +its absence broke a run: | Setting | Without it | |---|---| | `+token_capture.enabled=true` (hydra append) | Capture never engages. It is a `+` append because no yaml in the nano chain defines the key; a plain override is rejected. | +| `NG_TIC_FP_CANONICAL=1` | Token-in silently degrades to ~0: reasoning models echo history with `` blocks stripped, the gate's fingerprint of the served turn never matches, and every call falls back to text mode. With it, `token_in_rate ≈ 0.9999`. | | `NRL_DRIVER_PYTHONPATH=/opt/nemo-rl/3rdparty/Gym-workspace/Gym` | Driver `ModuleNotFoundError: nemo_gym` — the driver imports the staging record schema, and the baked driver venv has no nemo_gym. | | `NRL_DRIVER_PIP_INSTALL=orjson` | Driver `ModuleNotFoundError: orjson` — Gym's `token_id_capture/__init__` eagerly imports the store. | -| `VllmAsyncGenerationWorker` in `NRL_FORCE_REBUILD_VENVS_LIST` | Worker `ModuleNotFoundError: orjson` — venv caching is spec-unaware and silently reuses the legacy arm's venv (RL #3456 known issue). | -| capture env set *after* sourcing `swe_nano.env` | `swe_nano.env` exports `NRL_FORCE_REBUILD_VENVS_LIST` unconditionally and clobbers an env-prefix value. | - -## How the data flows - -``` -agent (nv-OpenHands) gate (vllm_model, gate mode) - /run body carries ng_rollout_id ───► registers rollout, admits calls - fingerprints history → resolves - parent → sends exact prefix ids - │ required_prefix_token_ids - ▼ -vLLM worker: splices prefix verbatim, renders only the new tail, - generates, then STAGES the delta (ids+mask+logprobs) to TransferQueue - (synchronous tq_put — durable before the call is acked) and returns - token-light CommitCoords on the response - │ coords (≈4 B/token) - ▼ -gate ingests coords into lineage; at /run end returns a token-free -RolloutReceipt (manifest of call_ids + staging keys) to the trainer - ▼ -finalizer: fetches staged deltas by key, digest-verifies, rebuilds the -exact training row, clears the staged rows -``` - -The heavy bytes (token arrays, logprobs) move exactly once, worker→TQ, -node-locally. The gate and the `/run` response stay token-light. +| `VllmAsyncGenerationWorker` in `NRL_FORCE_REBUILD_VENVS_LIST` | Worker `ModuleNotFoundError: orjson` — venv caching is spec-unaware and silently reuses a non-capture worker venv built by an earlier job. | +| capture env set *after* sourcing `swe_nano.env` | `swe_nano.env` exports `NRL_FORCE_REBUILD_VENVS_LIST` unconditionally and clobbers an env-prefix value — which is why these are dedicated launchers rather than an env prefix on `swe_nano_sc.sh`. | +| `CALL_TIMING=0` (optional) | Per-call latency JSONL is on by default in these launchers (`NRL_CALL_TIMING_DIR`/`NG_CALL_TIMING_DIR`); set 0 to disable. All probes are env-gated and dormant without the dir. | ## Verifying capture is really engaged Config echo is not evidence. Check, in order: -1. **Gate metrics in the SC step output** (grade from the SC worker - `.out` under `-logs/ray/session_*/logs/worker-*-.out` — the - driver log's actor-stdout forwarding is not reliable): +1. **Gate metrics in the SC step output.** Grade from the SC worker `.out` + under `-logs/ray/session_*/logs/worker-*-.out` — the driver + log's actor-stdout forwarding is not reliable and can make a healthy run + look stalled: ``` gate_metrics={'registered': 128.0, 'token_in': 7042.0, @@ -118,50 +128,36 @@ Config echo is not evidence. Check, in order: ``` `token_in / (token_in + fallback_*)` should be ≥ 0.99. A rate near 0 with - large `fallback_no_match` means canonical fingerprints are off (see above). - `unattributed_calls` > 0 means the harness is not forwarding - `ng_rollout_id` (attribution fix missing). + large `fallback_no_match` means canonical fingerprints are off (see + above). `unattributed_calls` > 0 means the harness is not forwarding + `ng_rollout_id` (attribution fix missing from the Gym pin). 2. **No finalize rejections.** `finalize: ... rejected (empty_manifest)` on - every rollout means calls are reaching the worker without gate context — - the run generates but nothing is trainable. + every rollout means calls reach the worker without gate context — the run + generates but nothing is trainable. -3. **TQ staging traffic**: `PUT_DATA` on the staging partition per model call - (tens of thousands per run), not just per training batch. +3. **TQ staging traffic**: `PUT_DATA` on the staging partition fires per + model call (tens of thousands per run), not just per training batch. 4. **Training equivalence**: `token_mult_prob_error` should sit near 1.0 - (max ≲ 3). `gen_kl_error` ~0.004 matches the legacy arm. - -## Perf instrumentation (optional) - -`CALL_TIMING=1` (default in these launchers) sets `NRL_CALL_TIMING_DIR` / -`NG_CALL_TIMING_DIR` and emits per-call JSONL segments from every server: -worker engine/preprocess/splice, `tq_put`, gate prepare/ingest, per-path HTTP -timings, event-loop lag. Aggregate with `swe/aggregate_call_timing.py`. -`cached_tokens` on each `worker_engine` record reports vLLM prefix-cache hits -(~94 % of prompt tokens on this workload). All probes are env-gated and -dormant unless the dir is set. + (max ≲ 3), `gen_kl_error` in the same band as a non-capture run (~0.004). ## Known limits - **Grade runs from the SC worker `.out` or W&B, never the driver log** — Ray's driver-log stdout forwarding dropped entire actors in testing (runs looked stalled while training normally). -- **Receipt-mode W&B rollout metrics are not yet comparable**: +- **Receipt-mode W&B rollout metrics are not yet comparable to legacy**: `gen_tokens_per_sample` counts the carried prompt tail and - `truncation_rate` is constant on the capture arm. Compare token counts via - the call-timing JSONL (engine `usage`) instead. -- **`global_valid_toks` runs ~15 % lower on capture** at equal work — - masking-semantics difference under investigation; do not quote - convergence-per-wallclock until resolved. -- **Router replay (R3)**: capture stages routed experts as TQ extras beside - the token delta (`routed_experts_delta`), keeping the multi-MB arrays off - HTTP — but vLLM 0.20.0's engine crashes with - `enable_return_routed_experts=true` on this model (`IndexError` in - `sample_tokens`); blocked on an engine fix. + `truncation_rate` is constant on the capture arm. - **Weight-version mixing** across spliced chains at a refit boundary is - guarded conservatively; a per-row `weight_version` check in the finalizer - is the hardening item. + tag-checked per call; a per-row guard in the finalizer is a pending + hardening item. +- **Router replay (R3)**: the capture path stages routed experts beside the + token delta (`routed_experts_delta` extras) so the per-token arrays never + transit HTTP — but vLLM 0.20.0's engine currently crashes with + `enable_return_routed_experts=true` on this model; blocked on an engine + fix. ## Related From 44c87a43a676a74af902f2869d07c5b5da5820e9 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Thu, 6 Aug 2026 14:43:18 -0700 Subject: [PATCH 08/52] docs(swe): rollout call-flow walkthrough in the token-capture guide Eight-step anatomy of one rollout (dispatch -> attribution -> gated turn loop -> stage -> commit -> verify/seal -> finalize -> train) ahead of the compressed token-path diagram, so a reader gets the lifecycle before the wire detail. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre --- docs/guides/nano-swe-token-capture.md | 60 ++++++++++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-) diff --git a/docs/guides/nano-swe-token-capture.md b/docs/guides/nano-swe-token-capture.md index 974f48a8bed..ba035d36885 100644 --- a/docs/guides/nano-swe-token-capture.md +++ b/docs/guides/nano-swe-token-capture.md @@ -33,7 +33,65 @@ Legacy SWE runs recover training tokens by echoing token ids in every model response and re-tokenizing the agent's rendered history each turn. Both are lossy for a reasoning model — the chat template strips `` content from history, and re-tokenizing an assistant turn can split differently than the -tokens the model actually sampled. Token capture replaces that path: +tokens the model actually sampled. Token capture replaces that path. + +### Anatomy of one rollout + +The call flow for a single SWE rollout, end to end: + +1. **Dispatch.** The `SingleControllerActor`'s rollout pump pulls a prompt + group from the dataset and hands it to the NemoGym environment actor, + which POSTs `/run` to the `swe_agents_train` Gym server. The run body + carries a fresh `ng_rollout_id`; the gate **registers** the rollout + (control plane) before any model call happens. +2. **Sandbox + agent.** The SWE harness materializes the SWE-bench instance + in a sandbox and starts the OpenHands agent. The rollout id is snapshotted + into the agent's config so every LLM request it makes can be attributed + back to this rollout (`metadata.ng_rollout_id` on the request body). +3. **Agent turn loop.** Each turn, OpenHands POSTs `/v1/chat/completions` + with the *full rendered history* to the policy model server, which runs + in **gate mode**: + - the gate admits the call (rejects unregistered rollouts), fingerprints + the incoming history, and resolves which previously served call this + one continues (its *parent*) in the lineage index; + - on a match it attaches the parent's **exact cumulative token ids** + (`required_prefix_token_ids`) plus a capture context + (`rollout_id`, `call_id`, `parent_call_id`, `prev_len`), and forwards + the request to a vLLM engine (rollout→engine affinity keeps a + rollout's calls on the engine holding its KV cache); + - no match (a true root, or a rewritten/condensed history) starts a new + chain — a fallback, never an error. +4. **Generate + stage.** The vLLM worker splices the supplied prefix + verbatim, renders only the new tail through the chat template, and + generates. Before acknowledging the call it **stages the call's token + delta** — `rendered_prompt[prev_len:] + generated` ids, a loss mask + (0.0 on carried prompt, 1.0 on generated), and per-token logprobs — to + the TransferQueue staging partition (synchronous `tq_put`: bytes are + durable before the response leaves the worker). The response back to the + gate carries only text plus token-light `CommitCoords` (~4 B/token). +5. **Commit.** The gate ingests the coords into its lineage index (this + *is* the authoritative commit — the next turn's parent resolution + depends on it), strips them, and returns a plain OpenAI-shaped + completion to the agent. Steps 3-5 repeat for every tool call the agent + makes (tool execution happens agent-side between turns). +6. **Verify + seal.** When the agent finishes, the harness runs the + SWE-bench verifier to score the patch (reward 0/1). The `/run` response + returns the reward plus a **token-free `RolloutReceipt`**: the ordered + manifest of committed calls (call ids, staging keys, digests, weight + versions) and a `terminal_call_id` naming the chain that is the + training row. +7. **Finalize.** Trainer-side, the `BlackboxFinalizer` fetches the staged + deltas the manifest names from TQ, re-verifies each (digest, lengths, + mask shape, weight-version tags), linearizes the terminal chain into one + exact token row, and publishes it to the training partition — a rejected + rollout becomes a masked placeholder so the GRPO group keeps its shape. + Staged rows are cleared after publish. +8. **Train.** Once a global batch of rows is buffered, the SC takes an + optimizer step and syncs weights to the engines; the bumped + `weight_version` is stamped on subsequent calls so refit boundaries are + visible in the data. + +The token path in that flow, compressed: ``` agent (nv-OpenHands) gate (vllm_model, gate mode) From d44ac3f35333ce831f4ed763ba0ec827e501d336 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Sat, 8 Aug 2026 13:03:17 -0700 Subject: [PATCH 09/52] chore(swe): drop the A/B experiment harness from the PR swe/ was the legacy-vs-capture measurement apparatus (arm launcher, config deriver, perf aggregator, methodology notes), not part of the capture feature. Nothing in nemo_rl/, configs, launchers, or tests references it, and the run-trigger story it half-covered now lives in docs/guides/nano-swe-token-capture.md + swe_nano_sc_capture*.sh. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre --- swe/EXPERIMENT_LAUNCH.md | 129 ------------------------- swe/SWE_RUN.md | 193 ------------------------------------- swe/aggregate_perf.py | 113 ---------------------- swe/launch_swe_ab.sh | 75 -------------- swe/make_capture_config.py | 50 ---------- 5 files changed, 560 deletions(-) delete mode 100644 swe/EXPERIMENT_LAUNCH.md delete mode 100644 swe/SWE_RUN.md delete mode 100644 swe/aggregate_perf.py delete mode 100755 swe/launch_swe_ab.sh delete mode 100644 swe/make_capture_config.py diff --git a/swe/EXPERIMENT_LAUNCH.md b/swe/EXPERIMENT_LAUNCH.md deleted file mode 100644 index f9f3e9801f4..00000000000 --- a/swe/EXPERIMENT_LAUNCH.md +++ /dev/null @@ -1,129 +0,0 @@ -# Experiment Launch Plan — SWE Token-Capture A/B (2026-07-28) - -Concrete launch record for the run described generically in `SWE_RUN.md`. -Everything here is pinned to what will actually execute; update this file if -any pin changes before submission. **Status: READY — not yet submitted.** - -## 1. Hypothesis and expected outcome - -The gate-authoritative token-capture pipeline -(`docs/design-docs/tq-gym-gate-authoritative.md`) reduces HTTP bytes per -trained token and wall-clock step time vs the legacy token-echo path on a -real multi-turn agentic workload, at equal training quality. - -| Expectation | Basis | -|---|---| -| Bytes/token reduction ≥ 35.6 % (the single-turn floor), toward/beyond −46.9 % (sync-prototype multi-turn) | S5 instrumented A/B; legacy pays per-turn history re-echo (~quadratic in turns), capture stays O(generated tokens) | -| `exposed_generation` and `total_step_time` at or below legacy | S5 saw −16.8 % / −9.0 % single-turn; expect relative timing win to compress on a 30B MoE (GPU decode dominates) while the bytes win grows | -| `gate/token_in_rate` → 1.0 for continuation calls | First real measurement — OpenHands marker survival is the biggest unknown (risk #1); a low rate is a correctness-preserving perf bug, not a training bug | -| Reward / KL curves in the same band on both arms | S5: identical rewards 10/10, same KL band | - -## 2. Exact pins - -| What | Value | -|---|---| -| RL repo | branch `yukih/sc-entrypoint` @ `e71055650` (local-only; no remote has it) | -| Gym fork (submodule) | branch `tq-gate-capture` @ `e3b3eac6` (= upstream #2124 head `32b555f04` + S1–S3 capture commits + S5 byte-counter middleware; local-only) | -| Code snapshot | `code_snapshots/sc-swe-capture` @ `e710556` (immutable run tree; both arms run from it) | -| Container | `nemo-rl:sc-swe-baked.sqsh` (biguo; nightly-062526 + baked venvs — **stale vs branch lock**, hence forced venv rebuild below) | -| Model | Qwen3-30B-A3B-Thinking-2507, from SWE1 `step_230_hf` checkpoint (bihu, run dc3m70us lineage) | -| Train data | R2E-Gym subset jsonl (sdevare), 4,518 samples | -| Site config | `test_assets/SWE/grpo_qwen3_30b_async_swe.yaml` (real sandbox-image paths; snapshot copy is authoritative at run time) | -| Derived arm configs | `test_assets/SWE/derived_configs/grpo_swe_ab_{legacy,capture}.yaml` — regenerated at each submit by `make_capture_config.py` | - -Verified diff between derived arm configs: **only** `token_capture.enabled: -true` on the capture arm; `rollout_max_attempts_to_avoid_lp_nan: 1` pinned on -**both** arms (capture hard-errors without it; pinning both removes the -NaN-retry confound). - -## 3. Compute shape (per job; one job per arm) - -- 16 nodes × 8 GPUs (128 GPUs), SLURM `batch` partition, account - `coreai_dlalgo_genai`, 4 h wall time, `--exclusive`, - `--dependency=singleton` (arms queue back-to-back on comparable - allocations), idle-GPU-reaper exemption (180 min) in the job comment. -- 8 training nodes: Megatron-Core, TP=4 EP=8 CP=4 PP=2, pad 32. -- 8 generation nodes: async vLLM, TP=2, non-colocated. -- Single-controller entrypoint (`SC_MODE=1`, - `examples/run_grpo_single_controller.py`) + TransferQueue data plane. -- Sequence budget 131,072; GBS=64 (8 prompts/step × 8 generations/prompt); - LR 1e-6; staleness `age=1`, `force_in_order=true`, `over_sampling=false`. -- Agent: NeMo-Gym OpenHands, per-instance Singularity sandboxes (`.sif` - under igitman/sdevare lustre trees), up to ~100–200 turns/rollout. - -## 4. Environment posture (identical on both arms) - -- `NRL_FORCE_REBUILD_VENVS=true` — container venvs predate the branch's - `uv.lock`; capture arm additionally needs the unbaked `VLLM_GYM` worker - venv. First node-local build may take ~30 min (reaper exemption covers it). -- `GYM_VENV_DIR=/tmp/nemo_gym_venvs` — baked `/opt/gym_venvs` deps predate - the Gym fork's floors (aiohttp bump). -- `BYTES=1` (capture-arm measurement run): `NG_HTTP_BYTES_DIR` + - `NRL_HTTP_BYTES_DIR` → per-server JSON under `http_bytes//`; - forwarding into the job env added in `e71055650`. -- Secrets: `profiles/env.sh` (mode 600) sourced by the site wrapper; wrapper - fails loudly if `WANDB_API_KEY`/`HF_TOKEN` are unset. - -## 5. Run sequence and gates - -All submissions via (from repo root): - -```bash -SITE_WRAPPER=$PWD/test_assets/SWE/grpo_swe_tests.sh MAX_NUM_STEPS= ARM= [BYTES=1] bash swe/launch_swe_ab.sh -``` - -| # | Run | Purpose | Gate to proceed | -|---|---|---|---| -| 1 | `ARM=legacy MAX_NUM_STEPS=3` | Flag-off smoke: branch + Gym pin reproduce the known-good legacy SWE run | Job completes 3 steps; reward/KL sane; no venv/import failures | -| 2 | `ARM=legacy MAX_NUM_STEPS=25` | Measurement baseline | Completes ≥ 20 steps cleanly | -| 3 | `ARM=capture MAX_NUM_STEPS=25 BYTES=1` | Measurement + mechanism + byte accounting | — | -| 4 | (optional) repeat pair ×2 | Variance bars; S5 assessment puts run-to-run variance at ±5–10 % | 1 pair = directional, 3 pairs = quotable | - -W&B: project `pthombre-swe-capture-ab`, runs named -`swe-ab--`. Checkpoints: `/results//`. - -## 6. Analysis plan - -- Medians over steps 5..N (skip warm-up): `timing/train/total_step_time`, - `timing/train/exposed_generation`, `timing/train/valid_tokens_per_sec_per_gpu`. -- Mechanism: `gate/token_in_rate` (the headline first-time measurement), - `gate/fallback_{no_marker,fingerprint_miss,unknown_marker}` — sustained - `fingerprint_miss` ⇒ OpenHands rewrites history (marker-survival risk #1; - report back to the design doc §11). -- Health: `gate/capture_failed`, `train/global_valid_seqs` == GBS both arms; - `finalize/*` latency (if it dominates dispatch → argues for H5 finalizer pool). -- Equivalence: overlay `train/reward` + `train/gen_kl_error` across arms. -- Bytes: `python swe/aggregate_perf.py ` - → total bytes, bytes/trained-token, per-hop table. -- Results recorded in - `docs/design-docs/tq-gym-gate-authoritative-implementation-log.md` (post-MVP - perf report per design § 10). - -## 7. Known risks (accepted, watched) - -- **Gate death = silent stall** (S5 chaos finding; H1 fix queued): control - plane retries a dead gate unboundedly → job hangs, doesn't fail. - Supervise via `squeue` + driver log in - `code_snapshots/sc-swe-capture/logs/slurm/`. -- **Gate buffer growth uncapped** (H5): ids-only delta forests; at 131k × - in-flight concurrency raw ids are ~100s MB but Python overhead is real — - watch policy-model server RSS. -- **OpenHands marker survival unknown** — measured, not assumed - (`token_in_rate`); low rate degrades perf, never correctness. -- **Secrets in job env**: the launcher embeds `WANDB_API_KEY`/HF token in - the sbatch command (visible in SLURM logs; pre-existing behavior). - Rotate tokens if logs are shared. -- Snapshot is immutable but the **derived configs are re-generated per - submit** — do not edit the site yaml between arms of a pair. - -## 8. Pre-flight checklist (all verified 2026-07-28) - -- [x] S1–S5 + launcher committed (RL `e71055650`, Gym `e3b3eac6`) -- [x] Snapshot `sc-swe-capture` @ `e710556`: S5 files, `examples/swe_bench/` - launcher, site yaml, Gym gate + byte middleware present -- [x] Dry-run both arms end-to-end (`DRY_RUN=1`): correct account, snapshot - paths, singleton, reaper comment; derived-config diff = capture flag only -- [x] Container / model / data / sandbox images readable by pthombre -- [x] SLURM account `coreai_dlalgo_genai` valid for user (nemotron_sw_post is not) -- [x] Secrets profile in place; wrapper verified with tokens stripped from env -- [ ] SUBMIT (awaiting explicit go) diff --git a/swe/SWE_RUN.md b/swe/SWE_RUN.md deleted file mode 100644 index 335eff4e6db..00000000000 --- a/swe/SWE_RUN.md +++ /dev/null @@ -1,193 +0,0 @@ -# SWE Run — Token-Capture vs Legacy Perf A/B at Scale - -Runbook for measuring the performance of the gate-authoritative token-capture -pipeline (`token_capture.enabled=true`, design: -`docs/design-docs/tq-gym-gate-authoritative.md`) against the legacy token-echo -path on a real workload: **async GRPO on SWE-bench** (Qwen3-30B-A3B thinking, -NeMo-Gym OpenHands agent, 16 nodes). This is the multi-turn run the 2-GPU S5 -experiments could not provide — SWE agents run up to 100–200 turns per -rollout, which is exactly where the legacy echo pays its per-turn token -re-transmission and where capture's `token_in_rate` and prefix serving are -finally exercised for real. - -Small-scale S5 results this run extends (implementation log, -`docs/design-docs/tq-gym-gate-authoritative-implementation-log.md` § S5): -−35.6 % HTTP bytes/trained token, −9 % step time, −16.8 % generation time on -a *single-turn* workload — the floor. The sync prototype measured −46.9 % -bytes/token multi-turn. - ---- - -## 1. The stack being launched - -``` -swe/launch_swe_ab.sh (this folder: picks the arm, derives the capture yaml) - └─ site wrapper e.g. .../nemo_rl-sc-test/test_assets/SWE/grpo_swe_tests.sh - (account, container, secrets, lustre paths — user-specific, gitignored) - └─ snapshot launcher /examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh - (env-agnostic: parallelism, async_rl/data_plane overrides, sbatch) - └─ ray.sub → SLURM (16 nodes: 8 train TP4/EP8/CP4/PP2 + 8 gen vLLM TP2, - seqlen 131072, SC_MODE=1 single-controller + TQ) -``` - -Jobs run from an immutable **code snapshot** -(`test_assets/SWE/sync_code.sh`). The snapshot rsyncs `3rdparty/` wholesale, -so the **local-only Gym fork commits ride along** — the open gitlink/CI -question does not block this experiment. - -## 2. The A/B design - -One independent variable. Everything else — snapshot, container, data, model, -parallelism, async_rl knobs, node count — identical between arms. - -| | Arm A: `legacy` | Arm B: `capture` | -|---|---|---| -| Config | site yaml as-is | site yaml + `token_capture.enabled: true` (derived by `make_capture_config.py`) | -| Token path | gate string-parses ids + `/tokenize` per call; ids/logprobs echoed through agent messages every turn; tokens ride the Ray return | gate custodies lineage; worker stages delta ids+logprobs to TQ once; markers + token-free receipts | -| Rows into training | `TQReplayBuffer.commit` tensorize | `BlackboxFinalizer` verify → publish | - -Both arms use the launcher's defaults, which sit inside the capture MVP -matrix: `batch_selection_strategy=staleness_window`, age 1, -`mixed_weight_version_policy=allow` (default), async vLLM engine, -non-colocated generation. `make_capture_config.py` also pins -`rollout_max_attempts_to_avoid_lp_nan: 1` — capture setup hard-errors -otherwise (NaN-retry would re-register create-only rollout ids); it is set on -**both** arms' derived configs so retry behavior is not a confound. - -### Run protocol (perf reading) - -1. **Flag-off smoke first** (`MAX_NUM_STEPS=3`): proves the branch + Gym pin - reproduce the legacy SWE run before anything is compared. The standing - dormant-by-default discipline. -2. **Measurement runs**: `MAX_NUM_STEPS=20`–`30` per arm (long enough that - medians beat setup noise; short enough to iterate). `--dependency=singleton` - is already in the launcher — submit both arms back-to-back under the same - job name family so they land on comparable allocations. -3. **Repeats**: 1 pair = directional; 3 pairs = quotable (the S5 assessment: - run-to-run variance on this stack is plausibly ±5–10 %). - -## 3. Launching - -### Prerequisites (once) - -- The branch state you want measured is **committed** (RL repo and Gym - submodule) — snapshots are cut from a checkout. -- A checkout of `yukih/sc-entrypoint` (with the `tq-gate-capture` submodule - state) on lustre to snapshot from, e.g. clone + check out, or reuse an - existing sc-test workspace updated to this branch. -- A personal site wrapper (copy - `test_assets/SWE/grpo_swe_tests.sh` and edit ACCOUNT / CONTAINER / - secrets-profile / cache / W&B paths for your user). -- Snapshot it: - -```bash -cd # repo root, branch checked out -bash test_assets/SWE/sync_code.sh sc-swe-capture -``` - -### Submitting the arms - -```bash -# from this repo's swe/ folder -SITE_WRAPPER=/lustre/.../your_grpo_swe_tests.sh \ - MAX_NUM_STEPS=3 ARM=legacy bash swe/launch_swe_ab.sh # flag-off smoke - -SITE_WRAPPER=... MAX_NUM_STEPS=25 ARM=legacy bash swe/launch_swe_ab.sh -SITE_WRAPPER=... MAX_NUM_STEPS=25 ARM=capture bash swe/launch_swe_ab.sh - -# optional: HTTP byte accounting (writes per-server JSON to lustre) -SITE_WRAPPER=... MAX_NUM_STEPS=25 ARM=capture BYTES=1 bash swe/launch_swe_ab.sh -``` - -`launch_swe_ab.sh` does four things: derives the arm's config -(`make_capture_config.py` for the capture arm), stamps -`EXP_SUFFIX=swe-ab--...` so W&B separates the arms, forces the venv -posture below, and `exec`s your site wrapper (all launcher knobs still pass -through the environment: `TP`, `PPS`, `OVER_SAMPLING`, `DRY_RUN=1`, …). - -### Environment posture (why the wrapper forces these) - -- **`NRL_FORCE_REBUILD_VENVS=true` on BOTH arms.** The baked container's - `/opt/ray_venvs` predate this branch's `uv.lock` (S1 regenerated it), and - the capture arm additionally needs the `VLLM_GYM` worker venv - (`--extra vllm --extra nemo_gym`) that no image bakes yet. Forcing rebuild - on both arms keeps setup cost out of the A/B. Pre-warm `LUSTRE_UV_CACHE` - once (see the launcher header) or the first node-local build is ~30 min — - the 180-min idle-GPU reaper exemption in the site wrapper covers it. -- **`GYM_VENV_DIR=/opt/gym_venvs` (image-baked) on BOTH arms.** A node-local - `/tmp` rebuild does NOT work multi-node: the venv build runs only on the - NemoGym actor's node while Gym spawns servers cluster-wide (job 14542017 - failed on exactly this). The baked venvs are dependency-compatible with the - fork (verified: `swe_agents`/`vllm_model` requirements unchanged vs the old - pin; only the core aiohttp CVE floor moved, irrelevant at runtime), and the - editable install serves the fork's *code* on every node. If a future fork - commit does change server deps, rebake via - `test_assets/SWE/prebuild_gym_venvs.sh`. -- **Pre-warm `LUSTRE_UV_CACHE` for the branch's `uv.lock`** (single process, - inside the job container — the source builds need `nvcc`): without it the - 16-node venv rebuild exceeds the idle-GPU reaper's *de facto* 60-minute - kill (observed twice; the 180-min exemption comment is valid per the - exemption guide but not honored beyond 60 — escalate to - @job-reaper-support with jobs 14516316/14521677 if 180 is ever needed). - -## 4. What to compare (the perf read) - -All of these land in W&B (both arms) — the capture-arm `gate/*` block comes -from the per-step SC gate-metrics logging added in S5. - -**Headline perf (medians over steps 5..N, skipping warm-up):** - -| Metric (W&B key) | Expectation | -|---|---| -| `timing/train/total_step_time` | capture ≤ legacy; S5 saw −9 % single-turn | -| `timing/train/exposed_generation` | the mechanism lives here (no `/tokenize` round-trip, ~40 % smaller per-call payloads, no echo growth); S5 saw −16.8 % | -| `timing/train/valid_tokens_per_sec_per_gpu` | inverse of the above | - -**Capture-arm health / mechanism metrics:** - -| Metric | What it tells you | -|---|---| -| `gate/token_in_rate` | THE number this run finally measures. High (→1.0 for continuations) = OpenHands echoes markers faithfully and exact-prefix serving works at depth. Low = fallbacks (see next row) — runs stay *correct* but pay full re-renders. | -| `gate/fallback_no_marker` / `fallback_fingerprint_miss` / `fallback_unknown_marker` | Which § 3.3 fallback is firing if token_in_rate is low. `fingerprint_miss` at scale ⇒ the OpenHands history pipeline rewrites messages (report back — this is marker-survival risk #1 for agent frameworks). | -| `gate/capture_failed`, `train/global_valid_seqs` | staging failures → placeholder rows. valid_seqs should equal GBS on both arms. | -| `finalize/*` (rollout metrics) | finalizer latency rides the dispatch task (MVP placement) — SWE's 131k-token multi-call rollouts make this worth watching; if it dominates dispatch, that argues for the H5 finalizer pool. | - -**Training equivalence (guards the perf claim):** overlay `train/reward` -curves and `train/gen_kl_error` between arms — same band = the perf delta is -not bought with training drift. (S5: identical rewards 10/10, same KL band.) - -**Optional byte accounting (`BYTES=1`):** env-gated ASGI counters -(`NG_HTTP_BYTES_DIR` on every Gym server — the SWE config runs -`num_workers: 1`, which the counter supports — and `NRL_HTTP_BYTES_DIR` on -the vLLM workers) write per-server, per-route JSON to -`/http_bytes/`. Aggregate with `swe/aggregate_perf.py - ` → total bytes, bytes per trained token, per-hop -table. Expect the multi-turn reduction to land between the S5 floor -(−35.6 %) and beyond the prototype's −46.9 % as turn count grows. - -## 5. Known risks at this scale (accepted, watched) - -- **Gate death = silent stall, not loud failure** (S5 chaos finding, fix - queued for H1): Gym's control-plane client retries a dead gate unboundedly, - so if the policy-model server dies mid-run the job hangs rather than - failing. Supervise; `squeue` + driver log tell you which. -- **Gate buffer growth is uncapped** (H5 adds caps): per-rollout delta - forests, ids only. At seqlen 131k × concurrency 768 the raw ids are modest - (~100s of MB), but Python-object overhead is real — watch the policy-model - server RSS. -- **Marker survival through OpenHands** is the biggest unknown — it uses the - same message carrier the legacy token echo uses (so it *should* survive - wherever legacy works today), but `token_in_rate` is the proof either way, - and a low rate is a correctness-preserving perf bug, not a training bug. -- The idle-GPU reaper exemption and `checkpoint_must_save_by` are already - handled by the site wrapper / launcher. - -## 6. Files in this folder - -| File | Purpose | -|---|---| -| `SWE_RUN.md` | this runbook | -| `EXPERIMENT_LAUNCH.md` | pinned launch record for the 2026-07-28 A/B (exact commits, snapshot, gates, checklist) | -| `launch_swe_ab.sh` | arm selector: derives config, names the run, forces venv posture, delegates to your site wrapper | -| `make_capture_config.py` | derives the arm configs from the site yaml (`token_capture.enabled` + NaN-retry pin) | -| `aggregate_perf.py` | offline aggregation: per-hop HTTP bytes, bytes/trained-token, timing medians, token_in_rate | diff --git a/swe/aggregate_perf.py b/swe/aggregate_perf.py deleted file mode 100644 index c5490a60ebd..00000000000 --- a/swe/aggregate_perf.py +++ /dev/null @@ -1,113 +0,0 @@ -#!/usr/bin/env python3 -# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Aggregate the instrumented legacy-vs-capture perf pair. - -Usage: python aggregate_perf.py - -Each run dir: http_bytes/*.json (per-server per-route byte counters), -metrics.json (TB dump), train_rows_*.jsonl (row dump for token counts). -""" - -import glob -import json -import os -import sys - - -def load_bytes(run_dir): - per_server = {} - for path in glob.glob(os.path.join(run_dir, "http_bytes", "*.json")): - name = os.path.basename(path).rsplit("_", 1)[0] - data = json.load(open(path)) - agg = per_server.setdefault(name, {}) - for route, c in data.items(): - e = agg.setdefault(route, {"requests": 0, "req_bytes": 0, "resp_bytes": 0}) - for k in e: - e[k] += c[k] - return per_server - - -def total_trained_tokens(run_dir): - total = 0 - for path in glob.glob(os.path.join(run_dir, "train_rows_*.jsonl")): - for line in open(path): - if line.strip(): - total += json.loads(line)["input_lengths"] - return total - - -def timing(run_dir): - d = json.load(open(os.path.join(run_dir, "metrics.json"))) - out = {} - for key in ( - "timing/train/total_step_time", - "timing/train/exposed_generation", - "timing/train/policy_training", - "timing/train/weight_sync", - "timing/train/valid_tokens_per_sec_per_gpu", - "train/gen_kl_error", - "train/reward", - "gate/token_in_rate", - "gate/token_in", - "gate/fallback_no_marker", - "gate/fallback_fingerprint_miss", - ): - s = d.get(key) - if isinstance(s, dict) and s: - vals = [v for _, v in sorted(s.items(), key=lambda kv: int(kv[0]))] - out[key] = vals - return out - - -def main(): - legacy_dir, capture_dir = sys.argv[1], sys.argv[2] - report = {} - for tag, run_dir in (("legacy", legacy_dir), ("capture", capture_dir)): - servers = load_bytes(run_dir) - toks = total_trained_tokens(run_dir) - grand = {"requests": 0, "req_bytes": 0, "resp_bytes": 0} - print(f"\n===== {tag} =====") - for name, routes in sorted(servers.items()): - s = {"requests": 0, "req_bytes": 0, "resp_bytes": 0} - for route, c in routes.items(): - for k in s: - s[k] += c[k] - for k in grand: - grand[k] += s[k] - print(f"{name:32s} req={s['requests']:6d} in={s['req_bytes']/1e6:8.2f} MB out={s['resp_bytes']/1e6:8.2f} MB") - for route, c in sorted(routes.items(), key=lambda kv: -(kv[1]["req_bytes"] + kv[1]["resp_bytes"]))[:4]: - print(f" {route:40s} n={c['requests']:5d} in={c['req_bytes']/1e6:7.2f} MB out={c['resp_bytes']/1e6:7.2f} MB") - total_bytes = grand["req_bytes"] + grand["resp_bytes"] - print(f"{'TOTAL':32s} req={grand['requests']:6d} bytes={total_bytes/1e6:.2f} MB trained_tokens={toks} bytes/token={total_bytes/max(toks,1):.1f}") - report[tag] = {"total_bytes": total_bytes, "tokens": toks, "timing": timing(run_dir)} - - lt, ct = report["legacy"], report["capture"] - print("\n===== comparison =====") - bt_l = lt["total_bytes"] / max(lt["tokens"], 1) - bt_c = ct["total_bytes"] / max(ct["tokens"], 1) - print(f"HTTP bytes/trained token: legacy {bt_l:.1f} -> capture {bt_c:.1f} ({(bt_c/bt_l-1)*100:+.1f}%)") - for key in ("timing/train/total_step_time", "timing/train/exposed_generation", "timing/train/valid_tokens_per_sec_per_gpu"): - lv, cv = lt["timing"].get(key), ct["timing"].get(key) - if lv and cv: - import statistics - ml, mc = statistics.median(lv), statistics.median(cv) - print(f"{key}: legacy median {ml:.3f} -> capture median {mc:.3f} ({(mc/ml-1)*100:+.1f}%)") - tir = ct["timing"].get("gate/token_in_rate") - if tir: - print(f"capture token_in_rate (cumulative, final): {tir[-1]:.3f}") - - -if __name__ == "__main__": - main() diff --git a/swe/launch_swe_ab.sh b/swe/launch_swe_ab.sh deleted file mode 100755 index aff2fc04f88..00000000000 --- a/swe/launch_swe_ab.sh +++ /dev/null @@ -1,75 +0,0 @@ -#!/bin/bash -# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================= -# SWE token-capture A/B arm launcher — see swe/SWE_RUN.md. -# -# Picks the arm, derives its config, names the run, forces the venv posture, -# then delegates to YOUR site wrapper (account/container/secrets live there). -# All launcher knobs (TP, PPS, MAX_NUM_STEPS, DRY_RUN=1, ...) pass through. -# -# Required: -# ARM=legacy|capture -# SITE_WRAPPER=/path/to/your grpo_swe_tests.sh-style wrapper -# Optional: -# BYTES=1 enable per-hop HTTP byte counters (JSON under the -# checkpoint dir; aggregate with swe/aggregate_perf.py) -# BASE_CONFIG site yaml to derive from (default: the wrapper's default, -# discovered via CONFIG_FILE after the wrapper sources it — -# set explicitly if your wrapper computes CONFIG_FILE late) -# ============================================================================= -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -ARM="${ARM:?set ARM=legacy|capture}" -SITE_WRAPPER="${SITE_WRAPPER:?set SITE_WRAPPER=/path/to/your site wrapper}" -[[ "${ARM}" == "legacy" || "${ARM}" == "capture" ]] || { echo "ARM must be legacy|capture" >&2; exit 1; } - -# ---- Derive the arm's config from the site yaml ----------------------------- -# Default BASE_CONFIG: the yaml sitting next to the site wrapper (the -# grpo_swe_tests.sh convention). Override for other layouts. -BASE_CONFIG="${BASE_CONFIG:-$(dirname "${SITE_WRAPPER}")/grpo_qwen3_30b_async_swe.yaml}" -[ -f "${BASE_CONFIG}" ] || { echo "BASE_CONFIG not found: ${BASE_CONFIG}" >&2; exit 1; } -DERIVED_DIR="${DERIVED_DIR:-$(dirname "${BASE_CONFIG}")/derived_configs}" -mkdir -p "${DERIVED_DIR}" -export CONFIG_FILE="${DERIVED_DIR}/grpo_swe_ab_${ARM}.yaml" -python3 "${SCRIPT_DIR}/make_capture_config.py" "${BASE_CONFIG}" "${CONFIG_FILE}" "${ARM}" - -# ---- Run naming (W&B separates the arms) ------------------------------------ -export EXP_SUFFIX="${EXP_SUFFIX:-swe-ab-${ARM}-$(date +%m%d%H%M)}" - -# ---- Venv posture: identical on both arms (see SWE_RUN.md § 3) -------------- -# Baked /opt/ray_venvs predate this branch's lock; the capture arm needs the -# unbaked VLLM_GYM worker venv. Forcing rebuild on both arms keeps setup cost -# out of the A/B. -export NRL_FORCE_REBUILD_VENVS=true -# Gym venvs: use the image-baked /opt/gym_venvs (present on EVERY node). -# A node-local GYM_VENV_DIR=/tmp/... does NOT work multi-node: the venv build -# runs only on the NemoGym actor's node while Gym spawns servers cluster-wide -# (learned from job 14542017). The fork's only dep-floor change vs the baked -# venvs is the aiohttp CVE bump (verified: swe_agents/vllm_model requirements -# unchanged), and editable installs serve the fork's *code* either way. -export GYM_VENV_DIR="${GYM_VENV_DIR:-/opt/gym_venvs}" - -# ---- Optional per-hop HTTP byte accounting ---------------------------------- -if [ "${BYTES:-0}" = "1" ]; then - BYTES_DIR="${BYTES_DIR:-${CHECKPOINT_ROOT:-$(dirname "${SITE_WRAPPER}")/../..}/http_bytes/${EXP_SUFFIX}}" - mkdir -p "${BYTES_DIR}" - export NG_HTTP_BYTES_DIR="${BYTES_DIR}" - export NRL_HTTP_BYTES_DIR="${BYTES_DIR}" - echo "[launch_swe_ab] byte counters on -> ${BYTES_DIR}" -fi - -echo "[launch_swe_ab] arm=${ARM} config=${CONFIG_FILE} exp=${EXP_SUFFIX}" -exec bash "${SITE_WRAPPER}" "$@" diff --git a/swe/make_capture_config.py b/swe/make_capture_config.py deleted file mode 100644 index 5880a736655..00000000000 --- a/swe/make_capture_config.py +++ /dev/null @@ -1,50 +0,0 @@ -# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Derive the SWE A/B arm configs from a site yaml (see swe/SWE_RUN.md). - -Usage: - python swe/make_capture_config.py legacy|capture - -Both arms get `env.nemo_gym.rollout_max_attempts_to_avoid_lp_nan: 1` (the -capture arm hard-errors without it; pinning it on both keeps NaN-retry -behavior out of the A/B). The capture arm additionally gets -`token_capture.enabled: true` — the gate config injection into the Gym -policy-model server happens in code (`environments/nemo_gym.py:_spinup`), -so no other yaml change is needed. -""" - -import sys - -import yaml - - -def main() -> None: - src, dst, arm = sys.argv[1], sys.argv[2], sys.argv[3] - assert arm in ("legacy", "capture"), f"arm must be legacy|capture, got {arm}" - with open(src) as f: - config = yaml.safe_load(f) - - config.setdefault("env", {}).setdefault("nemo_gym", {})[ - "rollout_max_attempts_to_avoid_lp_nan" - ] = 1 - if arm == "capture": - config.setdefault("token_capture", {})["enabled"] = True - - with open(dst, "w") as f: - yaml.safe_dump(config, f, sort_keys=False) - print(f"[make_capture_config] arm={arm}: {src} -> {dst}") - - -if __name__ == "__main__": - main() From cd7669301c7a372883eba4fed903570891b26147 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Sat, 8 Aug 2026 13:22:43 -0700 Subject: [PATCH 10/52] chore(swe): scope the PR to the capture feature + one Nano recipe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drop what the goal does not need: the Qwen3-30B async swe_bench recipe (the async path does not honour the data plane, so it cannot illustrate capture), the sync-TQ recipe variant, and the two non-capture launcher variants. Relocate the token_capture exemplar block from the unrelated Qwen3 config into nano_swe_teacher_sc.yaml (enabled: false) — the key now exists in the recipe chain, so the capture launchers use a plain override instead of a hydra + append. Guides trimmed to match. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre --- docs/guides/nano-swe-token-capture.md | 4 +- docs/guides/nano-swe-transferqueue.md | 9 - .../ultra/nano_swe_teacher_qwen3mesh.yaml | 2 +- .../configs/ultra/nano_swe_teacher_sc.yaml | 38 ++ .../ultra/nano_swe_teacher_sync_tq.yaml | 46 -- .../nemo_gym/grpo_qwen3_30ba3b_instruct.yaml | 37 -- examples/swe_bench/README.md | 85 --- .../swe_bench/grpo_qwen3_30b_async_swe.yaml | 436 -------------- .../swe_bench/run_grpo_qwen3_30b_async_swe.sh | 560 ------------------ swe_nano.env | 9 +- swe_nano_interactive.sh | 37 -- swe_nano_sc_capture.sh | 2 +- swe_nano_sc_capture_interactive.sh | 4 +- swe_nano_sc_interactive.sh | 2 +- swe_nano_tq_interactive.sh | 35 -- 15 files changed, 49 insertions(+), 1257 deletions(-) delete mode 100644 examples/configs/ultra/nano_swe_teacher_sync_tq.yaml delete mode 100644 examples/swe_bench/README.md delete mode 100644 examples/swe_bench/grpo_qwen3_30b_async_swe.yaml delete mode 100644 examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh delete mode 100644 swe_nano_interactive.sh delete mode 100644 swe_nano_tq_interactive.sh diff --git a/docs/guides/nano-swe-token-capture.md b/docs/guides/nano-swe-token-capture.md index ba035d36885..8a13f17ea3d 100644 --- a/docs/guides/nano-swe-token-capture.md +++ b/docs/guides/nano-swe-token-capture.md @@ -23,7 +23,7 @@ Run on 6 GB200 NVL72 nodes: | Config | `examples/configs/ultra/nano_swe_teacher_sc.yaml` | | Model | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` | | Shape | train 4 nodes (TP2·PP2·CP4, EP1) / gen 2 nodes (vLLM TP2 → 4 engines) | -| Capture flags | `+token_capture.enabled=true` (hydra append), `NG_TIC_FP_CANONICAL=1` | +| Capture flags | `token_capture.enabled=true`, `NG_TIC_FP_CANONICAL=1` | | Outcome | 15/15 steps, `token_in_rate ≈ 0.9999`, zero `empty_manifest`, zero `capture_failed`, `valid_seqs = GBS` | | Launched via | `swe_nano_sc_capture.sh` (batch) / `swe_nano_sc_capture_interactive.sh` (attach + iterate) | @@ -163,7 +163,7 @@ its absence broke a run: | Setting | Without it | |---|---| -| `+token_capture.enabled=true` (hydra append) | Capture never engages. It is a `+` append because no yaml in the nano chain defines the key; a plain override is rejected. | +| `token_capture.enabled=true` | Capture never engages. The `token_capture` block (with `enabled: false` and all defaults documented) lives in `nano_swe_teacher_sc.yaml`; the launcher flips the one flag. | | `NG_TIC_FP_CANONICAL=1` | Token-in silently degrades to ~0: reasoning models echo history with `` blocks stripped, the gate's fingerprint of the served turn never matches, and every call falls back to text mode. With it, `token_in_rate ≈ 0.9999`. | | `NRL_DRIVER_PYTHONPATH=/opt/nemo-rl/3rdparty/Gym-workspace/Gym` | Driver `ModuleNotFoundError: nemo_gym` — the driver imports the staging record schema, and the baked driver venv has no nemo_gym. | | `NRL_DRIVER_PIP_INSTALL=orjson` | Driver `ModuleNotFoundError: orjson` — Gym's `token_id_capture/__init__` eagerly imports the store. | diff --git a/docs/guides/nano-swe-transferqueue.md b/docs/guides/nano-swe-transferqueue.md index 2fa5d1709f5..7f39bc14bbf 100644 --- a/docs/guides/nano-swe-transferqueue.md +++ b/docs/guides/nano-swe-transferqueue.md @@ -67,12 +67,6 @@ whenever you expect to change anything: ```bash # async GRPO via SingleController + honoured TransferQueue (the verified recipe) bash swe_nano_sc_interactive.sh - -# sync GRPO + honoured TransferQueue (simplest data-plane smoke) -bash swe_nano_tq_interactive.sh - -# async GRPO, no data plane (baseline) -bash swe_nano_interactive.sh ``` The batch and interactive paths build the identical driver command; batch just @@ -132,9 +126,6 @@ nano_swe_teacher_sc.yaml SingleController + TQ (the recipe) └── _nano_smoke_gb200.yaml.inc Nano/GB200 mesh + smoke sizing ``` -`nano_swe_teacher_sync_tq.yaml` is the sync variant, branching off -`nano_swe_teacher_qwen3mesh.yaml`. - ### Why each SingleController-specific setting is there Every one of these came from a crash on the way to the working run: diff --git a/examples/configs/ultra/nano_swe_teacher_qwen3mesh.yaml b/examples/configs/ultra/nano_swe_teacher_qwen3mesh.yaml index 6d177bf0251..29c3593b8f5 100644 --- a/examples/configs/ultra/nano_swe_teacher_qwen3mesh.yaml +++ b/examples/configs/ultra/nano_swe_teacher_qwen3mesh.yaml @@ -54,7 +54,7 @@ policy: # async trainer builds the in-memory ReplayBuffer (grpo.py:3898), not the # TQ-backed TQReplayBuffer (async_utils/replay_buffer.py:638). # run_grpo_nemo_gym.py + async_grpo.enabled=false -> honoured by grpo_train_sync -# (see nano_swe_teacher_sync_tq.yaml). +# (sync path; this recipe uses the SingleController below). # run_grpo_single_controller.py -> honoured, and REQUIRED # (the entrypoint raises when data_plane.enabled is false). This is the # async + TQ combination — see nano_swe_teacher_sc.yaml. diff --git a/examples/configs/ultra/nano_swe_teacher_sc.yaml b/examples/configs/ultra/nano_swe_teacher_sc.yaml index 85d657e72a3..630670f33f0 100644 --- a/examples/configs/ultra/nano_swe_teacher_sc.yaml +++ b/examples/configs/ultra/nano_swe_teacher_sc.yaml @@ -56,3 +56,41 @@ loss_fn: # base sets kl_penalty=0 → no reference model → AttributeError 'reference_state_dict'. # Enable a small KL (matches grpo_math_1B_megatron_single_controller.yaml). reference_policy_kl_penalty: 0.01 + +# Gate-authoritative token capture (token-in/token-out via NeMo-Gym; see +# docs/guides/nano-swe-token-capture.md). Dormant by default: with +# enabled=false every legacy codepath behaves exactly as before — flip to +# true (or override token_capture.enabled=true) for the capture arm. +# Requires this SingleController path with data_plane.enabled=true and the +# async vLLM backend; defaults live on TokenCaptureConfig +# (nemo_rl/algorithms/single_controller_utils/config.py). +token_capture: + enabled: false + # TQ partition holding per-call staged token deltas (finalizer-cleared). + staging_partition: "rollout_staging" + # continue: a failed worker-side stage poisons the rollout (placeholder row); + # abort: fails the whole rollout at the gate. + on_capture_failure: "continue" + # allow: train groups whose calls span a refit (staleness = group's oldest + # call version); reject: placeholder such rollouts. + mixed_weight_version_policy: "allow" + # Drop the whole group when fewer than this fraction of its rollouts + # produced valid rows (null keeps every group). + min_valid_fraction_per_group: null + # Gate-side cleanup backstops (seconds). + registration_ttl_s: 3600.0 + staging_ttl_s: 3600.0 + # Gym LineageIndex capacity (each in-flight rollout's cumulative tokens + # live in the gate process). null = derived from the training config: + # rollouts = 2 x max in-flight, tokens = rollouts x max sequence length. + # Size explicitly for agentic workloads with deep per-rollout call trees. + lineage_max_rollouts: null + lineage_max_tokens: null + # Bearer token for the gate's /ng-control/* routes. null = minted per run. + control_auth_token: null + # Hard deadline per control-plane call (gate death must surface as failed + # dispatches + placeholders, not a silent retry stall). + control_timeout_s: 60.0 + # Directory for the Gym base capture layer the gate rides on (stays + # essentially empty on the gate path). null = /gym_token_capture. + capture_dir: null diff --git a/examples/configs/ultra/nano_swe_teacher_sync_tq.yaml b/examples/configs/ultra/nano_swe_teacher_sync_tq.yaml deleted file mode 100644 index a875fe93f3a..00000000000 --- a/examples/configs/ultra/nano_swe_teacher_sync_tq.yaml +++ /dev/null @@ -1,46 +0,0 @@ -# ============================================================================= -# Nano v3 30B-A3B SWE — SYNC GRPO + TransferQueue data plane (GB200) -# ============================================================================= -# Purpose: exercise the TransferQueue data plane (data_plane.impl=transfer_queue) -# under the SYNC trainer (grpo_train_sync). Of the two run_grpo_nemo_gym.py -# paths, only the sync one puts rollouts through TQ — async_grpo_train uses an -# in-memory ReplayBuffer — so this variant DISABLES async_grpo. -# -# Inherits the nano Qwen3-mesh recipe (TP2·PP2·CP4, vLLM TP2, 6-node shape) and -# only changes the trainer + data plane. -# -# VERIFIED: reached "🚀 Running synchronous GRPO training" → Step 1 (rewards → -# advantages → logprobs → forward/backward) on the 6-node nano shape, with a -# 64-rollout SWE batch taking ~11 min. -# -# Sync leaves the 16 training GPUs idle during that rollout, which is why the -# production SWE path is async. For async + honoured TQ use -# nano_swe_teacher_sc.yaml (SingleController) instead; this file is the simplest -# way to smoke-test the data plane itself. -# ============================================================================= - -defaults: - - nano_swe_teacher_qwen3mesh.yaml - -grpo: - async_grpo: - enabled: false # REQUIRED: TQ data plane runs under grpo_train_sync - -# TransferQueue-mediated bulk transfer between rollout and train. -# See nemo_rl/data_plane/factory.py. 'simple' = CPU backend (no mooncake). -data_plane: - enabled: true - impl: transfer_queue - backend: "simple" - storage_capacity: 1000000 - num_storage_units: 2 - claim_meta_poll_interval_s: 0.5 - # Required by DataPlaneConfig (ints, no defaults). Only read when - # backend == "mooncake_cpu"; the "simple" backend ignores them. - global_segment_size: 549755813888 # 512 GiB - local_buffer_size: 68719476736 # 64 GiB - -policy: - generation: - vllm_cfg: - async_engine: true # keep vLLM async engine (per request); independent of async_grpo diff --git a/examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml b/examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml index 67cf722c662..68ff288e2a4 100644 --- a/examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml +++ b/examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml @@ -172,40 +172,3 @@ checkpointing: # 4. The checkpoint time for this model is around 10 mins. checkpoint_must_save_by: "00:03:30:00" save_period: 1 - -# Gate-authoritative token capture (token-in/token-out via NeMo-Gym; see -# docs/design-docs/tq-gym-tokidcap-migration.md). Dormant by default: with -# enabled=false every legacy codepath behaves exactly as before. Requires the -# SingleController path with data_plane.enabled=true and the async vLLM -# backend; defaults live on TokenCaptureConfig -# (nemo_rl/algorithms/single_controller_utils/config.py). -token_capture: - enabled: false - # TQ partition holding per-call staged token deltas (finalizer-cleared). - staging_partition: "rollout_staging" - # continue: a failed worker-side stage poisons the rollout (placeholder row); - # abort: fails the whole rollout at the gate. - on_capture_failure: "continue" - # allow: train groups whose calls span a refit (staleness = group's oldest - # call version); reject: placeholder such rollouts. - mixed_weight_version_policy: "allow" - # Drop the whole group when fewer than this fraction of its rollouts - # produced valid rows (null keeps every group). - min_valid_fraction_per_group: null - # Gate-side cleanup backstops (seconds). - registration_ttl_s: 3600.0 - staging_ttl_s: 3600.0 - # Gym LineageIndex capacity (each in-flight rollout's cumulative tokens - # live in the gate process). null = derived from the training config: - # rollouts = 2 x max in-flight, tokens = rollouts x max sequence length. - # Size explicitly for agentic workloads with deep per-rollout call trees. - lineage_max_rollouts: null - lineage_max_tokens: null - # Bearer token for the gate's /ng-control/* routes. null = minted per run. - control_auth_token: null - # Hard deadline per control-plane call (gate death must surface as failed - # dispatches + placeholders, not a silent retry stall). - control_timeout_s: 60.0 - # Directory for the Gym base capture layer the gate rides on (stays - # essentially empty on the gate path). null = /gym_token_capture. - capture_dir: null diff --git a/examples/swe_bench/README.md b/examples/swe_bench/README.md deleted file mode 100644 index 3f30f5f4520..00000000000 --- a/examples/swe_bench/README.md +++ /dev/null @@ -1,85 +0,0 @@ -# Async GRPO on SWE-bench (Qwen3-30B-A3B) - -Launcher and recipe for agentic RL on SWE-bench via NeMo-Gym/OpenHands: -16 nodes by default (8 training + 8 generation, non-colocated), async GRPO -with staleness window 1. - -| File | Purpose | -|---|---| -| `run_grpo_qwen3_30b_async_swe.sh` | SLURM launcher (submits `ray.sub`) | -| `grpo_qwen3_30b_async_swe.yaml` | Recipe config | - -The launcher supports two entrypoints: - -- `SC_MODE=1` (default): `examples/run_grpo_single_controller.py` — - single-controller with the TransferQueue data plane. -- `SC_MODE=0`: `examples/nemo_gym/run_grpo_nemo_gym.py` — classic async GRPO - (async behavior comes from the yaml's `grpo.async_grpo` block). - -## Prerequisites - -1. A NeMo-RL checkout (or code snapshot) containing `ray.sub` at its root — - the launcher submits from the root it lives under. -2. An enroot container image of a recent NeMo-RL nightly. -3. The SWE train dataset (`.jsonl`). -4. The SWE sandbox images (apptainer/singularity `.sif` files for the - swe-bench / sweap instances). **Edit the `container_formatter` lists in the - yaml** to point at your local copies (they ship as `/path/to/...` - placeholders). -5. A HF checkpoint to train from. - -## Quick start - -Run from the repo root: - -```bash -ACCOUNT= \ -CONTAINER=/path/to/nemo-rl-nightly.sqsh \ -MODEL_PATH=/path/to/hf_checkpoint \ -TRAIN_DATA_PATH=/path/to/swe_train.jsonl \ -EXTRA_MOUNTS=/path/to/shared_fs:/path/to/shared_fs \ -bash examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh -``` - -`EXTRA_MOUNTS` must make the model / data / `.sif` locations visible inside -the container (the default mounts only cover the repo tree and the gym -source). Add `DRY_RUN=1` to print the sbatch command and config without -submitting. - -Secrets (`WANDB_API_KEY`, `HUGGINGFACE_TOKEN`, ...) are read from the calling -environment and never stored in this directory. The recommended pattern is a -small personal wrapper script that exports your site-specific paths, cluster -account, and secrets, then calls this launcher. - -## Common variations - -```bash -# Classic (non-single-controller) entrypoint -SC_MODE=0 bash examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh - -# Different training tensor parallelism -TP=2 bash examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh - -# streaming-2 dispatch semantics (over-generation + out-of-order consumption; -# default is streaming-1: strict repro of classic async_grpo age-1 dispatch) -OVER_SAMPLING=true FORCE_IN_ORDER=false \ -bash examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh - -# Finer intra-step dispatch: start training once 2 prompt groups are ready -# instead of waiting for the full batch (SC only; PPS=8 by default) -MIN_PROMPT_GROUPS=2 bash examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh - -# Smaller / shorter run -NUM_NODES=8 NUM_GEN_NODES=4 TIME=2:0:0 MAX_NUM_STEPS=5 \ -bash examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh -``` - -The full knob list (parallelism, batch sizes, staleness, agent turn limits, -caches, ...) is documented in the launcher's header comment. - -## Monitoring - -The submit banner prints the experiment name, W&B target, and log location. -The driver log lands at `/logs/slurm/-logs/ray-driver.log`; -training progress appears there as `train step ...` / `step_metrics=...` -lines and in W&B under `train/reward` and `timing/train/*`. diff --git a/examples/swe_bench/grpo_qwen3_30b_async_swe.yaml b/examples/swe_bench/grpo_qwen3_30b_async_swe.yaml deleted file mode 100644 index aa774156245..00000000000 --- a/examples/swe_bench/grpo_qwen3_30b_async_swe.yaml +++ /dev/null @@ -1,436 +0,0 @@ -# ============================================================================ -# Async GRPO SWE RL Training: Qwen3-30B-A3B-Thinking-2507 -# -# Model: Qwen3-30B-A3B-Thinking-2507 (MoE, 30B total / 3B active, thinking) -# Train data: R2E-Gym (r2e-gym subset, 4518 samples) -# Eval data: SWE-bench Verified -# Mode: Async GRPO with non-colocated generation -# Entry: examples/nemo_gym/run_grpo_nemo_gym.py -# Env: swe_agents (OpenHands agent, singularity sandbox) -# -# Based on: bihu/nemo-rl-qwen-swe/grpo_qwen3_30b_thinking_swe.yaml -# Gym: main branch (nemo-rl-async-swe repo) -# ============================================================================ - -checkpointing: - enabled: true - checkpoint_dir: "results/grpo-qwen3-30b-thinking-swe-rl" - metric_name: "train:total_reward/mean" - higher_is_better: true - keep_top_k: 100 - save_period: 5 - checkpoint_must_save_by: "00:03:35:00" - model_save_format: "safetensors" - save_consolidated: false - save_optimizer: true - -grpo: - num_prompts_per_step: 16 - num_generations_per_prompt: 16 - num_val_generations_per_prompt: 1 - max_rollout_turns: 1 - max_num_epochs: 100 - max_num_steps: 1000000 - normalize_rewards: true - use_leave_one_out_baseline: true - advantage_clip_low: -100 - advantage_clip_high: 100 - val_period: 10 - val_at_start: false - val_at_end: false - overlong_filtering: true - max_val_samples: null - val_batch_size: 256 - seed: 42 - invalid_tool_call_strategy: "" - - use_dynamic_sampling: false - dynamic_sampling_max_gen_batches: 10 - batch_multiplier: 1 - - penalize_invalid_tool_call: true - invalid_tool_call_advantage: -5.0 - penalize_malformed_thinking: true - malformed_thinking_advantage: -5.0 - - reward_shaping: - enabled: false - overlong_buffer_length: 128 - overlong_buffer_penalty: 1 - max_response_length: ${policy.max_total_sequence_length} - stop_properly_penalty_coef: null - reward_scaling: - enabled: false - source_min: 0.0 - source_max: 1.0 - target_min: 0.0 - target_max: 1.0 - - async_grpo: - enabled: true - max_trajectory_age_steps: 1 - in_flight_weight_updates: true - recompute_kv_cache_after_weight_updates: false - - seq_logprob_error_threshold: 2 - -loss_fn: - reference_policy_kl_penalty: 0.0 - reference_policy_kl_type: "k3" - kl_input_clamp_value: null - kl_output_clamp_value: null - ratio_clip_min: 0.2 - ratio_clip_max: 0.28 - ratio_clip_c: null - use_on_policy_kl_approximation: true - use_importance_sampling_correction: true - truncated_importance_sampling_ratio: 5.0 - truncated_importance_sampling_ratio_min: null - truncated_importance_sampling_type: tis - sequence_level_importance_ratios: false - token_level_loss: true - force_on_policy_ratio: true - use_kl_in_reward: false - -policy: - model_name: "Qwen/Qwen3-30B-A3B-Thinking-2507" - tokenizer: - name: ${policy.model_name} - chat_template_kwargs: - enable_thinking: true - hf_config_overrides: {} - train_global_batch_size: 256 - train_micro_batch_size: 1 - generation_batch_size: 64 - logprob_batch_size: 1 - max_total_sequence_length: 131072 - precision: "bfloat16" - logprob_chunk_size: 2048 - offload_optimizer_for_logprob: false - - dtensor_cfg: - _v2: true - enabled: false - cpu_offload: False - sequence_parallel: false - activation_checkpointing: false - tensor_parallel_size: 1 - context_parallel_size: 1 - custom_parallel_plan: null - - megatron_cfg: - enabled: true - gradient_accumulation_fusion: false - empty_unused_memory_level: 1 - activation_checkpointing: true - tensor_model_parallel_size: 2 - expert_tensor_parallel_size: 1 - expert_model_parallel_size: 8 - pipeline_model_parallel_size: 2 - num_layers_in_first_pipeline_stage: null - num_layers_in_last_pipeline_stage: null - context_parallel_size: 4 - pipeline_dtype: ${policy.precision} - sequence_parallel: true - freeze_moe_router: true - moe_router_dtype: "fp32" - moe_router_load_balancing_type: "none" - moe_router_bias_update_rate: 1.0e-3 - moe_permute_fusion: true - moe_enable_deepep: false - moe_token_dispatcher_type: "alltoall" - moe_aux_loss_coeff: 0.0 - moe_router_enable_expert_bias: true - moe_shared_expert_overlap: false - apply_rope_fusion: True - bias_activation_fusion: False - defer_fp32_logits: True - moe_per_layer_logging: True - - optimizer: - optimizer: "adam" - lr: 1.0e-6 - min_lr: 1.0e-6 - weight_decay: 0.0 - bf16: true - fp16: false - params_dtype: "float32" - adam_beta1: 0.9 - adam_beta2: 0.999 - adam_eps: 1e-8 - sgd_momentum: 0.9 - use_distributed_optimizer: true - use_precision_aware_optimizer: true - clip_grad: ${policy.max_grad_norm} - optimizer_cpu_offload: false - optimizer_offload_fraction: 0.0 - - scheduler: - start_weight_decay: ${policy.megatron_cfg.optimizer.weight_decay} - end_weight_decay: ${policy.megatron_cfg.optimizer.weight_decay} - weight_decay_incr_style: "constant" - lr_decay_style: "constant" - lr_decay_iters: 1000000 - lr_warmup_iters: 0 - lr_warmup_init: 0 - - distributed_data_parallel_config: - grad_reduce_in_fp32: false - overlap_grad_reduce: false - overlap_param_gather: false - use_custom_fsdp: false - data_parallel_sharding_strategy: "optim_grads_params" - - mtp_loss_scaling_factor: 0.0 - mtp_use_repeated_layer: false - mtp_num_layers: 0 - mtp_detach_heads: false - - fp8_cfg: - enabled: false - fp8: "e4m3" - fp8_recipe: "blockwise" - fp8_param: false - - env_vars: null - - dynamic_batching: - enabled: False - train_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.train_micro_batch_size}} - logprob_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.logprob_batch_size}} - sequence_length_round: 64 - - sequence_packing: - enabled: True - train_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.train_micro_batch_size}} - logprob_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.logprob_batch_size}} - algorithm: "modified_first_fit_decreasing" - sequence_length_round: 64 - - make_sequence_length_divisible_by: 8 - max_grad_norm: 1.0 - - optimizer: null - scheduler: null - - generation: - port_range_low: 11001 - port_range_high: 15000 - backend: "vllm" - max_new_tokens: ${policy.max_total_sequence_length} - temperature: 1.0 - top_p: 1.0 - top_k: null - stop_token_ids: null - stop_strings: null - vllm_cfg: - enable_prefix_caching: true - async_engine: true - precision: ${policy.precision} - kv_cache_dtype: "auto" - tensor_parallel_size: 2 - pipeline_parallel_size: 1 - expert_parallel_size: 1 - gpu_memory_utilization: 0.8 - max_model_len: ${policy.max_total_sequence_length} - enforce_eager: False - enforce_monotonicity: false - use_deep_gemm: False - num_last_layers_in_bf16: 0 - num_first_layers_in_bf16: 0 - enable_vllm_metrics_logger: true - vllm_metrics_logger_interval: 0.5 - expose_http_server: true - skip_tokenizer_init: false - enable_thinking: true - http_server_serving_chat_kwargs: - enable_auto_tools: true - tool_parser: hermes - reasoning_parser: deepseek_r1 - chat_template: | - {%- if tools %} - {{- '<|im_start|>system\n' }} - {%- if messages[0].role == 'system' %} - {{- messages[0].content + '\n\n' }} - {%- endif %} - {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n" }} - {%- for tool in tools %} - {{- "\n" }} - {{- tool | tojson }} - {%- endfor %} - {{- "\n\n\nFor each function call, return a json object with function name and arguments within XML tags:\n\n{\"name\": , \"arguments\": }\n<|im_end|>\n" }} - {%- else %} - {%- if messages[0].role == 'system' %} - {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }} - {%- endif %} - {%- endif %} - {%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %} - {%- for message in messages[::-1] %} - {%- set index = (messages|length - 1) - loop.index0 %} - {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %} - {%- set ns.multi_step_tool = false %} - {%- set ns.last_query_index = index %} - {%- endif %} - {%- endfor %} - {%- for message in messages %} - {%- if message.content is string %} - {%- set content = message.content %} - {%- else %} - {%- set content = '' %} - {%- endif %} - {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} - {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }} - {%- elif message.role == "assistant" %} - {%- set reasoning_content = '' %} - {%- if message.reasoning_content is string %} - {%- set reasoning_content = message.reasoning_content %} - {%- else %} - {%- if '' in content %} - {%- set reasoning_content = content.split('')[0].rstrip('\n').split('')[-1].lstrip('\n') %} - {%- set content = content.split('')[-1].lstrip('\n') %} - {%- endif %} - {%- endif %} - {%- if reasoning_content %} - {{- '<|im_start|>' + message.role + '\n\n' + reasoning_content.strip('\n') + '\n\n\n' + content.lstrip('\n') }} - {%- else %} - {{- '<|im_start|>' + message.role + '\n' + content }} - {%- endif %} - {%- if message.tool_calls %} - {%- for tool_call in message.tool_calls %} - {%- if (loop.first and content) or (not loop.first) %} - {{- '\n' }} - {%- endif %} - {%- if tool_call.function %} - {%- set tool_call = tool_call.function %} - {%- endif %} - {{- '\n{"name": "' }} - {{- tool_call.name }} - {{- '", "arguments": ' }} - {%- if tool_call.arguments is string %} - {{- tool_call.arguments }} - {%- else %} - {{- tool_call.arguments | tojson }} - {%- endif %} - {{- '}\n' }} - {%- endfor %} - {%- endif %} - {{- '<|im_end|>\n' }} - {%- elif message.role == "tool" %} - {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %} - {{- '<|im_start|>user' }} - {%- endif %} - {{- '\n\n' }} - {{- content }} - {{- '\n' }} - {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %} - {{- '<|im_end|>\n' }} - {%- endif %} - {%- endif %} - {%- endfor %} - {%- if add_generation_prompt %} - {{- '<|im_start|>assistant\n\n' }} - {%- endif %} - default_chat_template_kwargs: - enable_thinking: true - truncate_history_thinking: false - - vllm_kwargs: - mamba_ssm_cache_dtype: "float32" - # The default flashinfer MoE backend (trtllm_bf16_moe) uses a fused/quantized - # expert weight layout that does NOT match the megatron weight-refit broadcast, - # producing garbage generations (and can hang broadcast_weights_for_collective). - # triton uses the standard expert layout the refit broadcast expects. - # Mirrors ruit's grpo_qwen3_30b_async_swe_hsg.yaml on the swe2-scale-gen branch. - moe_backend: triton - compilation_config: - cudagraph_capture_sizes: [1,2,4,8,16,32,64] - - colocated: - enabled: false - resources: - gpus_per_node: 8 - num_nodes: 4 - -data: - max_input_seq_length: null - shuffle: false - num_workers: 1 - use_multiple_dataloader: false - train: - # Site-specific; the launcher always overrides this via ++data.train.data_path. - data_path: "/path/to/swe_train_dataset.jsonl" - validation: - # Site-specific; the launcher always overrides this via ++data.validation.data_path. - data_path: "/path/to/swe_val_dataset.jsonl" - default: - dataset_name: NemoGymDataset - env_name: "nemo_gym" - prompt_file: null - system_prompt_file: null - processor: "nemo_gym_data_processor" - -env: - should_use_nemo_gym: true - should_log_nemo_gym_responses: false - nemo_gym: - skip_venv_if_present: true - port_range_low: 15001 - port_range_high: 20000 - config_paths: - - responses_api_models/vllm_model/configs/vllm_model_for_training.yaml - - responses_api_agents/swe_agents/configs/swebench_openhands_training.yaml - swe_agents_train: - responses_api_agents: - swe_agents: - agent_max_turns: 100 - concurrency: 768 - swebench_agent_timeout: 3600 - run_with_mixed_prompts: true - dataset_path: ${data.train.data_path} - # Site-specific: apptainer/singularity .sif sandbox images for the SWE - # datasets, tried in order per instance. Point these at your local - # copies of the swe-bench / sweap image sets. - container_formatter: - - "/path/to/swe-bench-images/swebench_sweb.eval.x86_64.{instance_id}.sif" - - "/path/to/sweap-images/sweap.{instance_id}.sif" - - "/path/to/swe-bench-images/namanjain12_{instance_id}.sif" - - "/path/to/sweap-images/sweap.{instance_id}.sif" - - "/path/to/swe-bench-images/swebench_sweb.eval.x86_64{instance_id}.sif" - swe_agents_val: - responses_api_agents: - swe_agents: - agent_max_turns: 200 - concurrency: 768 - swebench_agent_timeout: 3600 - dataset_path: ${data.validation.data_path} - # Site-specific: see the note on swe_agents_train.container_formatter. - container_formatter: - - "/path/to/swe-bench-images/swebench_sweb.eval.x86_64.{instance_id}.sif" - - "/path/to/sweap-images/sweap.{instance_id}.sif" - - "/path/to/swe-bench-images/namanjain12_{instance_id}.sif" - - "/path/to/sweap-images/sweap.{instance_id}.sif" - - "/path/to/swe-bench-images/swebench_sweb.eval.x86_64{instance_id}.sif" - use_absolute_ip: true - -logger: - log_dir: "logs" - num_val_samples_to_print: 0 - wandb_enabled: true - tensorboard_enabled: false - mlflow_enabled: false - monitor_gpus: true - swanlab_enabled: false - wandb: - project: "ruit-nemo-rl" - name: "qwen3-30b-thinking-swe-rl" - tensorboard: {} - mlflow: - experiment_name: "qwen3-30b-thinking-swe-rl" - run_name: "qwen3-30b-thinking-swe-rl" - gpu_monitoring: - collection_interval: 10 - flush_interval: 10 - -cluster: - gpus_per_node: 8 - num_nodes: 16 diff --git a/examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh b/examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh deleted file mode 100644 index 182274a3c5a..00000000000 --- a/examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh +++ /dev/null @@ -1,560 +0,0 @@ -#!/bin/bash -# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================= -# Async GRPO on SWE-bench (NeMo-Gym / OpenHands) — Qwen3-30B-A3B, SLURM launcher. -# -# Submits a 16-node (8 train + 8 generation, non-colocated) async GRPO run via -# ray.sub. Supports both entrypoints: -# SC_MODE=1 (default) examples/run_grpo_single_controller.py -# (single-controller + TransferQueue data plane) -# SC_MODE=0 examples/nemo_gym/run_grpo_nemo_gym.py -# (classic async GRPO; async comes from the yaml's -# grpo.async_grpo block) -# -# This file is environment-agnostic: no secrets and no user-specific paths. -# Site-specific values come from the environment (see REQUIRED below) — wrap -# this script with your own launcher that exports them (see -# test_assets/SWE/grpo_swe_tests.sh for an example wrapper). -# -# --------------------------------------------------------------------------- -# REQUIRED environment: -# ACCOUNT SLURM account -# CONTAINER enroot .sqsh/.squashfs image (a recent NeMo-RL nightly; -# must postdate the TransferQueue pyproject dependency) -# MODEL_PATH HF checkpoint dir to train from -# TRAIN_DATA_PATH SWE train jsonl -# -# Common optional environment (see defaults inline for the full list): -# PARTITION (batch), NUM_NODES (16), NUM_GEN_NODES (8), TIME (4:0:0) -# VAL_DATA_PATH (=TRAIN_DATA_PATH), CONFIG_FILE (yaml next to this script) -# TP/EP/CP/PP/VLLM_TP, SEQLEN, PPS/GPP/GBS, LR, MAX_NUM_STEPS -# SC_MODE (1), MIN_PROMPT_GROUPS (=PPS) -# OVER_SAMPLING (false), FORCE_IN_ORDER (true) -# streaming-1 (default): OVER_SAMPLING=false FORCE_IN_ORDER=true -# — no over-generation, each step consumes the groups dispatched -# for it; 1:1 repro of the classic async_grpo (age=1) dispatch. -# streaming-2: OVER_SAMPLING=true FORCE_IN_ORDER=false -# — generation keeps producing, steps consume any groups within -# the staleness window; stale groups get evicted (wasted). -# PERSISTENT_CACHE compile/uv cache root on shared fs (~/.cache/... default) -# GYM_VENV_DIR /tmp/nemo_gym_venvs default; /opt/gym_venvs if your image -# has the gym server venvs baked in -# MOUNTS / EXTRA_MOUNTS container mounts (make model/data paths visible!) -# WANDB_API_KEY + WANDB_PROJ + EXP_SUFFIX logging & naming -# HUGGINGFACE_TOKEN / GITHUB_TOKEN / GITLAB_TOKEN passed through if set -# REAPER_COMMENT SLURM --comment payload (cluster-specific; empty default) -# DRY_RUN=1 print everything, submit nothing -# -# Usage (from a repo/snapshot root that contains ray.sub): -# ACCOUNT=... CONTAINER=... MODEL_PATH=... TRAIN_DATA_PATH=... \ -# bash examples/swe_bench/run_grpo_qwen3_30b_async_swe.sh -# ============================================================================= - -set -e - -# --------------------------------------------------------------------------- -# Locate the tree to submit from: this script lives at examples/swe_bench/ -# inside the repo (or a code snapshot); ray.sub must exist at its root. -# --------------------------------------------------------------------------- -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -RUN_DIR="${RUN_DIR:-$(cd "${SCRIPT_DIR}/../.." && pwd)}" -if [ ! -f "${RUN_DIR}/ray.sub" ]; then - echo "Error: ${RUN_DIR}/ray.sub not found — RUN_DIR must be a NeMo-RL checkout/snapshot root." >&2 - exit 1 -fi -RUN_COMMIT="unknown" -if git -C "${RUN_DIR}" rev-parse --short HEAD >/dev/null 2>&1; then - RUN_COMMIT="$(git -C "${RUN_DIR}" rev-parse --short HEAD)" -elif [ -f "${RUN_DIR}/commit.txt" ]; then - RUN_COMMIT="$(cut -c1-7 "${RUN_DIR}/commit.txt")" -fi - -# --------------------------------------------------------------------------- -# Required site-specific inputs — fail fast with a clear message. -# --------------------------------------------------------------------------- -missing="" -[ -n "${ACCOUNT:-}" ] || missing+=" ACCOUNT" -[ -n "${CONTAINER:-}" ] || missing+=" CONTAINER" -[ -n "${MODEL_PATH:-}" ] || missing+=" MODEL_PATH" -[ -n "${TRAIN_DATA_PATH:-}" ] || missing+=" TRAIN_DATA_PATH" -if [ -n "${missing}" ]; then - echo "Error: missing required environment variables:${missing}" >&2 - echo "See the header of this script for the full contract." >&2 - exit 1 -fi - -# --------------------------------------------------------------------------- -# Cluster / submission (site-specific, all overridable) -# --------------------------------------------------------------------------- -PARTITION="${PARTITION:-batch}" -GPUS_PER_NODE="${GPUS_PER_NODE:-8}" -CPUS_PER_WORKER="${CPUS_PER_WORKER:-114}" -# SLURM --comment payload (e.g. idle-GPU-reaper exemptions on some clusters). -REAPER_COMMENT="${REAPER_COMMENT:-}" - -# --------------------------------------------------------------------------- -# Scale / walltime -# --------------------------------------------------------------------------- -NUM_NODES="${NUM_NODES:-16}" # total allocation -NUM_GEN_NODES="${NUM_GEN_NODES:-8}" # carved out of NUM_NODES for generation -TIME="${TIME:-4:0:0}" -MAX_NUM_STEPS="${MAX_NUM_STEPS:-}" # empty => use the yaml's value - -# --------------------------------------------------------------------------- -# Paths -# --------------------------------------------------------------------------- -CONFIG_FILE="${CONFIG_FILE:-${SCRIPT_DIR}/grpo_qwen3_30b_async_swe.yaml}" -VAL_DATA_PATH="${VAL_DATA_PATH:-${TRAIN_DATA_PATH}}" - -# ============================ Parallelism ============================ -TP="${TP:-4}"; EP="${EP:-8}"; CP="${CP:-4}"; PP="${PP:-2}"; VLLM_TP="${VLLM_TP:-2}" -MIN_PAD=1 -[ "${CP}" -gt 1 ] && MIN_PAD=$((MIN_PAD * CP * 2)) -[ "${TP}" -gt 1 ] && MIN_PAD=$((MIN_PAD * TP)) -MAKE_SEQ_DIVISIBLE_BY="${MIN_PAD}" -SEQUENCE_PACKING=True - -# ===================== Sequence length ===================== -SEQLEN="${SEQLEN:-131072}" - -# ===== Single-controller async-RL knobs (SC_MODE=1: data_plane + async_rl) ===== -# Maps the classic async_grpo (age=1) 1:1 to async_rl. -MAX_TRAJECTORY_AGE_STEPS="${MAX_TRAJECTORY_AGE_STEPS:-1}" # -> async_rl.max_weight_staleness_versions -BATCH_SELECTION_STRATEGY=staleness_window -# Dispatch semantics ("streaming" modes). Defaults = strict 1:1 repro of the -# classic async_grpo age-1 behavior (streaming-1): no over-generation, rollouts -# consumed strictly by their dispatch-target step. streaming-2 = the code -# defaults (over_sampling=true force_in_order=false): generation keeps -# producing (stale groups get evicted/wasted) and steps consume any groups -# inside the staleness window, out of order. -OVER_SAMPLING="${OVER_SAMPLING:-false}" -FORCE_IN_ORDER="${FORCE_IN_ORDER:-true}" -FORCE_ON_POLICY_RATIO=True -SEQ_LOGPROB_ERROR_THRESHOLD=null -COLOCATED_ENABLED=False -VLLM_GPU_UTIL=0.8 -OVERLAP_GRAD_REDUCE=False -ADVANTAGE_CLIP_LOW=-100 -ADVANTAGE_CLIP_HIGH=100 -TIS_THRESHOLD=5 - -# ========================= GRPO / sampling ========================= -PPS="${PPS:-8}"; GPP="${GPP:-8}"; GBS="${GBS:-64}" -# Intra-step dispatch granularity (SC only): the sampler releases work to the -# trainer once this many complete prompt groups are ready (gradient -# accumulation; the optimizer still steps only after PPS groups). Not passed in -# => same as PPS = fully synchronous within the step; smaller values overlap -# trainer compute with the generation tail. -MIN_PROMPT_GROUPS="${MIN_PROMPT_GROUPS:-${PPS}}" -NORMALIZE_REWARDS=True -OVERLONG_FILTERING=True -VAL_PERIOD="${VAL_PERIOD:-1000}" - -# ========================== Loss function ========================== -KL=0 -CLIP_MIN=0.2 -CLIP_MAX=0.28 -USE_ON_POLICY_KL_APPROXIMATION=True -IMPORTANCE_SAMPLING_CORRECTION=True -SEQ_LEVEL_IS=False -TOKEN_LEVEL_LOSS=True - -# ============================ Optimizer ============================ -LR="${LR:-1e-06}" - -# =============================== MoE =============================== -MOE_FREEZE_ROUTER=True -MOE_PERMUTE_FUSION=True -MOE_ENABLE_DEEPEP=False -MOE_TOKEN_DISPATCHER_TYPE="alltoall" -MOE_AUX_LOSS_COEFF=0 -MOE_ROUTER_LOAD_BALANCING_TYPE="none" -MOE_ROUTER_BIAS_UPDATE_RATE="1e-3" - -# ======================= Generation / vLLM ======================= -TEMPERATURE=1.0 - -# =================== Checkpointing & validation =================== -SAVE_PERIOD="${SAVE_PERIOD:-5}" -KEEP_TOP_K="${KEEP_TOP_K:-2}" -MUST_SAVE_BY="${MUST_SAVE_BY:-00:03:35:00}" # graceful save+exit before TIME - -# ============================ SWE agent ============================ -AGENT_MAX_TURNS="${AGENT_MAX_TURNS:-200}" -AGENT_TIMEOUT="${AGENT_TIMEOUT:-1800}" - -# ============================== Logging ============================== -WANDB_PROJ="${WANDB_PROJ:-nemo-rl-swe-bench}" -LOG_GYM_RESPONSES=true - -# ========================= Experiment naming ========================= -SYNC_MODE="async-age${MAX_TRAJECTORY_AGE_STEPS}" -EXP_SUFFIX="${EXP_SUFFIX:-swe-sc@${RUN_COMMIT}-${SYNC_MODE}-pps${PPS}-gpp${GPP}-gbs${GBS}-lr${LR}-tp${TP}}" -WANDB_NAME="${EXP_SUFFIX}" -EXP_NAME="${EXP_SUFFIX}" -CHECKPOINT_ROOT="${CHECKPOINT_ROOT:-${RUN_DIR}/results}" -CHECKPOINT_DIR="${CHECKPOINT_DIR:-${CHECKPOINT_ROOT}/${EXP_SUFFIX}}" -BASE_LOG_DIR="${BASE_LOG_DIR:-${RUN_DIR}/logs/slurm}" - -# ========================= Runtime env ========================= -# Secrets are passed through from the caller's environment; never set here. -export HUGGINGFACE_TOKEN="${HUGGINGFACE_TOKEN:-${HF_TOKEN:-}}" -export GITHUB_TOKEN="${GITHUB_TOKEN:-}" -export GITLAB_TOKEN="${GITLAB_TOKEN:-}" -export WANDB_API_KEY="${WANDB_API_KEY:-}" -export HF_HOME="${HF_HOME:-${HOME}/.cache/huggingface}" -export HF_DATASETS_CACHE="${HF_DATASETS_CACHE:-${HF_HOME}/datasets}" -# Node-local uv cache (ephemeral). Each node builds its own venv independently — -# a shared-filesystem uv cache corrupts under concurrent multi-node builds. -# Pre-warm LUSTRE_UV_CACHE once with a single process to skip long compiles. -export UV_CACHE_DIR=/tmp/uv_cache -export UV_LOCK_TIMEOUT=3600 -export RAY_DEDUP_LOGS=1 -export SSL_CERT_FILE="${SSL_CERT_FILE:-/etc/ssl/certs/ca-certificates.crt}" -export REQUESTS_CA_BUNDLE="${REQUESTS_CA_BUNDLE:-/etc/ssl/certs/ca-certificates.crt}" -export CURL_CA_BUNDLE="${CURL_CA_BUNDLE:-/etc/ssl/certs/ca-certificates.crt}" -export OMP_NUM_THREADS="${OMP_NUM_THREADS:-16}" - -# ===================== Shared-fs compile caches ===================== -# Runtime caches stay node-local (/tmp); the shared-fs copies are only seeded -# from (read) at startup and written back by a periodic sidecar, so repeat runs -# skip the ~20min triton MoE JIT. -PERSISTENT_CACHE="${PERSISTENT_CACHE:-${HOME}/.cache/nemo_rl_swe_bench}" -export LUSTRE_VLLM_CACHE="${PERSISTENT_CACHE}/vllm_compile_cache" -export LUSTRE_INDUCTOR_CACHE="${PERSISTENT_CACHE}/inductor_cache" -export LUSTRE_TRITON_CACHE="${PERSISTENT_CACHE}/triton_cache" -export LUSTRE_UV_CACHE="${PERSISTENT_CACHE}/uv_cache" -export NRL_VLLM_LOCAL_CACHE_DIR="/tmp/nemo_rl_vllm_cache" -export NRL_VLLM_CACHE_SEED_DIR="/tmp/nemo_rl_vllm_cache_warm" -export INDUCTOR_CACHE_DIR="/tmp/nemo_rl_inductor_cache" -export TRITON_CACHE_DIR="/tmp/nemo_rl_triton_cache" -export CACHE_SYNC_FREQUENCY="${CACHE_SYNC_FREQUENCY:-120}" -mkdir -p "${LUSTRE_VLLM_CACHE}" "${LUSTRE_INDUCTOR_CACHE}" "${LUSTRE_TRITON_CACHE}" "${LUSTRE_UV_CACHE}" - -# ===================== NeMo-Gym server venvs ===================== -# Default: node-local build (safe everywhere). If your container has the gym -# server venvs baked in (vllm_model + swe_agents), export GYM_VENV_DIR=/opt/gym_venvs. -# Do NOT point this at a shared filesystem: the editable nemo-gym build hangs on -# uv's flock there, and interrupted builds leave empty venv shells that -# skip-if-present then reuses -> the gym policy_model server crashes on import. -GYM_VENV_DIR="${GYM_VENV_DIR:-/tmp/nemo_gym_venvs}" -case "${GYM_VENV_DIR}" in /opt/*|/tmp/*) ;; *) mkdir -p "${GYM_VENV_DIR}" ;; esac - -# ===== SETUP_COMMAND: install apptainer + seed caches + uv sync ===== -# Runs on all nodes before Ray starts (consumed by ray.sub). -read -r -d '' SETUP_COMMAND </dev/null || true -RET=1 -RETRIES=3 -for attempt in \$(seq 1 \$RETRIES); do - if command -v apptainer >/dev/null 2>&1 || command -v singularity >/dev/null 2>&1; then - echo "[SETUP] singularity/apptainer already available" - RET=0 - break - fi - cd /tmp && \ - wget --no-check-certificate -q https://github.com/apptainer/apptainer/releases/download/v1.3.1/apptainer_1.3.1_amd64.deb && \ - apt install -y ./apptainer_1.3.1_amd64.deb && \ - ln -sf /usr/bin/apptainer /usr/bin/singularity - if command -v apptainer >/dev/null 2>&1; then - echo "[SETUP] apptainer installed successfully" - RET=0 - break - fi - echo "[SETUP] apptainer install attempt \$attempt failed, retrying..." - sleep 10 -done -if [ \$RET -ne 0 ]; then - echo "[SETUP] WARNING: apptainer installation failed after \$RETRIES attempts" -fi - -echo "[CACHE SEED] Clearing stale /tmp caches and seeding from shared fs..." -rm -rf /tmp/nemo_rl_vllm_cache /tmp/nemo_rl_vllm_cache_* -rm -rf "${INDUCTOR_CACHE_DIR}" "${TRITON_CACHE_DIR}" -mkdir -p "${INDUCTOR_CACHE_DIR}" "${TRITON_CACHE_DIR}" - -find "${LUSTRE_INDUCTOR_CACHE}" -maxdepth 1 -name '.tmp_*' -mmin +30 -exec rm -rf {} + 2>/dev/null || true -find "${LUSTRE_TRITON_CACHE}" -maxdepth 1 -name '.tmp_*' -mmin +30 -exec rm -rf {} + 2>/dev/null || true - -_seed_cache() { - local lustre="\$1" local_dir="\$2" name="\$3" - if [ -d "\$lustre" ] && [ "\$(ls -A "\$lustre" 2>/dev/null)" ]; then - rsync -a --exclude '.tmp_*' "\$lustre/" "\$local_dir/" 2>/dev/null \ - && echo "[CACHE SEED] \$name: seeded from shared fs" \ - || echo "[CACHE SEED] \$name: seed failed (non-fatal)" - else - echo "[CACHE SEED] \$name: no warm cache on shared fs yet" - fi -} - -_seed_cache "${LUSTRE_INDUCTOR_CACHE}" "${INDUCTOR_CACHE_DIR}" "Inductor" -_seed_cache "${LUSTRE_TRITON_CACHE}" "${TRITON_CACHE_DIR}" "Triton" -# uv cache: read-only rsync of single-process-prebuilt wheels into node-local -# /tmp so the uv sync below is a cache hit, not a ~28min build. -# (NOTE: no backticks/'\$(...)' in this heredoc body -- they would be command- -# substituted on the LOGIN node when SETUP_COMMAND is read.) -mkdir -p "${UV_CACHE_DIR}" -_seed_cache "${LUSTRE_UV_CACHE}" "${UV_CACHE_DIR}" "uv (prebuilt wheels)" -echo "[CACHE SEED] Done." - -# ===== Compile-cache WRITE-BACK sidecar ===== -# The seed above only READS shared fs -> /tmp. This sidecar periodically rsyncs -# /tmp -> shared fs (and on TERM/INT) so compiled kernels persist across runs; -# --ignore-existing makes concurrent per-node writes first-writer-wins. -_sync_cache_one() { - local src="\$1" dst="\$2" name="\$3" - mkdir -p "\$dst" - if [ -d "\$src" ] && [ "\$(ls -A "\$src" 2>/dev/null)" ]; then - rsync -a --ignore-existing --exclude '.tmp_*' --exclude 'tmp*' "\$src/" "\$dst/" 2>/dev/null \ - && echo "[CACHE SYNC] \$name: /tmp -> shared fs" \ - || echo "[CACHE SYNC] \$name: sync failed (non-fatal)" - fi -} -_sync_compile_caches_to_lustre() { - _sync_cache_one "${INDUCTOR_CACHE_DIR}" "${LUSTRE_INDUCTOR_CACHE}" "Inductor" - _sync_cache_one "${TRITON_CACHE_DIR}" "${LUSTRE_TRITON_CACHE}" "Triton" -} -_start_cache_sync_sidecar() { - local pidfile="/tmp/nemo_rl_compile_cache_sync.pid" - if [ -f "\$pidfile" ] && kill -0 "\$(cat "\$pidfile" 2>/dev/null)" 2>/dev/null; then - echo "[CACHE SYNC] sidecar already running (pid=\$(cat "\$pidfile" 2>/dev/null))" - return - fi - ( - set +e - trap '_sync_compile_caches_to_lustre; exit 0' TERM INT - echo "[CACHE SYNC] sidecar started, frequency=${CACHE_SYNC_FREQUENCY}s" - while true; do - sleep "${CACHE_SYNC_FREQUENCY}" - _sync_compile_caches_to_lustre - done - ) > /tmp/nemo_rl_compile_cache_sync.log 2>&1 & - echo "\$!" > "\$pidfile" - echo "[CACHE SYNC] sidecar pid=\$!" -} -if [ "${CACHE_SYNC_FREQUENCY}" -gt 0 ] 2>/dev/null; then - _start_cache_sync_sidecar -else - echo "[CACHE SYNC] disabled (CACHE_SYNC_FREQUENCY=${CACHE_SYNC_FREQUENCY})" -fi - -UV_HTTP_TIMEOUT=3600 \ - uv sync --frozen --extra mcore -SETUPEOF -export SETUP_COMMAND - -# Optional extra grpo overrides (only emitted when set, so empty == use yaml). -EXTRA_GRPO="" -[ -n "${MAX_NUM_STEPS}" ] && EXTRA_GRPO="grpo.max_num_steps=${MAX_NUM_STEPS}" - -# ===== Entrypoint switch: SC (single-controller) vs classic async GRPO ===== -# SC_MODE=1 (default): examples/run_grpo_single_controller.py + data_plane/async_rl overrides. -# SC_MODE=0: examples/nemo_gym/run_grpo_nemo_gym.py — the gym-dedicated classic -# entry; async comes from the yaml's native grpo.async_grpo block. -# (The generic examples/run_grpo.py is NOT wired for nemo-gym: it -# discards the gym actor instead of binding task_to_env["nemo_gym"], -# and its configure_generation_config eos injection trips the gym -# rollout stop-criteria assert.) -SC_MODE="${SC_MODE:-1}" -if [ "${SC_MODE}" = "1" ]; then - ENTRYPOINT="./examples/run_grpo_single_controller.py" - SC_OVERRIDES="++data_plane.enabled=true \ - ++data_plane.impl=transfer_queue \ - ++data_plane.backend=simple \ - ++data_plane.storage_capacity=1000000 \ - ++data_plane.num_storage_units=2 \ - ++data_plane.claim_meta_poll_interval_s=0.5 \ - ++data_plane.global_segment_size=549755813888 \ - ++data_plane.local_buffer_size=68719476736 \ - ++async_rl.max_weight_staleness_versions=${MAX_TRAJECTORY_AGE_STEPS} \ - ++async_rl.min_prompt_groups_per_batch=${MIN_PROMPT_GROUPS} \ - ++async_rl.max_inflight_prompts=${PPS} \ - ++async_rl.max_buffered_rollouts=$((PPS * (MAX_TRAJECTORY_AGE_STEPS + 1))) \ - ++async_rl.batch_selection_strategy=${BATCH_SELECTION_STRATEGY} \ - ++async_rl.over_sampling=${OVER_SAMPLING} \ - ++async_rl.force_in_order=${FORCE_IN_ORDER}" -else - ENTRYPOINT="./examples/nemo_gym/run_grpo_nemo_gym.py" - SC_OVERRIDES="" -fi - -# ===== Training command ===== -export COMMAND="NRL_VLLM_USE_V1=1 \ - NRL_WG_USE_RAY_REF=1 \ - WANDB_API_KEY=${WANDB_API_KEY} \ - HUGGINGFACE_TOKEN=${HUGGINGFACE_TOKEN} \ - GITHUB_TOKEN=${GITHUB_TOKEN} \ - GITLAB_TOKEN=${GITLAB_TOKEN} \ - HF_HOME=${HF_HOME} \ - HF_DATASETS_CACHE=${HF_DATASETS_CACHE} \ - UV_CACHE_DIR=${UV_CACHE_DIR} \ - VLLM_ATTENTION_BACKEND=FLASH_ATTN \ - VLLM_CACHE_ROOT=${LUSTRE_VLLM_CACHE} \ - DG_JIT_CACHE_DIR=${LUSTRE_VLLM_CACHE}/deep_gemm \ - VLLM_DEEP_GEMM_WARMUP=skip \ - NRL_FORCE_REBUILD_VENVS=${NRL_FORCE_REBUILD_VENVS:-false} \ - NG_HTTP_BYTES_DIR=${NG_HTTP_BYTES_DIR:-} \ - NRL_HTTP_BYTES_DIR=${NRL_HTTP_BYTES_DIR:-} \ - NRL_SKIP_TQ_RUNTIME_ENV_PATCH=${NRL_SKIP_TQ_RUNTIME_ENV_PATCH:-1} \ - NRL_IGNORE_VERSION_MISMATCH=1 \ - RAY_ENABLE_UV_RUN_RUNTIME_ENV=0 \ - UV_HTTP_TIMEOUT=3600 \ - UV_LOCK_TIMEOUT=900 \ - TORCH_CUDA_ARCH_LIST='9.0 10.0' \ - NEMO_GYM_SKIP_VENV_IF_PRESENT=1 \ - NEMO_GYM_VENV_DIR=${GYM_VENV_DIR} \ - uv run --frozen --extra mcore ${ENTRYPOINT} \ - --config=${CONFIG_FILE} \ - cluster.num_nodes=${NUM_NODES} \ - cluster.gpus_per_node=${GPUS_PER_NODE} \ - ++data.train.data_path=${TRAIN_DATA_PATH} \ - ++data.validation.data_path=${VAL_DATA_PATH} \ - grpo.num_prompts_per_step=${PPS} \ - grpo.num_generations_per_prompt=${GPP} \ - grpo.val_at_start=False \ - grpo.normalize_rewards=${NORMALIZE_REWARDS} \ - grpo.overlong_filtering=${OVERLONG_FILTERING} \ - grpo.val_period=${VAL_PERIOD} \ - grpo.seq_logprob_error_threshold=${SEQ_LOGPROB_ERROR_THRESHOLD} \ - ${EXTRA_GRPO} \ - ${SC_OVERRIDES} \ - ++policy.draft.enabled=false \ - ++policy.draft.model_name=null \ - ++policy.draft.loss_weight=0.1 \ - ++policy.draft.num_layers=null \ - ++policy.draft.aux_layer_indices=null \ - env.should_log_nemo_gym_responses=${LOG_GYM_RESPONSES} \ - policy.generation.colocated.enabled=${COLOCATED_ENABLED} \ - policy.model_name=${MODEL_PATH} \ - policy.max_total_sequence_length=${SEQLEN} \ - policy.dynamic_batching.enabled=False \ - policy.train_global_batch_size=${GBS} \ - policy.make_sequence_length_divisible_by=${MAKE_SEQ_DIVISIBLE_BY} \ - policy.offload_optimizer_for_logprob=true \ - policy.sequence_packing.enabled=${SEQUENCE_PACKING} \ - policy.megatron_cfg.tensor_model_parallel_size=${TP} \ - policy.megatron_cfg.expert_model_parallel_size=${EP} \ - policy.megatron_cfg.context_parallel_size=${CP} \ - policy.megatron_cfg.pipeline_model_parallel_size=${PP} \ - policy.megatron_cfg.sequence_parallel=True \ - policy.megatron_cfg.bias_activation_fusion=False \ - ++policy.megatron_cfg.use_fused_weighted_squared_relu=false \ - policy.megatron_cfg.distributed_data_parallel_config.overlap_grad_reduce=${OVERLAP_GRAD_REDUCE} \ - policy.megatron_cfg.moe_permute_fusion=${MOE_PERMUTE_FUSION} \ - policy.megatron_cfg.moe_enable_deepep=${MOE_ENABLE_DEEPEP} \ - policy.megatron_cfg.moe_token_dispatcher_type=${MOE_TOKEN_DISPATCHER_TYPE} \ - policy.megatron_cfg.moe_aux_loss_coeff=${MOE_AUX_LOSS_COEFF} \ - policy.megatron_cfg.moe_router_load_balancing_type=${MOE_ROUTER_LOAD_BALANCING_TYPE} \ - policy.megatron_cfg.moe_router_bias_update_rate=${MOE_ROUTER_BIAS_UPDATE_RATE} \ - policy.megatron_cfg.freeze_moe_router=${MOE_FREEZE_ROUTER} \ - policy.megatron_cfg.optimizer.lr=${LR} \ - policy.megatron_cfg.optimizer.min_lr=${LR} \ - policy.megatron_cfg.optimizer.weight_decay=0 \ - policy.megatron_cfg.empty_unused_memory_level=2 \ - policy.megatron_cfg.activation_checkpointing=True \ - policy.generation.temperature=${TEMPERATURE} \ - policy.generation.vllm_cfg.tensor_parallel_size=${VLLM_TP} \ - policy.generation.vllm_cfg.gpu_memory_utilization=${VLLM_GPU_UTIL} \ - policy.generation.vllm_cfg.skip_tokenizer_init=False \ - loss_fn.reference_policy_kl_penalty=${KL} \ - loss_fn.ratio_clip_min=${CLIP_MIN} \ - loss_fn.ratio_clip_max=${CLIP_MAX} \ - loss_fn.use_on_policy_kl_approximation=${USE_ON_POLICY_KL_APPROXIMATION} \ - loss_fn.use_importance_sampling_correction=${IMPORTANCE_SAMPLING_CORRECTION} \ - loss_fn.sequence_level_importance_ratios=${SEQ_LEVEL_IS} \ - loss_fn.token_level_loss=${TOKEN_LEVEL_LOSS} \ - loss_fn.force_on_policy_ratio=${FORCE_ON_POLICY_RATIO} \ - checkpointing.checkpoint_dir=${CHECKPOINT_DIR} \ - checkpointing.save_period=${SAVE_PERIOD} \ - checkpointing.keep_top_k=${KEEP_TOP_K} \ - ++checkpointing.metric_name=train:total_reward/mean \ - ++checkpointing.checkpoint_must_save_by=${MUST_SAVE_BY} \ - logger.wandb_enabled=True \ - logger.wandb.name=${WANDB_NAME} \ - logger.wandb.project=${WANDB_PROJ}" - -# Async non-colocated: generation cluster + clipping + agent turn/timeout knobs. -export COMMAND="${COMMAND} \ - policy.generation.colocated.resources.num_nodes=${NUM_GEN_NODES} \ - policy.generation.colocated.resources.gpus_per_node=${GPUS_PER_NODE} \ - grpo.advantage_clip_low=${ADVANTAGE_CLIP_LOW} \ - grpo.advantage_clip_high=${ADVANTAGE_CLIP_HIGH} \ - loss_fn.truncated_importance_sampling_ratio=${TIS_THRESHOLD} \ - env.nemo_gym.swe_agents_train.responses_api_agents.swe_agents.agent_max_turns=${AGENT_MAX_TURNS} \ - env.nemo_gym.swe_agents_train.responses_api_agents.swe_agents.swebench_agent_timeout=${AGENT_TIMEOUT} \ - env.nemo_gym.swe_agents_val.responses_api_agents.swe_agents.agent_max_turns=${AGENT_MAX_TURNS} \ - env.nemo_gym.swe_agents_val.responses_api_agents.swe_agents.swebench_agent_timeout=${AGENT_TIMEOUT}" - -# --------------------------------------------------------------------------- -# Mounts: the run tree at its own path (so ./examples/... resolves) and the -# gym source over the container's bundled copy. Export MOUNTS to replace, or -# EXTRA_MOUNTS to append (e.g. the filesystem holding MODEL_PATH/data). -# --------------------------------------------------------------------------- -GYM_CODE="${RUN_DIR}/3rdparty/Gym-workspace/Gym" -MOUNTS="${MOUNTS:-${RUN_DIR}:${RUN_DIR},${GYM_CODE}:/opt/nemo-rl/3rdparty/Gym-workspace/Gym}" -[ -n "${EXTRA_MOUNTS:-}" ] && MOUNTS="${MOUNTS},${EXTRA_MOUNTS}" - -mkdir -p "${CHECKPOINT_DIR}" "${BASE_LOG_DIR}" - -# ray.sub reads these from the environment. -export CONTAINER MOUNTS COMMAND SETUP_COMMAND GPUS_PER_NODE CPUS_PER_WORKER BASE_LOG_DIR -[ -n "${UV_CACHE_DIR_OVERRIDE:-}" ] && export UV_CACHE_DIR_OVERRIDE - -sbatch_args=( - --nodes="${NUM_NODES}" - --account="${ACCOUNT}" - --job-name="${EXP_NAME}" - --partition="${PARTITION}" - --time="${TIME}" - --gres=gpu:"${GPUS_PER_NODE}" - --exclusive - --dependency=singleton - --output="${BASE_LOG_DIR}/slurm-%j.out" -) -[ -n "${REAPER_COMMENT}" ] && sbatch_args+=(--comment="${REAPER_COMMENT}") -# shellcheck disable=SC2206 -[ -n "${SBATCH_EXTRA_ARGS:-}" ] && sbatch_args+=(${SBATCH_EXTRA_ARGS}) - -echo "==========================================" -echo "SWE async GRPO | Experiment: ${EXP_SUFFIX}" -echo "Entrypoint: ${ENTRYPOINT} (SC_MODE=${SC_MODE})" -echo "Run tree: ${RUN_DIR} @ ${RUN_COMMIT}" -echo "Account: ${ACCOUNT} / ${PARTITION}" -echo "Nodes: ${NUM_NODES} total (generation carves out ${NUM_GEN_NODES}) Time: ${TIME}" -echo "Container: ${CONTAINER}" -echo "Parallelism: TP=${TP}, EP=${EP}, CP=${CP}, PP=${PP}, vLLM_TP=${VLLM_TP}, pad=${MAKE_SEQ_DIVISIBLE_BY}" -echo "Training: PPS=${PPS}, GPP=${GPP}, GBS=${GBS}, LR=${LR}, seqlen=${SEQLEN}, max_steps=${MAX_NUM_STEPS:-}, min_prompt_groups=${MIN_PROMPT_GROUPS}" -echo "Streaming: over_sampling=${OVER_SAMPLING}, force_in_order=${FORCE_IN_ORDER}, age=${MAX_TRAJECTORY_AGE_STEPS}" -echo "Model: ${MODEL_PATH}" -echo "Checkpoint: ${CHECKPOINT_DIR}" -echo "WandB: ${WANDB_PROJ}/${WANDB_NAME}" -echo "==========================================" - -if [ "${DRY_RUN:-0}" = "1" ]; then - echo "[DRY_RUN] sbatch ${sbatch_args[*]} ${RUN_DIR}/ray.sub" - echo "[DRY_RUN] COMMAND (first 400 chars): ${COMMAND:0:400}..." - exit 0 -fi - -cd "${RUN_DIR}" -out=$(sbatch "${sbatch_args[@]}" "${RUN_DIR}/ray.sub") -echo "${out}" -job_id=$(echo "${out}" | grep -oE '[0-9]+' | head -1) -if [ -n "${job_id}" ]; then - echo "Job ID: ${job_id}" - echo "Monitor: squeue -j ${job_id}" - echo "Driver log: ${BASE_LOG_DIR}/${job_id}-logs/ray-driver.log" -fi diff --git a/swe_nano.env b/swe_nano.env index 4bbe9e027fb..274fc578f97 100644 --- a/swe_nano.env +++ b/swe_nano.env @@ -2,12 +2,11 @@ # Nemotron 3 NANO v3 (30B-A3B) — SWE Teacher — MINIMAL 5-node smoke — GB200 # ============================================================================= # Small-scale SWE: swaps Ultra 550B for Nano v3 30B-A3B and shrinks to 6 nodes -# (train 4 + gen 2), GBS 64, 49k context. Shared by the three nano SWE -# launchers; each one overrides CONFIG_PATH (and the entrypoint) for its variant: +# (train 4 + gen 2), GBS 64, 49k context. Shared by the nano SWE launchers; +# each one overrides CONFIG_PATH (and the entrypoint) for its variant: # -# swe_nano_interactive.sh async GRPO, no TransferQueue (baseline) -# swe_nano_tq_interactive.sh sync GRPO, TransferQueue honoured -# swe_nano_sc_interactive.sh async GRPO via SingleController, TQ honoured +# swe_nano_sc[_interactive].sh async GRPO via SingleController, TQ honoured (legacy arm) +# swe_nano_sc_capture[_interactive].sh same, with gate-authoritative token capture # # See docs/guides/nano-swe-transferqueue.md for which code paths actually # honour the data plane and for the verified 5-step run. diff --git a/swe_nano_interactive.sh b/swe_nano_interactive.sh deleted file mode 100644 index 220f6aea24b..00000000000 --- a/swe_nano_interactive.sh +++ /dev/null @@ -1,37 +0,0 @@ -#!/bin/bash -# ============================================================================= -# swe_nano_interactive.sh — interactive launch of the minimal Nano SWE smoke. -# -# Nano v3 30B-A3B, 6-node Qwen3-mesh recipe: async GRPO through -# run_grpo_nemo_gym.py. This is the NO-TransferQueue baseline — that path drives -# an in-memory ReplayBuffer and ignores data_plane. For a honoured data plane -# use swe_nano_sc_interactive.sh (async) or swe_nano_tq_interactive.sh (sync). -# -# Allocates the nodes, starts Ray, and idles so you can attach and run/edit/ -# re-run the driver WITHOUT requeueing. -# -# Run from a NETWORKED shell (the sandbox can't reach slurmctld): -# bash swe_nano_interactive.sh -# bash swe_nano_interactive.sh grpo.max_num_steps=1 # extra hydra overrides pass through -# -# On submit it prints: -# bash -attach.sh # shell on the head node (Ray already up) -# source -run-cmd.sh # run the recipe inside that shell; edit + re-source to iterate -# Cancel with: scancel -# -# First run downloads ~60 GB (Nemotron-3-Nano-30B-A3B-BF16) + converts to -# Megatron, so give the alloc room, e.g.: -# INTERACTIVE_WALLTIME=2:00:00 bash swe_nano_interactive.sh -# ============================================================================= -set -euo pipefail -HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" - -set -a -# shellcheck disable=SC1091 -source "${HERE}/swe_nano.env" -set +a - -# INTERACTIVE_WAIT=1 blocks until Ray is up then prints the attach command; -# set INTERACTIVE_WAIT=0 to submit and return immediately. -INTERACTIVE=1 DRY_RUN=0 INTERACTIVE_WAIT="${INTERACTIVE_WAIT:-1}" \ - bash "${HERE}/ultra_launch.sh" "$@" diff --git a/swe_nano_sc_capture.sh b/swe_nano_sc_capture.sh index 53d3a0ba069..e979206f344 100644 --- a/swe_nano_sc_capture.sh +++ b/swe_nano_sc_capture.sh @@ -43,6 +43,6 @@ fi set +a bash "${HERE}/ultra_launch.sh" \ - +token_capture.enabled=true \ + token_capture.enabled=true \ +env.nemo_gym.rollout_max_attempts_to_avoid_lp_nan=1 \ "$@" diff --git a/swe_nano_sc_capture_interactive.sh b/swe_nano_sc_capture_interactive.sh index 1c639e537cd..731f801f3a4 100755 --- a/swe_nano_sc_capture_interactive.sh +++ b/swe_nano_sc_capture_interactive.sh @@ -5,7 +5,7 @@ # # Same 6-node SingleController shape as swe_nano_sc_interactive.sh; adds the # posture the capture path needs (learned on job 5764598, 2026-08-01): -# +token_capture.enabled=true hydra APPEND — no yaml in the nano chain +# token_capture.enabled=true the recipe yaml defines the block (enabled: false) # defines the key (pydantic default only) # +env.nemo_gym.rollout_max_attempts_to_avoid_lp_nan=1 # capture hard-errors otherwise; pin on the @@ -44,6 +44,6 @@ set +a INTERACTIVE=1 DRY_RUN=0 INTERACTIVE_WAIT="${INTERACTIVE_WAIT:-1}" \ bash "${HERE}/ultra_launch.sh" \ - +token_capture.enabled=true \ + token_capture.enabled=true \ +env.nemo_gym.rollout_max_attempts_to_avoid_lp_nan=1 \ "$@" diff --git a/swe_nano_sc_interactive.sh b/swe_nano_sc_interactive.sh index 5e68ef9b0a5..5233d56ddfc 100644 --- a/swe_nano_sc_interactive.sh +++ b/swe_nano_sc_interactive.sh @@ -4,7 +4,7 @@ # HONOURED TransferQueue data plane. This is the VERIFIED recipe (see # docs/guides/nano-swe-transferqueue.md). # -# Same 6-node nano shape as swe_nano_interactive.sh (train 4 / gen 2), but: +# 6-node nano shape (train 4 / gen 2): # entrypoint examples/run_grpo_single_controller.py (NOT run_grpo_nemo_gym.py) # config examples/configs/ultra/nano_swe_teacher_sc.yaml # diff --git a/swe_nano_tq_interactive.sh b/swe_nano_tq_interactive.sh deleted file mode 100644 index 2fcec20f305..00000000000 --- a/swe_nano_tq_interactive.sh +++ /dev/null @@ -1,35 +0,0 @@ -#!/bin/bash -# ============================================================================= -# swe_nano_tq_interactive.sh — interactive nano SWE run with the TransferQueue -# data plane ENABLED (sync GRPO). -# -# Same 6-node nano shape as swe_nano_interactive.sh, but points at the sync + TQ -# config (nano_swe_teacher_sync_tq.yaml): async_grpo disabled, data_plane.enabled -# =true, impl=transfer_queue. Use this to CHECK the data plane end to end. -# -# Run from a NETWORKED shell: -# bash swe_nano_tq_interactive.sh -# bash swe_nano_tq_interactive.sh grpo.max_num_steps=1 # extra hydra overrides pass through -# -# Then: bash -attach.sh ; source -run-cmd.sh (edit + re-source to iterate) -# -# Verify TQ engaged: the driver log must show "🚀 Running synchronous GRPO -# training" (not "Running async GRPO training") plus TransferQueueController / -# SimpleStorageUnit actors. For async + TQ use swe_nano_sc_interactive.sh. -# ============================================================================= -set -euo pipefail -HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" - -set -a -# shellcheck disable=SC1091 -source "${HERE}/swe_nano.env" - -# --- override for the sync + TransferQueue variant -------------------------- -EXP_NAME=nano-swe-tq-zhiyul -CONFIG_PATH="${CODE_DIR}/examples/configs/ultra/nano_swe_teacher_sync_tq.yaml" -RESULTS_DIR="${WORKSPACE_DIR}/results/${EXP_NAME}" -BASE_LOG_DIR="${WORKSPACE_DIR}/ray_logs/${EXP_NAME}" -set +a - -INTERACTIVE=1 DRY_RUN=0 INTERACTIVE_WAIT="${INTERACTIVE_WAIT:-1}" \ - bash "${HERE}/ultra_launch.sh" "$@" From bd84cb7792bf8866798c07ac56a44709f9ff78d8 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Tue, 11 Aug 2026 17:22:21 -0700 Subject: [PATCH 11/52] feat(sc): publish routed_experts through the token-capture finalizer (R3) Completes router replay on the capture path: Gym linearize rebuilds per-token routes from staged extras (submodule pin -> 980e919), the finalizer shapes them into [n, max_len, L, K] int16 beside the canonical row (sentinel for placeholders/uncovered spans; coverage + sentinel fraction metrics), SC setup registers the optional routed_experts train field and routed_experts_delta staging field, and the sink stages int16. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre (cherry picked from commit 09bf7255626f1b532d0a059819140263df631ad0) --- .../single_controller_utils/setup.py | 17 ++- nemo_rl/data_plane/tq_token_sink.py | 58 ++++--- nemo_rl/experience/blackbox_finalizer.py | 87 +++++++++++ .../data_plane/test_blackbox_finalizer.py | 144 ++++++++++++++++++ 4 files changed, 285 insertions(+), 21 deletions(-) diff --git a/nemo_rl/algorithms/single_controller_utils/setup.py b/nemo_rl/algorithms/single_controller_utils/setup.py index 10e383c199b..f0ab927dafe 100644 --- a/nemo_rl/algorithms/single_controller_utils/setup.py +++ b/nemo_rl/algorithms/single_controller_utils/setup.py @@ -636,21 +636,31 @@ def _build_generation_then_trainer( # (see TQDataPlaneClient.register_partition). token_capture_cfg = master_config.token_capture if token_capture_cfg.enabled: - from nemo_rl.data_plane.schema import DP_TRAIN_FIELDS + from nemo_rl.data_plane.schema import ( + DP_TRAIN_FIELDS, + fields_with_optional_routed_experts, + ) + from nemo_rl.data_plane.tq_token_sink import ( + ROUTED_EXPERTS_FIELD as STAGING_ROUTED_EXPERTS_FIELD, + ) from nemo_rl.data_plane.tq_token_sink import STAGING_FIELDS + r3_enabled = router_replay_enabled(master_config.policy) group_size = grpo_config.num_generations_per_prompt num_rollout_samples = master_config.async_rl.max_buffered_rollouts * group_size dp_client.register_partition( partition_id=partition_id, - fields=list(DP_TRAIN_FIELDS), + fields=fields_with_optional_routed_experts( + DP_TRAIN_FIELDS, enabled=r3_enabled + ), num_samples=num_rollout_samples, consumer_tasks=["prev_lp", "ref_lp", "train"], grpo_group_size=group_size, ) dp_client.register_partition( partition_id=token_capture_cfg.staging_partition, - fields=list(STAGING_FIELDS), + fields=list(STAGING_FIELDS) + + ([STAGING_ROUTED_EXPERTS_FIELD] if r3_enabled else []), num_samples=num_rollout_samples, consumer_tasks=["finalize"], ) @@ -718,6 +728,7 @@ def _build_generation_then_trainer( pad_token_id=pad_id, mixed_weight_version_policy=token_capture_cfg.mixed_weight_version_policy, min_valid_fraction_per_group=token_capture_cfg.min_valid_fraction_per_group, + router_replay_enabled=router_replay_enabled(policy_config), ) rollout_manager = RolloutManager( tokenizer=tokenizer, diff --git a/nemo_rl/data_plane/tq_token_sink.py b/nemo_rl/data_plane/tq_token_sink.py index 0f83c1e9fc9..c273b1dbcd7 100644 --- a/nemo_rl/data_plane/tq_token_sink.py +++ b/nemo_rl/data_plane/tq_token_sink.py @@ -79,24 +79,46 @@ def __init__(self, dp_client: Any, *, staging_partition: str) -> None: def stage(self, record: StagedCallRecord) -> StageResult: key = record.staging_key try: - fields = TensorDict( - { - "token_ids_delta": torch.tensor( - [record.token_ids_delta], dtype=torch.int64 - ), - "token_mask_delta": torch.tensor( - [record.token_mask_delta], dtype=torch.float32 - ), - "generation_logprobs_delta": torch.tensor( - [record.generation_logprobs_delta], dtype=torch.float32 - ), - "prev_len": torch.tensor([record.prev_len], dtype=torch.int64), - "weight_version": torch.tensor( - [record.weight_version], dtype=torch.int64 - ), - }, - batch_size=[1], - ) + field_dict = { + "token_ids_delta": torch.tensor( + [record.token_ids_delta], dtype=torch.int64 + ), + "token_mask_delta": torch.tensor( + [record.token_mask_delta], dtype=torch.float32 + ), + "generation_logprobs_delta": torch.tensor( + [record.generation_logprobs_delta], dtype=torch.float32 + ), + "prev_len": torch.tensor([record.prev_len], dtype=torch.int64), + "weight_version": torch.tensor( + [record.weight_version], dtype=torch.int64 + ), + } + routed = (record.extras or {}).get("routed_experts") + if routed is not None: + # Worker sends full prompt+generation coverage of this call's + # spliced sequence; the delta-aligned slice is [prev_len:]. + # A length mismatch stages the delta WITHOUT extras (replay + # degrades to Megatron's own router for this call) rather + # than poisoning the rollout. + delta_len = len(record.token_ids_delta) + # int16 covers every practical expert count (<32k) and the -1 + # sentinel; halves staged bytes vs int32. + experts = torch.tensor(routed, dtype=torch.int16) + if experts.shape[0] == record.prev_len + delta_len: + experts = experts[record.prev_len :] + if experts.shape[0] == delta_len: + field_dict[ROUTED_EXPERTS_FIELD] = experts.unsqueeze(0) + else: + logging.getLogger(__name__).warning( + "routed_experts length %d does not cover delta %d " + "(prev_len %d) for %s — staging without extras", + experts.shape[0], + delta_len, + record.prev_len, + key, + ) + fields = TensorDict(field_dict, batch_size=[1]) tags = [ { "rollout_id": record.rollout_id, diff --git a/nemo_rl/experience/blackbox_finalizer.py b/nemo_rl/experience/blackbox_finalizer.py index b59d96c73a2..fe3c28c47d1 100644 --- a/nemo_rl/experience/blackbox_finalizer.py +++ b/nemo_rl/experience/blackbox_finalizer.py @@ -40,6 +40,9 @@ from nemo_rl.data_plane.tq_token_sink import TQTokenSink, TQTokenSource from nemo_rl.experience.payload import pack_payload from nemo_rl.experience.row_dump import maybe_dump_train_rows +from nemo_rl.models.generation.interfaces import ( + ROUTED_EXPERTS_MISSING_ROUTE_SENTINEL as _ROUTED_EXPERTS_SENTINEL, +) @dataclass(frozen=True) @@ -57,6 +60,9 @@ class FinalizedRollout: staging_keys: list[str] min_wv: Optional[int] = None max_wv: Optional[int] = None + # Router replay (R3): [len(token_ids)][num_moe_layers][topk] from the + # rebuilt chain; None when the rollout staged no extras. + routed_experts: Optional[list] = None @dataclass @@ -85,12 +91,18 @@ def __init__( pad_token_id: int, mixed_weight_version_policy: str, min_valid_fraction_per_group: Optional[float], + router_replay_enabled: bool = False, ) -> None: self._dp_client = dp_client self._partition_id = partition_id self._pad_token_id = int(pad_token_id) self._mixed_weight_version_policy = mixed_weight_version_policy self._min_valid_fraction = min_valid_fraction_per_group + self._router_replay_enabled = router_replay_enabled + # (num_moe_layers, topk), learned from the first rebuilt row that + # carries routes; placeholder-only groups need it to shape their + # sentinel tensors consistently with the model. + self._routed_dims: Optional[tuple[int, int]] = None self._source = TQTokenSource(dp_client, staging_partition=staging_partition) # The sink's clear() is the staging-partition delete; no staging # writes happen here. @@ -210,6 +222,7 @@ def rejected(reason: str, staging_keys: list[str]) -> FinalizedRollout: staging_keys=staging_keys, min_wv=min_wv, max_wv=max_wv, + routed_experts=row.routed_experts, ) # ── per group ─────────────────────────────────────────────────────────── @@ -311,6 +324,10 @@ def finalize_group( "prompt_ids_for_adv": prompt_ids_for_adv, "total_reward": rewards_t, } + if self._router_replay_enabled: + train_batch["routed_experts"] = self._build_routed_experts_tensor( + rows, max_len=max_len, metrics=metrics + ) sample_ids, fields, tags = pack_payload( train_batch, weight_version=group_min_wv, group_id=group_id ) @@ -353,6 +370,76 @@ def finalize_group( # ── internals ─────────────────────────────────────────────────────────── + def _build_routed_experts_tensor( + self, + rows: list[FinalizedRollout], + *, + max_len: int, + metrics: dict[str, float], + ) -> torch.Tensor: + """[n, max_len, L, K] int16 routes for the group; sentinel elsewhere. + + Padding, placeholder rows, and valid rows whose rebuild carried no + routes are all-sentinel: Megatron's replay falls back to its own + router for exactly those positions. (L, K) is learned from the first + rebuilt row that carries routes and cached for placeholder-only + groups; a group arriving before any routed row has been seen cannot + be shaped and fails loudly (unreachable once the first real rollout + of the run finalizes). + """ + for row in rows: + if row.valid and row.routed_experts: + first = row.routed_experts[0] + self._routed_dims = (len(first), len(first[0])) + break + if self._routed_dims is None: + raise RuntimeError( + "policy.router_replay.enabled=true (token-capture mode) but no " + "finalized rollout has carried routed_experts yet, so the " + "placeholder group tensor cannot be shaped. Check vLLM " + "enable_return_routed_experts and the staging-extras path." + ) + num_moe_layers, topk = self._routed_dims + routed = torch.full( + (len(rows), max_len, num_moe_layers, topk), + _ROUTED_EXPERTS_SENTINEL, + dtype=torch.int16, + ) + rows_with_routes = 0 + valid_rows = 0 + sentinel_tokens = 0 + covered_tokens = 0 + for i, row in enumerate(rows): + if not row.valid: + continue + valid_rows += 1 + covered_tokens += len(row.token_ids) + if not row.routed_experts: + sentinel_tokens += len(row.token_ids) + continue + rows_with_routes += 1 + row_routes = torch.tensor(row.routed_experts, dtype=torch.int16) + if row_routes.shape != (len(row.token_ids), num_moe_layers, topk): + raise RuntimeError( + "rebuilt routed_experts shape " + f"{tuple(row_routes.shape)} does not match " + f"({len(row.token_ids)}, {num_moe_layers}, {topk}) for " + f"rollout {row.rollout_id}" + ) + routed[i, : row_routes.shape[0]] = row_routes + sentinel_tokens += int( + row_routes.eq(_ROUTED_EXPERTS_SENTINEL).all(-1).all(-1).sum().item() + ) + if valid_rows: + metrics["finalize/routed_experts_row_coverage"] = ( + rows_with_routes / valid_rows + ) + if covered_tokens: + metrics["finalize/routed_experts_sentinel_token_fraction"] = ( + sentinel_tokens / covered_tokens + ) + return routed + def _clear_staging(self, staging_keys: list[str]) -> None: if not staging_keys: return diff --git a/tests/unit/data_plane/test_blackbox_finalizer.py b/tests/unit/data_plane/test_blackbox_finalizer.py index 58ecdae854c..5903573ae50 100644 --- a/tests/unit/data_plane/test_blackbox_finalizer.py +++ b/tests/unit/data_plane/test_blackbox_finalizer.py @@ -245,3 +245,147 @@ def test_finalize_group_min_valid_fraction_drops(tq_client, partitions): with pytest.raises((KeyError, RuntimeError, ValueError)): rows = _fetch_rows(tq_client, rollout_ids) assert not rows # nothing published + + +# --------------------------------------------------------------------------- +# Router replay (R3): routed_experts rebuilt from staged extras and published +# --------------------------------------------------------------------------- + +_R3_PARTITION = "rollout_data_fin_r3_test" +_R3_STAGING = "rollout_staging_fin_r3_test" + + +@pytest.fixture() +def r3_partitions(tq_client): + from nemo_rl.data_plane.tq_token_sink import ROUTED_EXPERTS_FIELD + + tq_client.register_partition( + partition_id=_R3_STAGING, + fields=list(STAGING_FIELDS) + [ROUTED_EXPERTS_FIELD], + num_samples=64, + consumer_tasks=["finalize"], + ) + tq_client.register_partition( + partition_id=_R3_PARTITION, + fields=[ + "input_ids", + "input_lengths", + "generation_logprobs", + "token_mask", + "sample_mask", + "prompt_ids_for_adv", + "total_reward", + "routed_experts", + ], + num_samples=64, + consumer_tasks=["train"], + ) + yield + tq_client.clear_samples(sample_ids=None, partition_id=_R3_STAGING) + tq_client.clear_samples(sample_ids=None, partition_id=_R3_PARTITION) + + +def _routes_for_delta(call_idx: int, n_tokens: int) -> list: + """[n][L=2][K=2] rows, value = call*1000 + pos (recognizable per token).""" + return [ + [[call_idx * 1000 + pos, call_idx * 1000 + pos + 500] for _ in range(2)] + for pos in range(n_tokens) + ] + + +def _stage_fixture_with_routes(tq_client, name: str, *, rollout_id: str): + """Stage the golden fixture with per-call routed_experts extras attached. + + Returns (receipt_dict, expected LinearizedRow, routes_by_call). + """ + fixture = dict(load_fixture(name)) + fixture["rollout_id"] = rollout_id + records, _, receipt, row = build_fixture_artifacts(fixture) + sink = TQTokenSink(tq_client, staging_partition=_R3_STAGING) + routes_by_call = {} + for idx, record in enumerate(records): + routes = _routes_for_delta(idx, len(record.token_ids_delta)) + routes_by_call[record.call_id] = routes + staged = record.model_copy(update={"extras": {"routed_experts": routes}}) + assert sink.stage(staged).ok + return receipt.model_dump(), row, routes_by_call + + +def test_finalize_group_publishes_routed_experts(tq_client, r3_partitions): + group_id = "grpr3" + rollout_ids = [f"{group_id}_g0", f"{group_id}_g1"] + receipt, expected, routes_by_call = _stage_fixture_with_routes( + tq_client, "worked_example", rollout_id=rollout_ids[0] + ) + + finalizer = BlackboxFinalizer( + tq_client, + partition_id=_R3_PARTITION, + staging_partition=_R3_STAGING, + pad_token_id=PAD, + mixed_weight_version_policy="allow", + min_valid_fraction_per_group=None, + router_replay_enabled=True, + ) + finalized = finalizer.finalize_group( + group_id, + rollout_ids, + [receipt, None], # second rollout -> placeholder + [1.0, 0.0], + fallback_weight_version=9, + ) + assert not finalized.dropped + assert "routed_experts" in finalized.meta.fields + assert finalized.metrics["finalize/routed_experts_row_coverage"] == 1.0 + assert finalized.metrics["finalize/routed_experts_sentinel_token_fraction"] == 0.0 + + rows = tq_client.get_samples( + sample_ids=rollout_ids, + partition_id=_R3_PARTITION, + select_fields=["routed_experts", "input_lengths"], + ) + # Valid row: the delivered chain's staged extras, concatenated in chain + # order (the golden fixture is a single linear chain). + expected_routes = [ + row_routes + for call_id in expected.call_ids + for row_routes in routes_by_call[call_id] + ] + valid_len = len(expected.token_ids) + assert len(expected_routes) == valid_len + published = torch.as_tensor(rows["routed_experts"][0]).reshape(-1, 2, 2) + assert published[:valid_len].tolist() == expected_routes + # Placeholder row: all-sentinel (Megatron self-routes; sample_mask 0). + placeholder = torch.as_tensor(rows["routed_experts"][1]) + assert bool(placeholder.eq(-1).all().item()) + + +def test_finalize_group_router_replay_without_routes_fails_loudly( + tq_client, r3_partitions +): + group_id = "grpr3b" + rollout_id = f"{group_id}_g0" + fixture = dict(load_fixture("worked_example")) + fixture["rollout_id"] = rollout_id + records, _, receipt, _ = build_fixture_artifacts(fixture) + sink = TQTokenSink(tq_client, staging_partition=_R3_STAGING) + for record in records: + assert sink.stage(record).ok # no extras staged + + finalizer = BlackboxFinalizer( + tq_client, + partition_id=_R3_PARTITION, + staging_partition=_R3_STAGING, + pad_token_id=PAD, + mixed_weight_version_policy="allow", + min_valid_fraction_per_group=None, + router_replay_enabled=True, + ) + with pytest.raises(RuntimeError, match="routed_experts"): + finalizer.finalize_group( + group_id, + [rollout_id], + [receipt.model_dump()], + [1.0], + fallback_weight_version=9, + ) From afc30cf432762196922f0aa5a3d215a9db828cce Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 12 Aug 2026 04:39:43 -0700 Subject: [PATCH 12/52] fix(sc): drop all-placeholder groups when routed dims are unknown An all-poisoned first group must not kill the run via the (L, K) fail-loud; dropping loses no signal and keeps partition schema consistent. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre (cherry picked from commit aaf2ff82cfad14d11e7250e0f38007e08a86a008) --- nemo_rl/experience/blackbox_finalizer.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/nemo_rl/experience/blackbox_finalizer.py b/nemo_rl/experience/blackbox_finalizer.py index fe3c28c47d1..e7f378953db 100644 --- a/nemo_rl/experience/blackbox_finalizer.py +++ b/nemo_rl/experience/blackbox_finalizer.py @@ -325,6 +325,28 @@ def finalize_group( "total_reward": rewards_t, } if self._router_replay_enabled: + has_routed_row = any(r.valid and r.routed_experts for r in rows) + if not has_routed_row and self._routed_dims is None: + # Nothing to learn (L, K) from yet — e.g. an all-poisoned + # group before the first healthy rollout. Dropping loses no + # training signal (no valid rows or routes) and keeps the + # partition schema consistent for groups that do publish. + print( + f" finalize: group {group_id} dropped — router replay on " + "but no rollout carried routed_experts and (L, K) is " + "unknown yet", + flush=True, + ) + self._clear_staging(staging_keys) + metrics["finalize/group_dropped"] = 1.0 + return FinalizedGroup( + meta=None, + group_min_wv=group_min_wv, + group_max_wv=group_max_wv, + staging_keys=[], + metrics=metrics, + dropped=True, + ) train_batch["routed_experts"] = self._build_routed_experts_tensor( rows, max_len=max_len, metrics=metrics ) From 76e9af222a07bca7735a968114312e584b7f17fe Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 12 Aug 2026 06:48:29 -0700 Subject: [PATCH 13/52] fix(data-plane): decode nrlre1 envelopes in TQTokenSink.stage Post-#3292 the worker attaches routed_experts as the base64 string envelope; torch.tensor() on it raised per call, degrading every capture to capture_failed coords and poisoning all rollouts (the capture-arm smoke's failure mode). Decode the envelope (wire dtype preserved). Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre (cherry picked from commit 0482a980728e8c67576b1074557052acced6db95) --- nemo_rl/data_plane/tq_token_sink.py | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/nemo_rl/data_plane/tq_token_sink.py b/nemo_rl/data_plane/tq_token_sink.py index c273b1dbcd7..769601fda69 100644 --- a/nemo_rl/data_plane/tq_token_sink.py +++ b/nemo_rl/data_plane/tq_token_sink.py @@ -102,9 +102,18 @@ def stage(self, record: StagedCallRecord) -> StageResult: # degrades to Megatron's own router for this call) rather # than poisoning the rollout. delta_len = len(record.token_ids_delta) - # int16 covers every practical expert count (<32k) and the -1 - # sentinel; halves staged bytes vs int32. - experts = torch.tensor(routed, dtype=torch.int16) + if isinstance(routed, str): + # The worker ships routes as the nrlre1 base64 envelope + # (#3292); decode preserves the wire dtype (int8/int16). + from nemo_rl.utils.routed_experts_codec import ( + decode_routed_experts, + ) + + experts = decode_routed_experts(routed) + else: + # int16 covers every practical expert count (<32k) and + # the -1 sentinel; halves staged bytes vs int32. + experts = torch.tensor(routed, dtype=torch.int16) if experts.shape[0] == record.prev_len + delta_len: experts = experts[record.prev_len :] if experts.shape[0] == delta_len: From bc4f14194e01a4dc2f05d35e12388c734d948d0c Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 12 Aug 2026 08:14:37 -0700 Subject: [PATCH 14/52] fix(data-plane): pass dtype to decode_routed_experts in the sink The codec's decode takes (payload, dtype); the missing positional arg raised TypeError on every stage, poisoning all capture rollouts (now visible thanks to the stage-failure log). Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre (cherry picked from commit 76061561a89fd29ebb25ffcc9287dc180038c714) --- nemo_rl/data_plane/tq_token_sink.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/nemo_rl/data_plane/tq_token_sink.py b/nemo_rl/data_plane/tq_token_sink.py index 769601fda69..d715f2a18dc 100644 --- a/nemo_rl/data_plane/tq_token_sink.py +++ b/nemo_rl/data_plane/tq_token_sink.py @@ -104,12 +104,13 @@ def stage(self, record: StagedCallRecord) -> StageResult: delta_len = len(record.token_ids_delta) if isinstance(routed, str): # The worker ships routes as the nrlre1 base64 envelope - # (#3292); decode preserves the wire dtype (int8/int16). + # (#3292). int16 covers every practical expert count + # (<32k) and the -1 sentinel. from nemo_rl.utils.routed_experts_codec import ( decode_routed_experts, ) - experts = decode_routed_experts(routed) + experts = decode_routed_experts(routed, torch.int16) else: # int16 covers every practical expert count (<32k) and # the -1 sentinel; halves staged bytes vs int32. From 6b1cbc4f0da205c6a898fd590391180bb9e7e067 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 12 Aug 2026 07:58:04 -0700 Subject: [PATCH 15/52] fix(data-plane): log the stage failure reason (dropped downstream otherwise) Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre (cherry picked from commit 931a248019a3e6aba75241f6bac190868b242ecd) --- nemo_rl/data_plane/tq_token_sink.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/nemo_rl/data_plane/tq_token_sink.py b/nemo_rl/data_plane/tq_token_sink.py index d715f2a18dc..9464a7d0d90 100644 --- a/nemo_rl/data_plane/tq_token_sink.py +++ b/nemo_rl/data_plane/tq_token_sink.py @@ -150,6 +150,15 @@ def stage(self, record: StagedCallRecord) -> StageResult: tags=tags, ) except Exception as error: # noqa: BLE001 — any failure must poison, not crash serving + # The reason string is dropped downstream (_failed_coords carries + # only the disposition) — this log line is the only place the + # actual stage failure is visible. + logging.getLogger(__name__).warning( + "TQTokenSink.stage failed for %s: %s: %s", + key, + type(error).__name__, + error, + ) return StageResult( ok=False, staging_key=key, error=f"{type(error).__name__}: {error}" ) From 67fb817841e5414446f217c22cd51312f83f89ef Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 12 Aug 2026 03:23:38 -0700 Subject: [PATCH 16/52] fix(data-plane): broadcast int16 fields as int32 (NCCL has no Short type) routed_experts rides TQ as int16; the TP/CP fan-out broadcast in the worker mixin hit 'Input tensor data type is not supported for NCCL process group: Short'. Ship int16 tensors as int32 on the wire and narrow back after. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre (cherry picked from commit 180f1f03f1dba77a79b5929de92444c132a90ae9) --- nemo_rl/data_plane/worker_mixin.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/nemo_rl/data_plane/worker_mixin.py b/nemo_rl/data_plane/worker_mixin.py index 1125245c98a..b75ff9d8805 100644 --- a/nemo_rl/data_plane/worker_mixin.py +++ b/nemo_rl/data_plane/worker_mixin.py @@ -105,7 +105,15 @@ def _broadcast_batched_data_dict( dtype = getattr(torch, dtype_str.split(".")[-1]) tensor = torch.empty(shape, dtype=dtype, device=bcast_device) out[key] = tensor - torch.distributed.broadcast(tensor, src=src, group=group) + # NCCL has no int16 ("Short") type; ship as int32 and narrow back + # (routed_experts rides TQ as int16). + if tensor.dtype == torch.int16: + wire = tensor.to(torch.int32) + torch.distributed.broadcast(wire, src=src, group=group) + tensor = wire.to(torch.int16) + out[key] = tensor + else: + torch.distributed.broadcast(tensor, src=src, group=group) # Restore non-leader tensors to the leader's source device # so downstream code sees the same layout pre-broadcast. if ( From 9b9f28bf5e6b21fe02ae8ef3849660cf23f2e93b Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 12 Aug 2026 02:53:02 -0700 Subject: [PATCH 17/52] fix(nemo-gym): sentinel fallback instead of raise for routeless trainable items MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Context-overflow rollouts end on the gate's synthetic empty response and only the final completion record persists, so earlier turns' routes are unrecoverable on the echo path. Leave those messages routeless — the flatten-stage backfill sentinel-fills them (Megatron self-routes exactly those tokens) and total absence still fails loudly at the rollout actor's routed_experts guard. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre (cherry picked from commit 4bd706dd8136ba72d0e7ef40bbb148d98fd8fafd) --- nemo_rl/environments/nemo_gym.py | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/nemo_rl/environments/nemo_gym.py b/nemo_rl/environments/nemo_gym.py index 57638ef9af7..d4f6578f7b4 100644 --- a/nemo_rl/environments/nemo_gym.py +++ b/nemo_rl/environments/nemo_gym.py @@ -885,12 +885,22 @@ def _postprocess_nemo_gym_to_nemo_rl_result( f"{expected_tokens}." ) elif self.cfg.get("require_routed_experts", False): - raise ValueError( - "policy.router_replay.enabled=true requires NeMo Gym output " - "items to include routed_experts, but the field was missing. " - "Make sure the Gym repo includes routed_experts propagation " - "and the NeMo-RL vLLM OpenAI-compatible server is configured " - "with enable_return_routed_experts." + # Routes can be legitimately unrecoverable on the echo path + # (e.g. a context-overflow rollout whose only persisted + # completion record is the gate's synthetic empty response). + # Leave the message routeless: backfill_missing_routed_experts + # sentinel-fills it at flatten and Megatron self-routes those + # tokens; total absence across a batch still fails loudly at + # the rollout actor's ROUTED_EXPERTS_FIELD guard. + print( + "router_replay: trainable Gym output item without " + "routed_experts — falling back to the missing-route " + "sentinel for this message " + f"[item_idx={len(nemo_rl_message_log) // 2}, " + f"item_type={output_item_dict.get('type')!r}, " + f"n_prompt={len(prompt_token_ids)}, " + f"n_gen={len(generation_token_ids)}]", + flush=True, ) # The next prompt prefill supplies the real route for the previous From 3e40e66e122958d781e0dabefbd254a6cef70e23 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 12 Aug 2026 14:51:00 -0700 Subject: [PATCH 18/52] fix(sc): tolerate pre-R3 Gym pins in the capture finalizer LinearizedRow.routed_experts arrives with the Gym-side R3 change (Gym PR #2278 follow-up); until the submodule pin carries it, read via getattr (capture-R3 degrades to the sentinel/group-drop path) and skip the two routed-publish tests. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre --- nemo_rl/experience/blackbox_finalizer.py | 4 +++- tests/unit/data_plane/test_blackbox_finalizer.py | 14 ++++++++++++++ 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/nemo_rl/experience/blackbox_finalizer.py b/nemo_rl/experience/blackbox_finalizer.py index e7f378953db..0052309f7f3 100644 --- a/nemo_rl/experience/blackbox_finalizer.py +++ b/nemo_rl/experience/blackbox_finalizer.py @@ -222,7 +222,9 @@ def rejected(reason: str, staging_keys: list[str]) -> FinalizedRollout: staging_keys=staging_keys, min_wv=min_wv, max_wv=max_wv, - routed_experts=row.routed_experts, + # getattr: pre-R3 Gym pins' LinearizedRow has no routed_experts; + # capture-R3 then degrades to the sentinel/group-drop path. + routed_experts=getattr(row, "routed_experts", None), ) # ── per group ─────────────────────────────────────────────────────────── diff --git a/tests/unit/data_plane/test_blackbox_finalizer.py b/tests/unit/data_plane/test_blackbox_finalizer.py index 5903573ae50..11b6c1aee84 100644 --- a/tests/unit/data_plane/test_blackbox_finalizer.py +++ b/tests/unit/data_plane/test_blackbox_finalizer.py @@ -311,6 +311,16 @@ def _stage_fixture_with_routes(tq_client, name: str, *, rollout_id: str): return receipt.model_dump(), row, routes_by_call +def _gym_linearize_supports_routes() -> bool: + from nemo_gym.token_id_capture.staging.rebuild import LinearizedRow + + return "routed_experts" in getattr(LinearizedRow, "__dataclass_fields__", {}) + + +@pytest.mark.skipif( + not _gym_linearize_supports_routes(), + reason="Gym pin predates LinearizedRow.routed_experts (Gym PR #2278 R3 follow-up)", +) def test_finalize_group_publishes_routed_experts(tq_client, r3_partitions): group_id = "grpr3" rollout_ids = [f"{group_id}_g0", f"{group_id}_g1"] @@ -360,6 +370,10 @@ def test_finalize_group_publishes_routed_experts(tq_client, r3_partitions): assert bool(placeholder.eq(-1).all().item()) +@pytest.mark.skipif( + not _gym_linearize_supports_routes(), + reason="Gym pin predates LinearizedRow.routed_experts (Gym PR #2278 R3 follow-up)", +) def test_finalize_group_router_replay_without_routes_fails_loudly( tq_client, r3_partitions ): From c1f5b1905044ebb057dc5e821b7254e8250fe1b1 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 12 Aug 2026 15:29:04 -0700 Subject: [PATCH 19/52] chore(sc): bump Gym pin to the rebased tokidcap-capture head (R3 linearize + route stitch) Activates capture-R3 end to end: LinearizedRow.routed_experts is now present, so the finalizer publishes routes and the two guarded tests un-skip. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre --- 3rdparty/Gym-workspace/Gym | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/3rdparty/Gym-workspace/Gym b/3rdparty/Gym-workspace/Gym index b6051536d9a..cbe939999d7 160000 --- a/3rdparty/Gym-workspace/Gym +++ b/3rdparty/Gym-workspace/Gym @@ -1 +1 @@ -Subproject commit b6051536d9ac569c853200051accf81ad8af0085 +Subproject commit cbe939999d701602923be7e73419f1a9076d3fe9 From 40617ad9a01aba30d9edd6df5d5f55ea8730b061 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Tue, 18 Aug 2026 10:35:58 -0700 Subject: [PATCH 20/52] feat(sc): add token-capture finalizer actor pool Signed-off-by: Pranav Thombre --- 3rdparty/Gym-workspace/Gym | 2 +- .../configs/ultra/nano_swe_teacher_sc.yaml | 31 +- .../algorithms/async_utils/replay_buffer.py | 74 ++++- nemo_rl/algorithms/single_controller.py | 294 ++++++++++++++++-- .../single_controller_utils/config.py | 39 ++- .../single_controller_utils/setup.py | 55 +++- nemo_rl/data_plane/preshard.py | 2 + nemo_rl/data_plane/schema.py | 8 + nemo_rl/data_plane/tq_token_sink.py | 175 ++++++++++- nemo_rl/data_plane/worker_mixin.py | 196 ++++++++++++ nemo_rl/experience/blackbox_finalizer.py | 174 ++++++++++- nemo_rl/experience/finalizer_actor.py | 150 +++++++++ nemo_rl/experience/rollout_manager.py | 102 +++++- nemo_rl/experience/route_plan.py | 230 ++++++++++++++ nemo_rl/models/megatron/router_replay.py | 12 + nemo_rl/models/policy/tq_policy.py | 56 +++- .../policy/workers/megatron_policy_worker.py | 9 +- pyrefly.toml | 2 + .../data_plane/test_blackbox_finalizer.py | 162 ++++++++++ tests/unit/data_plane/test_preshard_extras.py | 11 + .../unit/data_plane/test_tq_policy_routes.py | 92 ++++++ tests/unit/data_plane/test_tq_token_sink.py | 36 +++ .../data_plane/test_worker_route_assembly.py | 148 +++++++++ tests/unit/experience/test_finalizer_actor.py | 79 +++++ tests/unit/experience/test_route_plan.py | 102 ++++++ .../test_finalizer_lifecycle.py | 246 +++++++++++++++ .../single_controller/test_rollout_pump.py | 79 +++++ .../test_single_controller_setup.py | 12 + .../test_tq_replay_buffer.py | 74 +++++ .../unit/single_controller/test_train_pump.py | 1 + ultra_launch.sh | 9 +- 31 files changed, 2556 insertions(+), 106 deletions(-) create mode 100644 nemo_rl/experience/finalizer_actor.py create mode 100644 nemo_rl/experience/route_plan.py create mode 100644 tests/unit/data_plane/test_tq_policy_routes.py create mode 100644 tests/unit/data_plane/test_worker_route_assembly.py create mode 100644 tests/unit/experience/test_finalizer_actor.py create mode 100644 tests/unit/experience/test_route_plan.py create mode 100644 tests/unit/single_controller/test_finalizer_lifecycle.py diff --git a/3rdparty/Gym-workspace/Gym b/3rdparty/Gym-workspace/Gym index cbe939999d7..96a3cb0faf4 160000 --- a/3rdparty/Gym-workspace/Gym +++ b/3rdparty/Gym-workspace/Gym @@ -1 +1 @@ -Subproject commit cbe939999d701602923be7e73419f1a9076d3fe9 +Subproject commit 96a3cb0faf4bf25fb8b5da074e10839f42f5b0c8 diff --git a/examples/configs/ultra/nano_swe_teacher_sc.yaml b/examples/configs/ultra/nano_swe_teacher_sc.yaml index 630670f33f0..7436b591c3d 100644 --- a/examples/configs/ultra/nano_swe_teacher_sc.yaml +++ b/examples/configs/ultra/nano_swe_teacher_sc.yaml @@ -35,11 +35,29 @@ async_rl: grpo: val_period: 0 # SC does not support validation yet + # Main-era SC refuses legacy async config (async_rl above is the SC + # equivalent); the block is inherited from swe_teacher.yaml. + async_grpo: null checkpointing: enabled: false # SC does not support checkpointing yet policy: + generation: + # SingleController has no validation loop yet, so keep the required + # validation sampling profile identical to rollout sampling. + val_temperature: ${.temperature} + val_top_p: ${.top_p} + val_top_k: ${.top_k} + vllm_kwargs: + # vLLM 0.25's auto MoE backend picks FLASHINFER_TRTLLM on GB200, which + # re-lays w13/w2 into a 4D block format at init; the RL refit + # (model.load_weights of HF per-expert 2D weights) then dies with + # "shard_dim=0 is not a valid data dimension for a 3D tensor" + # (smoke 6107601, all 8 workers; same class as sglang#27787). Triton + # keeps the canonical [E, I, H] layout that refit can load — matches + # the other MoE recipes (e.g. grpo-nemotron3-super-120BA12B). + moe_backend: triton # run_grpo_single_controller.py:111 reads policy.draft.enabled (speculative # draft weights). The Ultra/SWE chain has no draft block; add it, disabled. draft: @@ -66,7 +84,8 @@ loss_fn: # (nemo_rl/algorithms/single_controller_utils/config.py). token_capture: enabled: false - # TQ partition holding per-call staged token deltas (finalizer-cleared). + # TQ partition holding per-call staged token deltas. Deferred-route mode + # retains these through policy consumption; direct mode clears in finalizer. staging_partition: "rollout_staging" # continue: a failed worker-side stage poisons the rollout (placeholder row); # abort: fails the whole rollout at the gate. @@ -77,7 +96,8 @@ token_capture: # Drop the whole group when fewer than this fraction of its rollouts # produced valid rows (null keeps every group). min_valid_fraction_per_group: null - # Gate-side cleanup backstops (seconds). + # Gate registration expiry. staging_ttl_s is reserved configuration only; + # TQ does not currently provide an operational staging TTL/reaper. registration_ttl_s: 3600.0 staging_ttl_s: 3600.0 # Gym LineageIndex capacity (each in-flight rollout's cumulative tokens @@ -94,3 +114,10 @@ token_capture: # Directory for the Gym base capture layer the gate rides on (stays # essentially empty on the gate path). null = /gym_token_capture. capture_dir: null + # Keep route bytes out of canonical rows and assemble them on policy workers. + defer_routed_experts_to_policy: false + # Independent rollback control for the bounded CPU finalizer actor pool and + # generation/finalization overlap. + finalizer_actor_pool_enabled: false + # Fixed validation pool; actors have no automatic restart/task retry. + num_finalizer_workers: 2 diff --git a/nemo_rl/algorithms/async_utils/replay_buffer.py b/nemo_rl/algorithms/async_utils/replay_buffer.py index 51828a9c445..524873ad21b 100644 --- a/nemo_rl/algorithms/async_utils/replay_buffer.py +++ b/nemo_rl/algorithms/async_utils/replay_buffer.py @@ -24,7 +24,7 @@ from nemo_rl.algorithms.async_utils.interfaces import ReplayBufferProtocol from nemo_rl.data_plane import KVBatchMeta -from nemo_rl.data_plane.schema import ROUTED_EXPERTS_FIELD +from nemo_rl.data_plane.schema import ROUTE_PLAN_TAG, ROUTED_EXPERTS_FIELD from nemo_rl.experience.interfaces import PromptGroupRecord from nemo_rl.experience.payload import pack_payload, record_to_train_batch from nemo_rl.experience.row_dump import maybe_dump_train_rows @@ -765,7 +765,6 @@ async def commit( fields=fields, tags=tags, ) - # mirrors kv_first_write lengths = train_batch["input_lengths"] meta = KVBatchMeta( @@ -862,6 +861,30 @@ async def commit_finalized( f"TQReplayBuffer.commit_finalized: group {group_id} has no " "live slot (evicted or never reserved)" ) from None + tagged_plans = [ + tag[ROUTE_PLAN_TAG] for tag in (meta.tags or []) if ROUTE_PLAN_TAG in tag + ] + if tagged_plans: + if len(tagged_plans) != len(meta.sample_ids): + raise ValueError( + "commit_finalized received mixed deferred/direct route plans" + ) + from nemo_rl.experience.route_plan import decode_route_plan + + plan_cleanup_keys = { + key + for encoded in tagged_plans + for key in decode_route_plan(encoded).cleanup_staging_keys + } + provided_staging_keys = list(staging_keys or []) + if len(provided_staging_keys) != len(set(provided_staging_keys)): + raise ValueError("commit_finalized staging_keys contains duplicates") + if set(provided_staging_keys) != plan_cleanup_keys: + raise ValueError( + "commit_finalized staging ownership does not match route plans: " + f"provided={sorted(provided_staging_keys)!r}, " + f"planned={sorted(plan_cleanup_keys)!r}" + ) self.meta_list[idx] = meta self.start_weight_list[idx] = group_min_wv self.end_weight_list[idx] = group_max_wv @@ -875,9 +898,10 @@ def abort(self, group_id: str) -> bool: """Drop an unready slot whose dispatch failed or was cancelled. Token-capture mode; called from the failed dispatch path. - No DataPlane rows are cleared: an unready slot published no canonical - rows, and its staged deltas (keys unknown before a receipt) are swept - by the staging TTL backstop. + No DataPlane rows are cleared here. Callers with sealed receipts must + clear their deterministic canonical IDs and full staging manifests + before dropping this ownership record. Before receipt sealing, orphan + cleanup remains an explicit controlled-validation limitation. Returns: True when a slot was dropped; False when the group_id has no @@ -935,20 +959,38 @@ async def remove(self, idxs: list[int], remove_in_dp: bool) -> int: staging_keys = self._staging_keys_list[i] if staging_keys: dropped_staging_keys.extend(staging_keys) - self._delete_slot(i) if remove_in_dp: - await self._call_dp( - "clear_samples", - sample_ids=dropped_sample_ids, - partition_id=self._partition_id, - ) + if dropped_sample_ids: + try: + await self._call_dp( + "clear_samples", + sample_ids=dropped_sample_ids, + partition_id=self._partition_id, + ) + except Exception as error: + raise RuntimeError( + "canonical cleanup failed; retained replay-buffer ownership " + f"partition={self._partition_id!r}, " + f"sample_ids={dropped_sample_ids!r}" + ) from error if dropped_staging_keys and self._staging_partition_id is not None: - await self._call_dp( - "clear_samples", - sample_ids=dropped_staging_keys, - partition_id=self._staging_partition_id, - ) + try: + await self._call_dp( + "clear_samples", + sample_ids=dropped_staging_keys, + partition_id=self._staging_partition_id, + ) + except Exception as error: + raise RuntimeError( + "staging cleanup failed; retained replay-buffer ownership " + f"partition={self._staging_partition_id!r}, " + f"staging_keys={dropped_staging_keys!r}; canonical rows " + "may already be cleared" + ) from error + + for i in drop_idxs: + self._delete_slot(i) return len(drop_idxs) diff --git a/nemo_rl/algorithms/single_controller.py b/nemo_rl/algorithms/single_controller.py index 433b3408bb1..bfaecaa2b6c 100644 --- a/nemo_rl/algorithms/single_controller.py +++ b/nemo_rl/algorithms/single_controller.py @@ -35,9 +35,10 @@ from __future__ import annotations import asyncio +import statistics import time from functools import partial -from typing import Any, Optional, Union, cast +from typing import TYPE_CHECKING, Any, Optional, Union, cast import ray import torch @@ -61,13 +62,18 @@ from nemo_rl.data.interfaces import DatumSpec from nemo_rl.data_plane import KVBatchMeta from nemo_rl.data_plane.schema import DP_CALIB_INPUT_FIELDS +from nemo_rl.data_plane.schema import ROUTE_PLAN_TAG from nemo_rl.distributed.batched_data_dict import BatchedDataDict +from nemo_rl.experience.route_plan import decode_route_plan from nemo_rl.models.generation.sglang.sglang_generation import SGLangGeneration from nemo_rl.models.generation.vllm import VllmGeneration from nemo_rl.models.policy.tq_policy import TQPolicy from nemo_rl.utils.logger import Logger from nemo_rl.utils.timer import Timer +if TYPE_CHECKING: + from nemo_rl.experience.finalizer_actor import FinalizationRequest + Generation = Union[VllmGeneration, SGLangGeneration] @@ -123,6 +129,14 @@ def __init__( # Rebind so writer and sampler share one buffer instance even # when Ray deserializes rollout_manager and tq_buffer separately. self._rollout_manager._tq_buffer = self._buffer + self._finalizer_actors = list(actor_args.finalizer_actors) + self._available_finalizers: asyncio.Queue[Any] = asyncio.Queue() + for actor in self._finalizer_actors: + self._available_finalizers.put_nowait(actor) + self._active_finalizers = 0 + self._finalizer_waiters = 0 + self._finalizer_unknown_outcomes = 0 + self._finalizer_metrics_by_group: dict[str, dict[str, float]] = {} # Built here, not on the driver: Logger backends (wandb/tb/...) hold # _thread.lock that Ray can't cloudpickle into the actor. @@ -215,6 +229,11 @@ async def run(self) -> dict[str, Any]: rollout_task.cancel() train_task.cancel() await asyncio.gather(rollout_task, train_task, return_exceptions=True) + for actor in self._finalizer_actors: + try: + ray.kill(actor, no_restart=True) + except Exception as error: + print(f"finalizer actor termination failed: {error}", flush=True) self._logger.finish() return { @@ -231,6 +250,10 @@ async def ping(self) -> dict[str, Any]: "inflight_rollouts": self._inflight_rollouts, "rollout_permitted": self._rollout_permitted.is_set(), "epoch": self._current_epoch, + "active_finalizers": self._active_finalizers, + "finalizer_waiters": self._finalizer_waiters, + "finalizer_queue_depth": self._available_finalizers.qsize(), + "finalizer_unknown_outcomes": self._finalizer_unknown_outcomes, } # ── internal helpers ─────────────────────────────────────────────────── @@ -250,6 +273,203 @@ async def _call_dp(self, method_name: str, **kwargs) -> Any: return await result return result + @staticmethod + def _request_staging_keys(request: "FinalizationRequest") -> list[str]: + """Return the full receipt-manifest staging ownership for a request.""" + keys: list[str] = [] + for receipt in request.receipts: + if receipt is None: + continue + manifest = receipt.get("manifest") + if not isinstance(manifest, list): + continue + for record in manifest: + if isinstance(record, dict) and isinstance( + record.get("staging_key"), str + ): + keys.append(record["staging_key"]) + return list(dict.fromkeys(keys)) + + async def _cleanup_known_finalization_request( + self, request: "FinalizationRequest" + ) -> None: + """Clear known request ownership after a pre-publication/known outcome.""" + errors: list[BaseException] = [] + try: + await self._call_dp( + "clear_samples", + sample_ids=list(request.rollout_ids), + partition_id=self._partition_id, + ) + except Exception as error: + errors.append( + RuntimeError( + "pre-publication canonical cleanup failed for " + f"group={request.group_id!r}, ids={request.rollout_ids!r}" + ) + ) + errors[-1].__cause__ = error + staging_keys = self._request_staging_keys(request) + if staging_keys: + try: + await self._call_dp( + "clear_samples", + sample_ids=staging_keys, + partition_id=self._master_config.token_capture.staging_partition, + ) + except Exception as error: + errors.append( + RuntimeError( + "pre-publication staging cleanup failed for " + f"group={request.group_id!r}, keys={staging_keys!r}" + ) + ) + errors[-1].__cause__ = error + if errors: + raise BaseExceptionGroup( + f"known-outcome cleanup failed for group {request.group_id}", + errors, + ) + self._buffer.abort(request.group_id) + + async def _finalize_with_actor(self, request: "FinalizationRequest") -> None: + """Submit one metadata request to the bounded fixed actor pool.""" + self._finalizer_waiters += 1 + queue_depth = max( + 0, + self._finalizer_waiters - self._available_finalizers.qsize(), + ) + queue_start = time.perf_counter() + try: + actor = await self._available_finalizers.get() + except asyncio.CancelledError: + await self._cleanup_known_finalization_request(request) + raise + finally: + self._finalizer_waiters -= 1 + queue_wait_ms = (time.perf_counter() - queue_start) * 1000.0 + self._active_finalizers += 1 + active_actor_count = self._active_finalizers + finalize_start = time.perf_counter() + try: + finalized = await actor.finalize.remote(request) + except BaseException: + self._finalizer_unknown_outcomes += 1 + print( + "FATAL: finalizer actor RPC failed after submission; canonical " + f"publication outcome is unknown for group {request.group_id}. " + "Stopping validation without actor replacement or retry.", + flush=True, + ) + raise + else: + self._available_finalizers.put_nowait(actor) + finally: + self._active_finalizers -= 1 + finalize_total_ms = (time.perf_counter() - finalize_start) * 1000.0 + + if finalized.dropped: + try: + await self._cleanup_known_finalization_request(request) + except BaseException as cleanup_error: + raise RuntimeError( + "finalizer dropped the group and known-key cleanup failed " + f"for group {request.group_id}" + ) from cleanup_error + raise RuntimeError( + f"token capture: group {request.group_id} dropped " + "(min_valid_fraction_per_group)" + ) + if finalized.meta is None: + try: + await self._cleanup_known_finalization_request(request) + except BaseException as cleanup_error: + raise RuntimeError( + "finalizer returned no metadata and known-key cleanup failed " + f"for group {request.group_id}" + ) from cleanup_error + raise RuntimeError( + f"finalizer returned no metadata for non-dropped group {request.group_id}" + ) + try: + await self._buffer.commit_finalized( + request.group_id, + finalized.meta, + finalized.group_min_wv, + finalized.group_max_wv, + staging_keys=finalized.staging_keys, + ) + except BaseException as commit_error: + try: + await self._cleanup_known_finalization_request(request) + except BaseException as cleanup_error: + raise BaseExceptionGroup( + f"finalizer commit and known-key cleanup failed for " + f"group {request.group_id}", + [commit_error, cleanup_error], + ) + raise + finalized.metrics.update( + { + "finalize/queue_wait_ms": queue_wait_ms, + "finalize/total_ms": finalize_total_ms, + "finalize/queue_depth": float(queue_depth), + "finalize/active_actor_count": float(active_actor_count), + } + ) + self._finalizer_metrics_by_group[request.group_id] = dict(finalized.metrics) + + async def _cleanup_consumed_metas(self, metas: list[KVBatchMeta]) -> None: + """Clear canonical rows and full-manifest staging keys after train success.""" + canonical_by_partition: dict[str, list[str]] = {} + staging_by_partition: dict[str, list[str]] = {} + for meta in metas: + canonical_by_partition.setdefault(meta.partition_id, []).extend( + meta.sample_ids + ) + for tag in meta.tags or []: + encoded_plan = tag.get(ROUTE_PLAN_TAG) + if encoded_plan is None: + continue + plan = decode_route_plan(encoded_plan) + staging_by_partition.setdefault(plan.staging_partition, []).extend( + plan.cleanup_staging_keys + ) + + errors: list[BaseException] = [] + for partition_id, sample_ids in canonical_by_partition.items(): + unique_ids = list(dict.fromkeys(sample_ids)) + try: + await self._call_dp( + "clear_samples", + sample_ids=unique_ids, + partition_id=partition_id, + ) + except Exception as error: + cleanup_error = RuntimeError( + "post-train canonical cleanup failed: " + f"partition={partition_id!r}, sample_ids={unique_ids!r}" + ) + cleanup_error.__cause__ = error + errors.append(cleanup_error) + for partition_id, staging_keys in staging_by_partition.items(): + unique_keys = list(dict.fromkeys(staging_keys)) + try: + await self._call_dp( + "clear_samples", + sample_ids=unique_keys, + partition_id=partition_id, + ) + except Exception as error: + cleanup_error = RuntimeError( + "post-train staging cleanup failed: " + f"partition={partition_id!r}, staging_keys={unique_keys!r}" + ) + cleanup_error.__cause__ = error + errors.append(cleanup_error) + if errors: + raise BaseExceptionGroup("post-train DataPlane cleanup failed", errors) + # ── the three pumps + the inline advantage stage ─────────────────────── async def _rollout_pump(self) -> None: @@ -279,20 +499,39 @@ async def _dispatch_one_prompt( ) -> None: task_started_event.set() self._inflight_rollouts += 1 + generation_permit_released = False + inflight_count_released = False + ownership_transferred = False try: - await self._rollout_manager.generate_and_push( - prompt, - target_step=target_step, - inflight_registry=self._inflight_by_group_id, - ) + if self._finalizer_actors: + request = await self._rollout_manager.generate_for_finalization( + prompt, + target_step=target_step, + inflight_registry=self._inflight_by_group_id, + ) + self._inflight_rollouts -= 1 + inflight_count_released = True + sem.release() + generation_permit_released = True + await self._finalize_with_actor(request) + else: + await self._rollout_manager.generate_and_push( + prompt, + target_step=target_step, + inflight_registry=self._inflight_by_group_id, + ) + ownership_transferred = True except BaseException: # On success ownership transfers to the train pump, which # releases this permit after consuming the committed group. - self._buffer_capacity.release() + if not ownership_transferred: + self._buffer_capacity.release() raise finally: - self._inflight_rollouts -= 1 - sem.release() + if not inflight_count_released: + self._inflight_rollouts -= 1 + if not generation_permit_released: + sem.release() if self._async_cfg.diagnostics: content = "" @@ -379,6 +618,9 @@ async def _train_pump(self) -> None: min_sample_version = None step_open = False calibration_batches: list[BatchedDataDict[Any]] = [] + consumed_metas: list[KVBatchMeta] = [] + consumed_group_count = 0 + step_finalizer_metrics: dict[str, list[float]] = {} with self._timer.time("total_step_time"): while groups_dispatched < grpo_cfg.num_prompts_per_step: @@ -435,9 +677,19 @@ async def _train_pump(self) -> None: await asyncio.sleep(0.005) continue - # Release buffer capacity - for _ in range(num_groups): - self._buffer_capacity.release() + consumed_metas.append(train_meta) + consumed_group_count += num_groups + selected_group_ids = { + sample_id.rsplit("_g", 1)[0] + for sample_id in train_meta.sample_ids + } + for group_id in selected_group_ids: + for name, value in self._finalizer_metrics_by_group.pop( + group_id, {} + ).items(): + step_finalizer_metrics.setdefault(name, []).append( + float(value) + ) # Compute prev_logprobs / ref_logprobs if ( @@ -509,19 +761,23 @@ async def _train_pump(self) -> None: else: min_sample_version = curr_min_sample_version - # Remove consumed sample_ids from the buffer - await self._call_dp( - "clear_samples", - sample_ids=list(train_meta.sample_ids), - partition_id=self._partition_id, - ) - groups_dispatched += num_groups with self._timer.time("policy_training"): result = await asyncio.to_thread(self._trainer.finish_train_step) + await self._cleanup_consumed_metas(consumed_metas) + for _ in range(consumed_group_count): + self._buffer_capacity.release() + step_metrics = aggregate_step_metrics(result) + step_metrics.update( + { + name: statistics.fmean(values) + for name, values in step_finalizer_metrics.items() + if values + } + ) step_metrics.update( reduce_advantage_pump_metrics(**self._step_log_dict) ) diff --git a/nemo_rl/algorithms/single_controller_utils/config.py b/nemo_rl/algorithms/single_controller_utils/config.py index 44cb67223f4..ed1ee8f966a 100644 --- a/nemo_rl/algorithms/single_controller_utils/config.py +++ b/nemo_rl/algorithms/single_controller_utils/config.py @@ -14,10 +14,11 @@ from __future__ import annotations +import warnings from dataclasses import dataclass, field from typing import Any, Literal, Optional -from pydantic import BaseModel, Field +from pydantic import BaseModel, Field, PositiveInt from nemo_rl.algorithms.async_utils.staleness_sampler import ( InOrderSamplerConfig, @@ -99,6 +100,15 @@ class TokenCaptureConfig(BaseModel, extra="allow"): # dir stays essentially empty on the gate path). None = derived at setup # under the run's log dir. capture_dir: Optional[str] = None + # Keep routed_experts out of canonical rows and assemble them on policy + # workers from strict staged-fragment plans. + defer_routed_experts_to_policy: bool = False + # Move finalization into a bounded CPU Ray actor pool. Disabled is the + # serial rollback path; enabled releases generation capacity before the + # actor RPC and overlaps finalization with the next rollout group. + finalizer_actor_pool_enabled: bool = False + # Fixed pool size for validation; actors are never automatically replaced. + num_finalizer_workers: PositiveInt = 2 class MasterConfig(BaseModel, extra="allow"): @@ -168,6 +178,33 @@ def validate_single_controller_config(master_config: MasterConfig) -> None: sampler_name=async_config.sampler.name, ) + token_capture_config = master_config.token_capture + if token_capture_config.defer_routed_experts_to_policy and not ( + token_capture_config.enabled + ): + raise ValueError( + "token_capture.defer_routed_experts_to_policy requires " + "token_capture.enabled=true" + ) + if ( + token_capture_config.finalizer_actor_pool_enabled + and not token_capture_config.enabled + ): + raise ValueError( + "token_capture.finalizer_actor_pool_enabled requires " + "token_capture.enabled=true" + ) + if ( + token_capture_config.finalizer_actor_pool_enabled + and token_capture_config.num_finalizer_workers + > async_config.max_buffered_rollouts + ): + warnings.warn( + "token_capture.num_finalizer_workers exceeds " + "async_rl.max_buffered_rollouts; excess finalizer actors cannot be busy", + stacklevel=2, + ) + # A non-zero reference-policy KL penalty makes the loss read # ``reference_policy_logprobs``, but the SC train pump only computes them # when ``skip_reference_policy_logprobs_calculation`` is false (see diff --git a/nemo_rl/algorithms/single_controller_utils/setup.py b/nemo_rl/algorithms/single_controller_utils/setup.py index f0ab927dafe..7f55b8d1353 100644 --- a/nemo_rl/algorithms/single_controller_utils/setup.py +++ b/nemo_rl/algorithms/single_controller_utils/setup.py @@ -93,6 +93,7 @@ class SingleControllerActorArgs: rollout_manager: RolloutManager tq_buffer: TQReplayBuffer partition_id: str + finalizer_actors: list[Any] def _build_clusters( @@ -640,18 +641,25 @@ def _build_generation_then_trainer( DP_TRAIN_FIELDS, fields_with_optional_routed_experts, ) - from nemo_rl.data_plane.tq_token_sink import ( + from nemo_rl.data_plane.schema import ( ROUTED_EXPERTS_FIELD as STAGING_ROUTED_EXPERTS_FIELD, ) from nemo_rl.data_plane.tq_token_sink import STAGING_FIELDS r3_enabled = router_replay_enabled(master_config.policy) + if token_capture_cfg.defer_routed_experts_to_policy and not r3_enabled: + raise ValueError( + "token_capture.defer_routed_experts_to_policy requires " + "policy.router_replay.enabled=true" + ) group_size = grpo_config.num_generations_per_prompt num_rollout_samples = master_config.async_rl.max_buffered_rollouts * group_size dp_client.register_partition( partition_id=partition_id, fields=fields_with_optional_routed_experts( - DP_TRAIN_FIELDS, enabled=r3_enabled + DP_TRAIN_FIELDS, + enabled=r3_enabled + and not token_capture_cfg.defer_routed_experts_to_policy, ), num_samples=num_rollout_samples, consumer_tasks=["prev_lp", "ref_lp", "train"], @@ -662,7 +670,7 @@ def _build_generation_then_trainer( fields=list(STAGING_FIELDS) + ([STAGING_ROUTED_EXPERTS_FIELD] if r3_enabled else []), num_samples=num_rollout_samples, - consumer_tasks=["finalize"], + consumer_tasks=["finalize", "prev_lp", "train"], ) # Host Gym's capture core in every vLLM DP leader (in-worker DP # client + TQTokenSink + the single install_capture call), and give @@ -718,18 +726,40 @@ def _build_generation_then_trainer( ), ) finalizer = None + finalizer_actors: list[Any] = [] if token_capture_cfg.enabled: from nemo_rl.experience.blackbox_finalizer import BlackboxFinalizer - finalizer = BlackboxFinalizer( - dp_client, - partition_id=partition_id, - staging_partition=token_capture_cfg.staging_partition, - pad_token_id=pad_id, - mixed_weight_version_policy=token_capture_cfg.mixed_weight_version_policy, - min_valid_fraction_per_group=token_capture_cfg.min_valid_fraction_per_group, - router_replay_enabled=router_replay_enabled(policy_config), - ) + if token_capture_cfg.finalizer_actor_pool_enabled: + from nemo_rl.experience.finalizer_actor import ( + FinalizerActorConfig, + create_finalizer_actors, + ) + + finalizer_actors = create_finalizer_actors( + dp_config, + FinalizerActorConfig( + partition_id=partition_id, + staging_partition=token_capture_cfg.staging_partition, + pad_token_id=pad_id, + mixed_weight_version_policy=token_capture_cfg.mixed_weight_version_policy, + min_valid_fraction_per_group=token_capture_cfg.min_valid_fraction_per_group, + router_replay_enabled=router_replay_enabled(policy_config), + defer_routed_experts_to_policy=token_capture_cfg.defer_routed_experts_to_policy, + ), + num_workers=token_capture_cfg.num_finalizer_workers, + ) + else: + finalizer = BlackboxFinalizer( + dp_client, + partition_id=partition_id, + staging_partition=token_capture_cfg.staging_partition, + pad_token_id=pad_id, + mixed_weight_version_policy=token_capture_cfg.mixed_weight_version_policy, + min_valid_fraction_per_group=token_capture_cfg.min_valid_fraction_per_group, + router_replay_enabled=router_replay_enabled(policy_config), + defer_routed_experts_to_policy=token_capture_cfg.defer_routed_experts_to_policy, + ) rollout_manager = RolloutManager( tokenizer=tokenizer, task_to_env=env_handles, @@ -765,5 +795,6 @@ def _build_generation_then_trainer( rollout_manager=rollout_manager, tq_buffer=tq_buffer, partition_id=partition_id, + finalizer_actors=finalizer_actors, ) return actor_args, setup_timing_metrics diff --git a/nemo_rl/data_plane/preshard.py b/nemo_rl/data_plane/preshard.py index f9ce2fdc6c7..9790cdd60cb 100644 --- a/nemo_rl/data_plane/preshard.py +++ b/nemo_rl/data_plane/preshard.py @@ -142,6 +142,7 @@ def shard_meta_for_dp( flat_idx.extend(idx_list) rank_sample_ids = [meta.sample_ids[i] for i in idx_list] rank_seqlens = [seq_lens[i] for i in idx_list] + rank_tags = [meta.tags[i] for i in idx_list] if meta.tags is not None else None rank_extra = dict(base_extra) # Per-shard packing metadata — set by ``shard_by_batch_size`` when # sequence_packing or dynamic_batching is enabled. Workers' @@ -165,6 +166,7 @@ def shard_meta_for_dp( sample_ids=rank_sample_ids, fields=meta.fields, sequence_lengths=rank_seqlens, + tags=rank_tags, extra_info=rank_extra, ) ) diff --git a/nemo_rl/data_plane/schema.py b/nemo_rl/data_plane/schema.py index 49cf79422e7..0d6a97df01f 100644 --- a/nemo_rl/data_plane/schema.py +++ b/nemo_rl/data_plane/schema.py @@ -62,6 +62,13 @@ DP_CALIB_INPUT_FIELDS = (INPUT_IDS, INPUT_LENGTHS, "multi_modal_inputs") ROUTED_EXPERTS_FIELD = "routed_experts" +ROUTED_LEN_FIELD = "routed_len" + +# Deferred route storage. Canonical rows carry one strict encoded route plan +# per tag; policy workers omit the absent canonical route column and assemble +# it from staging immediately before previous-policy logprob or training. +ROUTE_PLAN_TAG = "route_assembly_plan" +ROUTE_PASSTHROUGH_FLAG = "route_passthrough" # Per-sample 1D scalar fields. The TQ adapter promotes these to ``(N, 1)`` # on write to work around TQ v0.1.9's KVStorageManager schema/data mismatch on @@ -79,6 +86,7 @@ INPUT_LENGTHS, "total_reward", SAMPLE_MASK, + ROUTED_LEN_FIELD, } ) diff --git a/nemo_rl/data_plane/tq_token_sink.py b/nemo_rl/data_plane/tq_token_sink.py index 9464a7d0d90..330510d7321 100644 --- a/nemo_rl/data_plane/tq_token_sink.py +++ b/nemo_rl/data_plane/tq_token_sink.py @@ -32,6 +32,8 @@ from __future__ import annotations +import logging +from dataclasses import dataclass from typing import Any, Optional import ray @@ -44,15 +46,27 @@ StageResult, ) +from nemo_rl.data_plane.schema import ROUTED_EXPERTS_FIELD, ROUTED_LEN_FIELD + STAGING_FIELDS = [ "token_ids_delta", "token_mask_delta", "generation_logprobs_delta", "prev_len", "weight_version", + ROUTED_LEN_FIELD, ] +@dataclass(frozen=True) +class FetchedStagedCall: + """One explicitly identified small-column finalization fetch result.""" + + staging_key: str + snapshot: StagedCallSnapshot + routed_len: int + + def _call_dp(dp_client: Any, method_name: str, **kwargs: Any) -> Any: """Call a DataPlaneClient method on a local client or a Ray actor handle.""" method = getattr(dp_client, method_name) @@ -95,6 +109,7 @@ def stage(self, record: StagedCallRecord) -> StageResult: ), } routed = (record.extras or {}).get("routed_experts") + routed_len = 0 if routed is not None: # Worker sends full prompt+generation coverage of this call's # spliced sequence; the delta-aligned slice is [prev_len:]. @@ -119,6 +134,7 @@ def stage(self, record: StagedCallRecord) -> StageResult: experts = experts[record.prev_len :] if experts.shape[0] == delta_len: field_dict[ROUTED_EXPERTS_FIELD] = experts.unsqueeze(0) + routed_len = int(experts.shape[0]) else: logging.getLogger(__name__).warning( "routed_experts length %d does not cover delta %d " @@ -128,6 +144,7 @@ def stage(self, record: StagedCallRecord) -> StageResult: record.prev_len, key, ) + field_dict[ROUTED_LEN_FIELD] = torch.tensor([routed_len], dtype=torch.int64) fields = TensorDict(field_dict, batch_size=[1]) tags = [ { @@ -179,10 +196,15 @@ def clear(self, staging_keys: list[str]) -> None: class TQTokenSource: """Gym ``StagingSource`` over ``DataPlaneClient.get_samples``. - Rows are fetched one key at a time (deltas are jagged across calls) in - the order requested. A missing or unreadable row raises ``KeyError`` per - the protocol — the finalizer maps that to a placeholder, never a silent - skip. + All requested rows are fetched in a single batched ``get_samples`` call + (TQ returns jagged delta columns as nested tensors; ``_from_wire`` + preserves the raggedness), in the order requested. A missing or + unreadable row raises ``KeyError`` per the protocol — the finalizer maps + that to a placeholder, never a silent skip. TQ's field-readiness check + is all-or-nothing across a batch, so the extras fallback is batch-level: + extras-free runs land in the base schema exactly like the old per-key + probe, but a batch with *mixed* extras presence degrades every row to + the base schema (worker feature-gating makes presence uniform per run). """ def __init__(self, dp_client: Any, *, staging_partition: str) -> None: @@ -190,26 +212,108 @@ def __init__(self, dp_client: Any, *, staging_partition: str) -> None: self._staging_partition = staging_partition def fetch(self, staging_keys: list[str]) -> list[StagedCallSnapshot]: - snapshots: list[StagedCallSnapshot] = [] - for key in staging_keys: - _, _, call_id = key.rpartition("/") + """Fetch Gym snapshots, retaining legacy optional route materialization.""" + if not staging_keys: + return [] + try: + # Extras are optional per row (feature-gated at the worker); + # try the extended selection first, fall back to the base + # schema so extras-free rows keep fetching. try: - row = _call_dp( + rows = _call_dp( self._dp_client, "get_samples", - sample_ids=[key], + sample_ids=list(staging_keys), + partition_id=self._staging_partition, + select_fields=STAGING_FIELDS + [ROUTED_EXPERTS_FIELD], + ) + except Exception: # noqa: BLE001 — field-not-present probe + rows = _call_dp( + self._dp_client, + "get_samples", + sample_ids=list(staging_keys), partition_id=self._staging_partition, select_fields=STAGING_FIELDS, ) - snapshots.append(_row_to_snapshot(call_id, row)) - except KeyError: - raise - except Exception as error: # noqa: BLE001 — protocol maps any miss to KeyError - raise KeyError( - f"staged row {key!r} could not be fetched: {error}" - ) from error + except Exception as error: # noqa: BLE001 — protocol maps any miss to KeyError + raise KeyError( + f"staged rows for {len(staging_keys)} keys could not be " + f"fetched from {self._staging_partition!r}: {error}" + ) from error + # TQ's kv path only errors when *zero* keys resolve; a partial miss + # returns fewer rows with no error. Guard explicitly so a lost row + # rejects the rollout as missing_staging_row instead of surfacing + # later as a misleading digest mismatch from misaligned zipping. + n_rows = int(rows.batch_size[0]) if len(rows.batch_size) else 0 + if n_rows != len(staging_keys): + raise KeyError( + f"staged rows missing: requested {len(staging_keys)} keys " + f"from {self._staging_partition!r}, got {n_rows} rows" + ) + # Row order mirrors the requested key order; the finalizer's digest + # recomputation is the byte-exact backstop if that ever breaks. + snapshots: list[StagedCallSnapshot] = [] + for index, key in enumerate(staging_keys): + _, _, call_id = key.rpartition("/") + snapshots.append(_row_to_snapshot(call_id, _select_row(rows, index))) return snapshots + def fetch_for_finalization( + self, staging_keys: list[str] + ) -> list[FetchedStagedCall]: + """Fetch only digest-covered columns plus required route length metadata.""" + if not staging_keys: + return [] + if len(set(staging_keys)) != len(staging_keys): + raise KeyError("finalization staging request contains duplicate keys") + try: + rows = _call_dp( + self._dp_client, + "get_samples", + sample_ids=list(staging_keys), + partition_id=self._staging_partition, + select_fields=STAGING_FIELDS, + ) + except Exception as error: # noqa: BLE001 — protocol maps misses to KeyError + raise KeyError( + f"staged rows for {len(staging_keys)} keys could not be " + f"fetched from {self._staging_partition!r}: {error}" + ) from error + n_rows = int(rows.batch_size[0]) if len(rows.batch_size) else 0 + if n_rows != len(staging_keys): + raise KeyError( + f"staged rows missing: requested {len(staging_keys)} keys " + f"from {self._staging_partition!r}, got {n_rows} rows" + ) + fetched: list[FetchedStagedCall] = [] + for index, key in enumerate(staging_keys): + _, separator, call_id = key.rpartition("/") + if not separator or not call_id: + raise KeyError(f"invalid staging key without call id: {key!r}") + row = _select_row(rows, index) + fetched.append( + FetchedStagedCall( + staging_key=key, + snapshot=_row_to_snapshot(call_id, row), + routed_len=_row_scalar_int(row, ROUTED_LEN_FIELD), + ) + ) + return fetched + + +def _select_row(rows: TensorDict, index: int) -> dict[str, torch.Tensor]: + """Slice one row out of a batched fetch, restoring single-row shapes. + + ``_row_to_snapshot`` predates batching and expects each field with a + leading batch dim of 1 (the shape a single-key ``get_samples`` returns), + so re-add it after indexing. Indexing a nested tensor yields that row's + dense component, which is exactly the jagged-row payload. + """ + row: dict[str, torch.Tensor] = {} + for field in rows.keys(): + row[str(field)] = rows.get(field)[index].unsqueeze(0) + return row + def _row_to_snapshot(call_id: str, row: Any) -> StagedCallSnapshot: def _leaf(name: str) -> torch.Tensor: @@ -219,6 +323,16 @@ def _leaf(name: str) -> torch.Tensor: prev_len = int(_leaf("prev_len")[0].item()) weight_version: Optional[int] = int(_leaf("weight_version")[0].item()) + extras: Optional[dict[str, Any]] = None + try: + routed = row[ROUTED_EXPERTS_FIELD] + except KeyError: + routed = None + if routed is not None: + # [1, delta_len, ...] -> [delta_len, ...]; keep the nested list wire + # shape so the snapshot stays a plain-pydantic record. + experts = routed[0] if routed.dim() > 2 or routed.shape[0] == 1 else routed + extras = {"routed_experts": experts.tolist()} return StagedCallSnapshot( call_id=call_id, prev_len=prev_len, @@ -226,4 +340,33 @@ def _leaf(name: str) -> torch.Tensor: token_mask_delta=[float(m) for m in _leaf("token_mask_delta").tolist()], logprobs_delta=[float(p) for p in _leaf("generation_logprobs_delta").tolist()], weight_version=weight_version, + extras=extras, ) + + +def _row_scalar_int(row: Any, field_name: str) -> int: + """Read one required scalar from a single-row TQ result.""" + value = row[field_name] + if not isinstance(value, torch.Tensor): + raise TypeError( + f"staging field {field_name!r} must be a tensor, got {type(value).__name__}" + ) + integer_dtypes = { + torch.uint8, + torch.int8, + torch.int16, + torch.int32, + torch.int64, + } + if value.dtype not in integer_dtypes: + raise TypeError( + f"staging field {field_name!r} must use an integer dtype, got {value.dtype}" + ) + tensor = value[0] if value.dim() > 1 or value.numel() > 1 else value + flattened = tensor.reshape(-1) + if flattened.numel() != 1: + raise ValueError( + f"staging field {field_name!r} must contain one scalar, got " + f"shape {tuple(value.shape)}" + ) + return int(flattened[0].item()) diff --git a/nemo_rl/data_plane/worker_mixin.py b/nemo_rl/data_plane/worker_mixin.py index b75ff9d8805..6f6231473ce 100644 --- a/nemo_rl/data_plane/worker_mixin.py +++ b/nemo_rl/data_plane/worker_mixin.py @@ -27,6 +27,9 @@ from __future__ import annotations +import logging +import time +from collections import Counter from typing import TYPE_CHECKING, Any, Literal, Optional import torch @@ -39,6 +42,9 @@ GLOBAL_FORWARD_PAD_SEQLEN, MICRO_BATCH_INDICES, MICRO_BATCH_LENGTHS, + ROUTE_PASSTHROUGH_FLAG, + ROUTE_PLAN_TAG, + ROUTED_EXPERTS_FIELD, Layout, ) from nemo_rl.distributed.batched_data_dict import BatchedDataDict, SequencePackingArgs @@ -136,6 +142,7 @@ class TQWorkerMixin: """ _dp_client: Optional[DataPlaneClient] = None + _route_fallback_counts: Counter[str] = Counter() def setup_data_plane(self, cfg: DataPlaneConfig) -> None: """Connect this worker process's client to the existing TQ controller. @@ -150,6 +157,7 @@ def setup_data_plane(self, cfg: DataPlaneConfig) -> None: ) if self._dp_client is not None: return + self._route_fallback_counts = Counter() from nemo_rl.data_plane import build_data_plane_client # bootstrap=False — the driver already created the named @@ -176,6 +184,12 @@ def _get_replica_group(self) -> Optional[Any]: """ return None + def _routed_experts_dimensions(self) -> tuple[int, int]: + """Return model-owned ``(num_moe_layers, top_k)`` route dimensions.""" + raise NotImplementedError( + "the router-replay policy worker must provide route dimensions" + ) + def _pad_value_dict(self) -> dict[str, Any]: """Per-field pad value used by :func:`materialize` to detile the jagged wire format. @@ -255,6 +269,7 @@ def _fetch( pad_value_dict=pad_value_dict, pad_to_seqlen=pad_to_seqlen, ) + data = self._maybe_assemble_routed_experts(meta, data) else: data = None data = _broadcast_batched_data_dict( @@ -286,6 +301,7 @@ def _fetch( pad_value_dict=pad_value_dict, pad_to_seqlen=pad_to_seqlen, ) + data = self._maybe_assemble_routed_experts(meta, data) attach_message_log_view(data) trace_tq_fetch_payload( stage=meta.task_name or "unknown", @@ -296,6 +312,186 @@ def _fetch( data = preprocess(self, data) return data + def _fetch_route_fragments( + self, + *, + keys: list[str], + partition_id: str, + ) -> dict[str, torch.Tensor]: + """Fetch a unique key set in one request and preserve request identity.""" + if not keys: + return {} + rows = self._require_dp_client().get_samples( + sample_ids=keys, + partition_id=partition_id, + select_fields=[ROUTED_EXPERTS_FIELD], + ) + n_rows = int(rows.batch_size[0]) if len(rows.batch_size) else 0 + if n_rows != len(keys): + raise KeyError(f"requested {len(keys)} route rows, got {n_rows}") + column = rows.get(ROUTED_EXPERTS_FIELD) + return {key: column[index] for index, key in enumerate(keys)} + + def _route_fragments_by_row( + self, + plans: list[Any], + ) -> tuple[list[dict[str, torch.Tensor]], int, float]: + """Use one normal-path batch read; isolate error retries per rollout.""" + from nemo_rl.experience.route_plan import decode_route_plan + + decoded = [decode_route_plan(plan) for plan in plans] + partitions = {plan.staging_partition for plan in decoded} + if len(partitions) != 1: + raise RuntimeError( + f"deferred route plans use mixed staging partitions: {partitions}" + ) + partition_id = next(iter(partitions)) + keys = list( + dict.fromkeys( + span.staging_key + for plan in decoded + for span in plan.spans + if span.staged_route_len > 0 + ) + ) + fetch_start = time.perf_counter() + try: + fragments = self._fetch_route_fragments( + keys=keys, + partition_id=partition_id, + ) + except Exception as batch_error: # noqa: BLE001 - isolate fallback by rollout + logging.getLogger(__name__).warning( + "deferred route batch fetch failed; isolating by rollout: %s", + batch_error, + ) + per_row: list[dict[str, torch.Tensor]] = [] + for plan in decoded: + row_keys = list( + dict.fromkeys( + span.staging_key + for span in plan.spans + if span.staged_route_len > 0 + ) + ) + try: + per_row.append( + self._fetch_route_fragments( + keys=row_keys, + partition_id=partition_id, + ) + ) + except Exception: # noqa: BLE001 - this rollout becomes sentinel + per_row.append({}) + return ( + per_row, + len(keys), + (time.perf_counter() - fetch_start) * 1000.0, + ) + return ( + [fragments for _ in decoded], + len(keys), + (time.perf_counter() - fetch_start) * 1000.0, + ) + + def _maybe_assemble_routed_experts( + self, + meta: "KVBatchMeta", + data: BatchedDataDict[Any], + ) -> BatchedDataDict[Any]: + """Materialize deferred routes at the policy worker consumption boundary.""" + if not (meta.extra_info or {}).get(ROUTE_PASSTHROUGH_FLAG): + return data + + from nemo_rl.experience.route_plan import ( + classify_route_span, + decode_route_plan, + ) + from nemo_rl.models.generation.interfaces import ( + ROUTED_EXPERTS_MISSING_ROUTE_SENTINEL, + ) + + tags = meta.tags or [] + if len(tags) != len(meta.sample_ids): + raise RuntimeError( + "deferred route tags must align with sample_ids: " + f"{len(tags)} tags for {len(meta.sample_ids)} rows" + ) + encoded_plans = [] + for index, tag in enumerate(tags): + if ROUTE_PLAN_TAG not in tag: + raise RuntimeError( + f"deferred route plan missing for row {meta.sample_ids[index]!r}" + ) + encoded_plans.append(tag[ROUTE_PLAN_TAG]) + plans = [decode_route_plan(plan) for plan in encoded_plans] + fragments_by_row, _, _ = self._route_fragments_by_row(encoded_plans) + + num_moe_layers, top_k = self._routed_experts_dimensions() + input_ids = data["input_ids"] + input_lengths = data["input_lengths"].reshape(-1) + routed = torch.full( + ( + len(meta.sample_ids), + int(input_ids.shape[1]), + num_moe_layers, + top_k, + ), + ROUTED_EXPERTS_MISSING_ROUTE_SENTINEL, + dtype=torch.int16, + ) + request_fallbacks: Counter[str] = Counter() + for row_index, (plan, fragments) in enumerate(zip(plans, fragments_by_row)): + reason: Optional[str] = None + canonical_len = int(input_lengths[row_index].item()) + if canonical_len != plan.expected_token_length: + reason = "canonical_length_mismatch" + position = 0 + if reason is None: + for span in plan.spans: + contribution = span.carry_len + span.generation_len + mode = classify_route_span(span) + if mode != "sentinel": + fragment = fragments.get(span.staging_key) + if fragment is None: + reason = "missing_fragment" + break + if fragment.dim() != 3: + reason = "fragment_rank" + break + if int(fragment.shape[0]) != span.staged_route_len: + reason = "fragment_length" + break + if tuple(fragment.shape[1:]) != (num_moe_layers, top_k): + reason = "fragment_model_shape" + break + if mode == "full": + routed[row_index, position : position + contribution] = ( + fragment.to(torch.int16) + ) + else: + tail_start = position + span.carry_len + routed[row_index, tail_start : position + contribution] = ( + fragment[-span.generation_len :].to(torch.int16) + ) + position += contribution + if reason is None and plan.spans and position != canonical_len: + reason = "assembled_length_mismatch" + if reason is not None: + routed[row_index].fill_(ROUTED_EXPERTS_MISSING_ROUTE_SENTINEL) + request_fallbacks[reason] += 1 + + self._route_fallback_counts.update(request_fallbacks) + if request_fallbacks: + logging.getLogger(__name__).warning( + "deferred route fallback for %d/%d rollouts: %s", + sum(request_fallbacks.values()), + len(plans), + dict(request_fallbacks), + ) + data[ROUTED_EXPERTS_FIELD] = routed + return data + def _apply_packing_prep(self, data: BatchedDataDict[Any]) -> BatchedDataDict[Any]: """Re-derive ``micro_batch_indices`` / ``micro_batch_lengths`` on the local slice. diff --git a/nemo_rl/experience/blackbox_finalizer.py b/nemo_rl/experience/blackbox_finalizer.py index 0052309f7f3..f6e0932faee 100644 --- a/nemo_rl/experience/blackbox_finalizer.py +++ b/nemo_rl/experience/blackbox_finalizer.py @@ -24,20 +24,32 @@ placeholders copy ``prompt_ids_for_adv`` from a valid sibling so per-prompt baselines stay well-formed. -The finalizer is the only reader of the staging partition and clears a -group's staged rows after its canonical rows are durably published. +In deferred-route mode the finalizer reads only small columns, publishes a +strict route plan beside each canonical row, and leaves staged route fragments +live until policy consumption completes. The direct-route rollback path keeps +the prior finalizer-side materialization behavior. """ from __future__ import annotations import math +import time from dataclasses import dataclass, field from typing import Any, Optional import torch from nemo_rl.data_plane import KVBatchMeta +from nemo_rl.data_plane.schema import ROUTE_PLAN_TAG from nemo_rl.data_plane.tq_token_sink import TQTokenSink, TQTokenSource +from nemo_rl.experience.route_plan import ( + ROUTE_PLAN_SCHEMA_VERSION, + RouteAssemblyPlan, + RouteSpan, + classify_route_span, + encode_route_plan, + encoded_route_plan_size_bytes, +) from nemo_rl.experience.payload import pack_payload from nemo_rl.experience.row_dump import maybe_dump_train_rows from nemo_rl.models.generation.interfaces import ( @@ -63,6 +75,7 @@ class FinalizedRollout: # Router replay (R3): [len(token_ids)][num_moe_layers][topk] from the # rebuilt chain; None when the rollout staged no extras. routed_experts: Optional[list] = None + route_plan: Optional[RouteAssemblyPlan] = None @dataclass @@ -92,6 +105,7 @@ def __init__( mixed_weight_version_policy: str, min_valid_fraction_per_group: Optional[float], router_replay_enabled: bool = False, + defer_routed_experts_to_policy: bool = False, ) -> None: self._dp_client = dp_client self._partition_id = partition_id @@ -99,6 +113,12 @@ def __init__( self._mixed_weight_version_policy = mixed_weight_version_policy self._min_valid_fraction = min_valid_fraction_per_group self._router_replay_enabled = router_replay_enabled + self._defer_routed_experts_to_policy = defer_routed_experts_to_policy + if self._defer_routed_experts_to_policy and not self._router_replay_enabled: + raise ValueError( + "defer_routed_experts_to_policy requires router replay to be enabled" + ) + self._staging_partition = staging_partition # (num_moe_layers, topk), learned from the first rebuilt row that # carries routes; placeholder-only groups need it to shape their # sentinel tensors consistently with the model. @@ -152,11 +172,42 @@ def rejected(reason: str, staging_keys: list[str]) -> FinalizedRollout: return rejected("capture_poisoned", staging_keys) if not parsed.manifest: return rejected("empty_manifest", staging_keys) + if len(set(staging_keys)) != len(staging_keys): + return rejected( + "duplicate_staging_key", + list(dict.fromkeys(staging_keys)), + ) + records_by_call = {record.call_id: record for record in parsed.manifest} + if len(records_by_call) != len(parsed.manifest): + return rejected("duplicate_manifest_call_id", staging_keys) try: - snapshots = self._source.fetch(staging_keys) + fetched = ( + self._source.fetch_for_finalization(staging_keys) + if self._defer_routed_experts_to_policy + else None + ) + snapshots = ( + [item.snapshot for item in fetched] + if fetched is not None + else self._source.fetch(staging_keys) + ) except KeyError as error: return rejected(f"missing_staging_row:{error}", staging_keys) + except (TypeError, ValueError) as error: + return rejected(f"invalid_staging_row:{error}", staging_keys) + fetched_by_call = {} + if fetched is not None: + for record, item in zip(parsed.manifest, fetched): + if item.staging_key != record.staging_key: + return rejected( + f"staging_key_mismatch:{record.call_id}", staging_keys + ) + if item.snapshot.call_id != record.call_id: + return rejected(f"call_id_mismatch:{record.call_id}", staging_keys) + fetched_by_call[record.call_id] = item + if len(fetched_by_call) != len(fetched): + return rejected("duplicate_fetched_call_id", staging_keys) for record, snapshot in zip(parsed.manifest, snapshots): if not ( @@ -204,12 +255,57 @@ def rejected(reason: str, staging_keys: list[str]) -> FinalizedRollout: ) except (RebuildError, NotImplementedError) as error: return rejected(f"rebuild_failed:{error}", staging_keys) - weight_versions = [record.weight_version for record in parsed.manifest] min_wv, max_wv = min(weight_versions), max(weight_versions) if self._mixed_weight_version_policy == "reject" and min_wv != max_wv: return rejected(f"mixed_weight_versions:{min_wv}..{max_wv}", staging_keys) + route_plan = None + if self._router_replay_enabled and self._defer_routed_experts_to_policy: + link_spans = row.link_spans + if link_spans is None: + return rejected("missing_link_spans", staging_keys) + route_spans: list[RouteSpan] = [] + seen_span_call_ids: set[str] = set() + for call_id, carry_len, generation_len in link_spans: + if call_id in seen_span_call_ids: + return rejected(f"duplicate_route_span:{call_id}", staging_keys) + seen_span_call_ids.add(call_id) + record = records_by_call.get(call_id) + item = fetched_by_call.get(call_id) + if record is None or item is None: + return rejected(f"route_span_identity:{call_id}", staging_keys) + if item.routed_len not in (0, record.delta_len): + return rejected(f"routed_len_mismatch:{call_id}", staging_keys) + if generation_len < 0 or generation_len > record.delta_len: + return rejected( + f"route_generation_span_mismatch:{call_id}", staging_keys + ) + if carry_len < 0: + return rejected( + f"route_carry_span_mismatch:{call_id}", staging_keys + ) + span = RouteSpan( + staging_key=record.staging_key, + carry_len=int(carry_len), + generation_len=int(generation_len), + staged_route_len=item.routed_len, + ) + classify_route_span(span) + route_spans.append(span) + if sum(span.carry_len + span.generation_len for span in route_spans) != len( + row.token_ids + ): + return rejected("route_span_length_mismatch", staging_keys) + route_plan = RouteAssemblyPlan( + schema_version=ROUTE_PLAN_SCHEMA_VERSION, + staging_partition=self._staging_partition, + spans=tuple(route_spans), + cleanup_staging_keys=tuple(staging_keys), + expected_token_length=len(row.token_ids), + ) + encode_route_plan(route_plan) + return FinalizedRollout( rollout_id=rollout_id, valid=True, @@ -224,7 +320,12 @@ def rejected(reason: str, staging_keys: list[str]) -> FinalizedRollout: max_wv=max_wv, # getattr: pre-R3 Gym pins' LinearizedRow has no routed_experts; # capture-R3 then degrades to the sentinel/group-drop path. - routed_experts=getattr(row, "routed_experts", None), + routed_experts=( + None + if self._defer_routed_experts_to_policy + else getattr(row, "routed_experts", None) + ), + route_plan=route_plan, ) # ── per group ─────────────────────────────────────────────────────────── @@ -248,10 +349,12 @@ def finalize_group( assert len(rollout_ids) == len(receipts) == len(rewards), ( "rollout_ids, receipts, and rewards must be parallel" ) + _group_t0 = time.perf_counter() rows = [ self.finalize_rollout(rollout_id, receipt, reward=reward) for rollout_id, receipt, reward in zip(rollout_ids, receipts, rewards) ] + _rollouts_ms = (time.perf_counter() - _group_t0) * 1000.0 valid_rows = [row for row in rows if row.valid] staging_keys = [key for row in rows for key in row.staging_keys] metrics = { @@ -291,6 +394,7 @@ def finalize_group( dropped=True, ) + _tensorize_t0 = time.perf_counter() # Placeholders borrow a valid sibling's prompt ids so per-prompt # baselines group correctly; an all-placeholder group uses a single # pad token (its rows all carry sample_mask 0 and never train). @@ -326,9 +430,9 @@ def finalize_group( "prompt_ids_for_adv": prompt_ids_for_adv, "total_reward": rewards_t, } - if self._router_replay_enabled: + if self._router_replay_enabled and not self._defer_routed_experts_to_policy: has_routed_row = any(r.valid and r.routed_experts for r in rows) - if not has_routed_row and self._routed_dims is None: + if not has_routed_row and self._routed_dims is None and not valid_rows: # Nothing to learn (L, K) from yet — e.g. an all-poisoned # group before the first healthy rollout. Dropping loses no # training signal (no valid rows or routes) and keeps the @@ -355,6 +459,32 @@ def finalize_group( sample_ids, fields, tags = pack_payload( train_batch, weight_version=group_min_wv, group_id=group_id ) + if self._defer_routed_experts_to_policy: + encoded_sizes = 0 + span_count = 0 + for tag, row, expected_length in zip(tags, rows, seq_lens): + plan = row.route_plan + if plan is None: + plan = RouteAssemblyPlan( + schema_version=ROUTE_PLAN_SCHEMA_VERSION, + staging_partition=self._staging_partition, + spans=(), + cleanup_staging_keys=tuple(row.staging_keys), + expected_token_length=expected_length, + ) + encoded = encode_route_plan(plan) + tag[ROUTE_PLAN_TAG] = encoded + encoded_sizes += encoded_route_plan_size_bytes(plan) + span_count += len(plan.spans) + metrics["finalize/route_plan_span_count"] = float(span_count) + metrics["finalize/route_plan_encoded_bytes"] = float(encoded_sizes) + valid_route_rows = sum( + 1 for row in valid_rows if row.route_plan and row.route_plan.spans + ) + if valid_rows: + metrics["finalize/routed_experts_row_coverage"] = ( + valid_route_rows / len(valid_rows) + ) maybe_dump_train_rows( source="finalizer", group_id=group_id, @@ -366,6 +496,8 @@ def finalize_group( "canonical sample ids must equal the gate-registered rollout ids: " f"{sample_ids} != {rollout_ids}" ) + _tensorize_ms = (time.perf_counter() - _tensorize_t0) * 1000.0 + _put_t0 = time.perf_counter() self._call_dp( "put_samples", sample_ids=sample_ids, @@ -373,8 +505,19 @@ def finalize_group( fields=fields, tags=tags, ) - self._clear_staging(staging_keys) - + _put_ms = (time.perf_counter() - _put_t0) * 1000.0 + _clear_ms = 0.0 + if not self._defer_routed_experts_to_policy: + _clear_t0 = time.perf_counter() + self._clear_staging(staging_keys) + _clear_ms = (time.perf_counter() - _clear_t0) * 1000.0 + # Per-step W&B breakdown of training-row assembly (capture arm) rides + # FinalizedGroup.metrics into the controller's rollout metrics. + metrics["row_assembly/rollouts_ms"] = _rollouts_ms + metrics["row_assembly/tensorize_ms"] = _tensorize_ms + metrics["row_assembly/tq_put_ms"] = _put_ms + if not self._defer_routed_experts_to_policy: + metrics["row_assembly/clear_staging_ms"] = _clear_ms meta = KVBatchMeta( partition_id=self._partition_id, task_name="train", @@ -387,8 +530,7 @@ def finalize_group( meta=meta, group_min_wv=group_min_wv, group_max_wv=group_max_wv, - # Already cleared; nothing left for eviction to clear. - staging_keys=[], + staging_keys=(staging_keys if self._defer_routed_experts_to_policy else []), metrics=metrics, ) @@ -469,11 +611,11 @@ def _clear_staging(self, staging_keys: list[str]) -> None: return try: self._staging.clear(staging_keys) - except Exception as error: # noqa: BLE001 — cleanup must not fail the group; TTL sweeps leftovers - print( - f" finalize: staging clear failed ({error}); TTL will sweep", - flush=True, - ) + except Exception as error: + raise RuntimeError( + "finalizer staging cleanup failed for known keys " + f"partition={self._staging_partition!r}, keys={staging_keys!r}" + ) from error def _call_dp(self, method_name: str, **kwargs: Any) -> Any: import ray diff --git a/nemo_rl/experience/finalizer_actor.py b/nemo_rl/experience/finalizer_actor.py new file mode 100644 index 00000000000..ab85af2ab85 --- /dev/null +++ b/nemo_rl/experience/finalizer_actor.py @@ -0,0 +1,150 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""CPU Ray actors for metadata-only token-capture finalization.""" + +from __future__ import annotations + +from dataclasses import dataclass, fields, is_dataclass +from typing import Any, Optional + +import ray +import torch + +from nemo_rl.data_plane import DataPlaneConfig, build_data_plane_client +from nemo_rl.experience.blackbox_finalizer import BlackboxFinalizer, FinalizedGroup + +_FORBIDDEN_RPC_KEYS = frozenset( + { + "input_ids", + "token_ids", + "token_ids_delta", + "token_mask", + "token_mask_delta", + "generation_logprobs", + "generation_logprobs_delta", + "logprobs_delta", + "routed_experts", + } +) + + +@dataclass(frozen=True) +class FinalizationRequest: + """Metadata-only input for one prompt group's finalization.""" + + group_id: str + rollout_ids: tuple[str, ...] + receipts: tuple[Optional[dict[str, Any]], ...] + rewards: tuple[float, ...] + fallback_weight_version: int + + +@dataclass(frozen=True) +class FinalizerActorConfig: + """Internal constructor values shared by every finalizer actor.""" + + partition_id: str + staging_partition: str + pad_token_id: int + mixed_weight_version_policy: str + min_valid_fraction_per_group: Optional[float] + router_replay_enabled: bool + defer_routed_experts_to_policy: bool + + +def assert_metadata_only(value: Any, *, path: str = "rpc") -> None: + """Reject tensors and known heavy row fields reachable from an RPC graph.""" + if isinstance(value, torch.Tensor): + raise TypeError( + f"{path} contains a torch.Tensor with shape {tuple(value.shape)}" + ) + if value is None or isinstance(value, (str, int, float, bool)): + return + if is_dataclass(value) and not isinstance(value, type): + for field_info in fields(value): + assert_metadata_only( + getattr(value, field_info.name), + path=f"{path}.{field_info.name}", + ) + return + if isinstance(value, dict): + for key, item in value.items(): + if key in _FORBIDDEN_RPC_KEYS: + raise TypeError(f"{path} contains forbidden heavy field {key!r}") + assert_metadata_only(key, path=f"{path}.key") + assert_metadata_only(item, path=f"{path}[{key!r}]") + return + if isinstance(value, (list, tuple)): + for index, item in enumerate(value): + assert_metadata_only(item, path=f"{path}[{index}]") + return + raise TypeError(f"{path} contains unsupported RPC type {type(value).__name__}") + + +@ray.remote( + num_cpus=1, + num_gpus=0, + max_restarts=0, + max_task_retries=0, +) +class FinalizerActor: # pragma: no cover + """Own a connect-only TQ client and lightweight finalizer in one process.""" + + def __init__( + self, + dp_config: DataPlaneConfig, + config: FinalizerActorConfig, + ) -> None: + dp_client = build_data_plane_client(dp_config, bootstrap=False) + self._finalizer = BlackboxFinalizer( + dp_client, + partition_id=config.partition_id, + staging_partition=config.staging_partition, + pad_token_id=config.pad_token_id, + mixed_weight_version_policy=config.mixed_weight_version_policy, + min_valid_fraction_per_group=config.min_valid_fraction_per_group, + router_replay_enabled=config.router_replay_enabled, + defer_routed_experts_to_policy=config.defer_routed_experts_to_policy, + ) + + def finalize(self, request: FinalizationRequest) -> FinalizedGroup: + """Finalize one request without allowing tensor payloads across Ray RPC.""" + assert_metadata_only(request) + if not ( + len(request.rollout_ids) == len(request.receipts) == len(request.rewards) + ): + raise ValueError( + "finalizer request rollout_ids, receipts, and rewards must be parallel" + ) + result = self._finalizer.finalize_group( + request.group_id, + list(request.rollout_ids), + list(request.receipts), + list(request.rewards), + fallback_weight_version=request.fallback_weight_version, + ) + assert_metadata_only(result) + return result + + +def create_finalizer_actors( + dp_config: DataPlaneConfig, + config: FinalizerActorConfig, + *, + num_workers: int, +) -> list[Any]: + """Construct the fixed validation pool after TQ partitions are registered.""" + if num_workers <= 0: + raise ValueError(f"num_finalizer_workers must be positive, got {num_workers}") + return [FinalizerActor.remote(dp_config, config) for _ in range(num_workers)] diff --git a/nemo_rl/experience/rollout_manager.py b/nemo_rl/experience/rollout_manager.py index 94f1b1c63cd..b0e1d46e186 100644 --- a/nemo_rl/experience/rollout_manager.py +++ b/nemo_rl/experience/rollout_manager.py @@ -16,7 +16,7 @@ import copy import json import uuid -from typing import Any, Optional +from typing import TYPE_CHECKING, Any, Optional import torch from transformers import PreTrainedTokenizerBase @@ -44,6 +44,9 @@ TokenizerType = PreTrainedTokenizerBase +if TYPE_CHECKING: + from nemo_rl.experience.finalizer_actor import FinalizationRequest + class AsyncRolloutImpl: """Manages per-prompt multi-turn rollouts, producing a PromptGroupRecord per call. @@ -840,7 +843,11 @@ async def generate_and_push( "generate_and_push requires tq_buffer to be set at __init__" ) if self._finalizer is not None: - await self._generate_and_finalize(input_sample, target_step=target_step) + await self._generate_and_finalize( + input_sample, + target_step=target_step, + inflight_registry=inflight_registry, + ) return start_version = self._weight_version group_id = self._tq_buffer.reserve( @@ -877,7 +884,11 @@ async def generate_and_push( raise async def _generate_and_finalize( - self, input_sample: DatumSpec, *, target_step: Optional[int] = None + self, + input_sample: DatumSpec, + *, + target_step: Optional[int] = None, + inflight_registry: Optional[dict[str, tuple[asyncio.Task[None], int]]] = None, ) -> None: """Token-capture dispatch: receipts in, canonical rows via the finalizer. @@ -899,7 +910,15 @@ async def _generate_and_finalize( rollout_ids=rollout_ids, ) try: - record = await self.run_rollout(input_sample, rollout_ids=rollout_ids) + if inflight_registry is not None: + current_task = asyncio.current_task() + assert current_task is not None + inflight_registry[group_id] = (current_task, start_version) + try: + record = await self.run_rollout(input_sample, rollout_ids=rollout_ids) + finally: + if inflight_registry is not None: + inflight_registry.pop(group_id, None) receipts = [c.env_extras.get("ng_receipt") for c in record.completions] rewards = [float(c.reward) for c in record.completions] finalized = await asyncio.to_thread( @@ -935,6 +954,77 @@ async def _generate_and_finalize( await nemo_gym_env.fail_rollouts.remote( rollout_ids, reason="dispatch_failed" ) - except Exception as error: # noqa: BLE001 — TTL is the backstop - print(f"fail_rollouts({group_id}) failed: {error}", flush=True) + except Exception as error: + raise RuntimeError( + "failed to close Gym rollout ownership after dispatch " + f"failure for group {group_id!r}" + ) from error + raise + + async def generate_for_finalization( + self, + input_sample: DatumSpec, + *, + target_step: Optional[int] = None, + inflight_registry: Optional[dict[str, tuple[asyncio.Task[None], int]]] = None, + ) -> "FinalizationRequest": + """Run capture generation and return a metadata-only actor request. + + The replay-buffer slot remains reserved and unready. The caller owns + finalizer submission and must either commit the returned group or stop + the validation run on an unknown publication outcome. + """ + from nemo_rl.experience.finalizer_actor import FinalizationRequest + + assert self._tq_buffer is not None, ( + "generate_for_finalization requires tq_buffer to be set at __init__" + ) + if self._finalizer is not None: + raise RuntimeError( + "generate_for_finalization cannot be combined with a local finalizer" + ) + start_version = self._weight_version + group_id = str(uuid.uuid4()) + rollout_ids = tuple( + f"{group_id}_g{i}" for i in range(self._num_generations_per_prompt) + ) + self._tq_buffer.reserve( + weight_version=start_version, + target_step=target_step, + group_id=group_id, + rollout_ids=list(rollout_ids), + ) + try: + if inflight_registry is not None: + current_task = asyncio.current_task() + assert current_task is not None + inflight_registry[group_id] = (current_task, start_version) + try: + record = await self.run_rollout( + input_sample, + rollout_ids=list(rollout_ids), + ) + finally: + if inflight_registry is not None: + inflight_registry.pop(group_id, None) + receipts = tuple(c.env_extras.get("ng_receipt") for c in record.completions) + rewards = tuple(float(c.reward) for c in record.completions) + request = FinalizationRequest( + group_id=group_id, + rollout_ids=rollout_ids, + receipts=receipts, + rewards=rewards, + fallback_weight_version=start_version, + ) + from nemo_rl.experience.finalizer_actor import assert_metadata_only + + assert_metadata_only(request) + return request + except BaseException: + self._tq_buffer.abort(group_id) + nemo_gym_env = self._env_handles.get("nemo_gym") + if nemo_gym_env is not None: + await nemo_gym_env.fail_rollouts.remote( + list(rollout_ids), reason="dispatch_failed" + ) raise diff --git a/nemo_rl/experience/route_plan.py b/nemo_rl/experience/route_plan.py new file mode 100644 index 00000000000..8ecb092b7ac --- /dev/null +++ b/nemo_rl/experience/route_plan.py @@ -0,0 +1,230 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Strict metadata contract for deferred routed-expert assembly.""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from typing import Any, Literal + +ROUTE_PLAN_SCHEMA_VERSION = 1 + + +@dataclass(frozen=True) +class RouteSpan: + """One root-first route contribution to a canonical rollout row.""" + + staging_key: str + carry_len: int + generation_len: int + staged_route_len: int + + +@dataclass(frozen=True) +class RouteAssemblyPlan: + """How a policy worker reconstructs routes from staged fragments.""" + + schema_version: int + staging_partition: str + spans: tuple[RouteSpan, ...] + cleanup_staging_keys: tuple[str, ...] + expected_token_length: int + + +RouteSpanMode = Literal["full", "tail", "sentinel"] + + +def classify_route_span(span: RouteSpan) -> RouteSpanMode: + """Apply Gym's route-linearization decision table using metadata only.""" + expected = span.carry_len + span.generation_len + if span.staged_route_len > 0 and span.staged_route_len == expected: + return "full" + if span.staged_route_len > 0 and 0 < span.generation_len <= span.staged_route_len: + return "tail" + return "sentinel" + + +def _require_exact_keys( + value: dict[str, Any], expected: set[str], *, where: str +) -> None: + actual = set(value) + if actual != expected: + raise ValueError( + f"{where} fields must be exactly {sorted(expected)}, got {sorted(actual)}" + ) + + +def _require_int(value: Any, *, where: str) -> int: + if type(value) is not int: + raise TypeError(f"{where} must be int, got {type(value).__name__}") + return value + + +def _require_nonnegative_int(value: Any, *, where: str) -> int: + parsed = _require_int(value, where=where) + if parsed < 0: + raise ValueError(f"{where} must be non-negative, got {parsed}") + return parsed + + +def _require_string(value: Any, *, where: str) -> str: + if not isinstance(value, str) or not value: + raise TypeError(f"{where} must be a non-empty string") + return value + + +def _validate_plan(plan: RouteAssemblyPlan) -> None: + if plan.schema_version != ROUTE_PLAN_SCHEMA_VERSION: + raise ValueError( + "unsupported route plan schema version " + f"{plan.schema_version}; expected {ROUTE_PLAN_SCHEMA_VERSION}" + ) + _require_string(plan.staging_partition, where="route_plan.staging_partition") + _require_nonnegative_int( + plan.expected_token_length, + where="route_plan.expected_token_length", + ) + cleanup_keys = set(plan.cleanup_staging_keys) + if len(cleanup_keys) != len(plan.cleanup_staging_keys): + raise ValueError("route_plan.cleanup_staging_keys contains duplicates") + for index, key in enumerate(plan.cleanup_staging_keys): + _require_string(key, where=f"route_plan.cleanup_staging_keys[{index}]") + for index, span in enumerate(plan.spans): + _require_string( + span.staging_key, where=f"route_plan.spans[{index}].staging_key" + ) + _require_nonnegative_int( + span.carry_len, where=f"route_plan.spans[{index}].carry_len" + ) + _require_nonnegative_int( + span.generation_len, + where=f"route_plan.spans[{index}].generation_len", + ) + _require_nonnegative_int( + span.staged_route_len, + where=f"route_plan.spans[{index}].staged_route_len", + ) + if span.staging_key not in cleanup_keys: + raise ValueError( + f"route_plan.spans[{index}] key {span.staging_key!r} is outside " + "cleanup_staging_keys" + ) + classify_route_span(span) + if plan.spans: + contribution = sum(span.carry_len + span.generation_len for span in plan.spans) + if contribution != plan.expected_token_length: + raise ValueError( + f"route plan spans contribute {contribution} tokens, expected " + f"{plan.expected_token_length}" + ) + + +def encode_route_plan(plan: RouteAssemblyPlan) -> dict[str, Any]: + """Encode a validated plan into primitive ``KVBatchMeta.tags`` data.""" + _validate_plan(plan) + return { + "schema_version": plan.schema_version, + "staging_partition": plan.staging_partition, + "spans": [ + { + "staging_key": span.staging_key, + "carry_len": span.carry_len, + "generation_len": span.generation_len, + "staged_route_len": span.staged_route_len, + } + for span in plan.spans + ], + "cleanup_staging_keys": list(plan.cleanup_staging_keys), + "expected_token_length": plan.expected_token_length, + } + + +def decode_route_plan(value: Any) -> RouteAssemblyPlan: + """Strictly decode a plan without defaults or compatibility guesses.""" + if not isinstance(value, dict): + raise TypeError(f"route plan must be a dict, got {type(value).__name__}") + _require_exact_keys( + value, + { + "schema_version", + "staging_partition", + "spans", + "cleanup_staging_keys", + "expected_token_length", + }, + where="route_plan", + ) + spans_value = value["spans"] + if not isinstance(spans_value, list): + raise TypeError("route_plan.spans must be a list") + spans: list[RouteSpan] = [] + for index, span_value in enumerate(spans_value): + if not isinstance(span_value, dict): + raise TypeError(f"route_plan.spans[{index}] must be a dict") + _require_exact_keys( + span_value, + {"staging_key", "carry_len", "generation_len", "staged_route_len"}, + where=f"route_plan.spans[{index}]", + ) + spans.append( + RouteSpan( + staging_key=_require_string( + span_value["staging_key"], + where=f"route_plan.spans[{index}].staging_key", + ), + carry_len=_require_nonnegative_int( + span_value["carry_len"], + where=f"route_plan.spans[{index}].carry_len", + ), + generation_len=_require_nonnegative_int( + span_value["generation_len"], + where=f"route_plan.spans[{index}].generation_len", + ), + staged_route_len=_require_nonnegative_int( + span_value["staged_route_len"], + where=f"route_plan.spans[{index}].staged_route_len", + ), + ) + ) + cleanup_value = value["cleanup_staging_keys"] + if not isinstance(cleanup_value, list): + raise TypeError("route_plan.cleanup_staging_keys must be a list") + cleanup_keys = tuple( + _require_string(key, where=f"route_plan.cleanup_staging_keys[{index}]") + for index, key in enumerate(cleanup_value) + ) + plan = RouteAssemblyPlan( + schema_version=_require_int( + value["schema_version"], where="route_plan.schema_version" + ), + staging_partition=_require_string( + value["staging_partition"], where="route_plan.staging_partition" + ), + spans=tuple(spans), + cleanup_staging_keys=cleanup_keys, + expected_token_length=_require_nonnegative_int( + value["expected_token_length"], + where="route_plan.expected_token_length", + ), + ) + _validate_plan(plan) + return plan + + +def encoded_route_plan_size_bytes(plan: RouteAssemblyPlan) -> int: + """Return the compact UTF-8 encoded size used for observability.""" + return len( + json.dumps(encode_route_plan(plan), separators=(",", ":")).encode("utf-8") + ) diff --git a/nemo_rl/models/megatron/router_replay.py b/nemo_rl/models/megatron/router_replay.py index 0dc3dfb8c23..6ffe0b6e4cd 100644 --- a/nemo_rl/models/megatron/router_replay.py +++ b/nemo_rl/models/megatron/router_replay.py @@ -122,6 +122,18 @@ def _global_moe_layer_numbers(model_config: Any) -> list[int]: return [layer_idx + 1 for layer_idx, is_moe in enumerate(pattern) if is_moe] +def router_replay_dimensions(model_config: Any) -> tuple[int, int]: + """Return model-owned ``(num_moe_layers, top_k)`` route dimensions.""" + num_moe_layers = len(_global_moe_layer_numbers(model_config)) + top_k = int(getattr(model_config, "moe_router_topk")) + if num_moe_layers <= 0 or top_k <= 0: + raise ValueError( + "router replay requires positive route dimensions, got " + f"num_moe_layers={num_moe_layers}, top_k={top_k}" + ) + return num_moe_layers, top_k + + def _router_replay_instances_for_model(model: Any) -> list[tuple[Any, int]]: instances: list[tuple[Any, int]] = [] seen: set[int] = set() diff --git a/nemo_rl/models/policy/tq_policy.py b/nemo_rl/models/policy/tq_policy.py index 13f04d7d77c..6dc3458a18d 100644 --- a/nemo_rl/models/policy/tq_policy.py +++ b/nemo_rl/models/policy/tq_policy.py @@ -45,6 +45,8 @@ DP_TRAIN_FIELDS, GLOBAL_FORWARD_PAD_SEQLEN, LP_SEED_FIELDS, + ROUTE_PASSTHROUGH_FLAG, + ROUTE_PLAN_TAG, fields_with_optional_routed_experts, ) from nemo_rl.distributed.batched_data_dict import BatchedDataDict @@ -267,6 +269,38 @@ def _packing_args( return args, None return None, None + def _with_route_fields( + self, + meta: KVBatchMeta, + base_fields: tuple[str, ...], + *, + task_name: str, + want_routes: bool, + ) -> KVBatchMeta: + """Resolve direct versus deferred route storage for one worker request.""" + want = self._router_replay_enabled and want_routes + plan_presence = [ROUTE_PLAN_TAG in tag for tag in (meta.tags or [])] + if want and any(plan_presence) and not all(plan_presence): + raise RuntimeError( + "router replay does not support mixed direct/deferred route " + "storage in one worker fetch" + ) + passthrough = bool(want and plan_presence and all(plan_presence)) + extra_info = dict(meta.extra_info or {}) + if passthrough: + extra_info[ROUTE_PASSTHROUGH_FLAG] = True + else: + extra_info.pop(ROUTE_PASSTHROUGH_FLAG, None) + return replace( + meta, + fields=fields_with_optional_routed_experts( + base_fields, + enabled=want and not passthrough, + ), + task_name=task_name, + extra_info=extra_info, + ) + def _logprob_dispatch( self, meta: KVBatchMeta, @@ -289,13 +323,11 @@ def _logprob_dispatch( """ self._stamp_pad_seqlen(meta) spa, dba = self._packing_args("logprob_mb_tokens") - lp_meta = replace( + lp_meta = self._with_route_fields( meta, - fields=fields_with_optional_routed_experts( - LP_SEED_FIELDS, - enabled=self._router_replay_enabled and include_router_replay, - ), + LP_SEED_FIELDS, task_name=task_name, + want_routes=include_router_replay, ) with timer.time(f"{timer_prefix}/shard_meta") if timer else nullcontext(): metas, _ = shard_meta_for_dp( @@ -397,12 +429,11 @@ def train_from_meta( # default ``DP_TRAIN_FIELDS``) must be in TQ before this call — written # by workers + driver delta-writes. Caller may narrow to drop columns # skipped this step (e.g. ``prev_logprobs`` under force_on_policy_ratio). - train_meta = replace( + train_meta = self._with_route_fields( meta, - fields=fields_with_optional_routed_experts( - train_fields, enabled=self._router_replay_enabled - ), + train_fields, task_name="train", + want_routes=True, ) with timer.time("policy_training/shard_meta") if timer else nullcontext(): dp_metas, _ = shard_meta_for_dp( @@ -521,12 +552,11 @@ def train_microbatches_from_meta( """ self._stamp_pad_seqlen(meta) spa, dba = self._packing_args("train_mb_tokens") - train_meta = replace( + train_meta = self._with_route_fields( meta, - fields=fields_with_optional_routed_experts( - DP_TRAIN_FIELDS, enabled=self._router_replay_enabled - ), + DP_TRAIN_FIELDS, task_name="train", + want_routes=True, ) with timer.time("policy_training/shard_meta") if timer else nullcontext(): dp_metas, _ = shard_meta_for_dp( diff --git a/nemo_rl/models/policy/workers/megatron_policy_worker.py b/nemo_rl/models/policy/workers/megatron_policy_worker.py index 3a9ff57dbfe..2c1e6882442 100644 --- a/nemo_rl/models/policy/workers/megatron_policy_worker.py +++ b/nemo_rl/models/policy/workers/megatron_policy_worker.py @@ -69,7 +69,10 @@ broadcast_obj_from_pp_rank, broadcast_tensors_from_last_stage, ) -from nemo_rl.models.megatron.router_replay import router_replay_enabled +from nemo_rl.models.megatron.router_replay import ( + router_replay_dimensions, + router_replay_enabled, +) from nemo_rl.models.megatron.setup import ( finalize_megatron_setup, handle_model_import, @@ -278,6 +281,10 @@ def _local_coords(self) -> dict[str, int]: "pipeline_parallel": parallel_state.get_pipeline_model_parallel_rank(), } + def _routed_experts_dimensions(self) -> tuple[int, int]: + """Return route dimensions from the initialized Megatron model config.""" + return router_replay_dimensions(self._get_model_config()) + def _get_replica_group(self) -> Optional[Any]: """Replica group = TP × CP × PP siblings within this DP rank. diff --git a/pyrefly.toml b/pyrefly.toml index 877c6a1b80c..71eddf3b3d1 100644 --- a/pyrefly.toml +++ b/pyrefly.toml @@ -146,12 +146,14 @@ project-includes = [ "nemo_rl/evals/answer_parsing.py", "nemo_rl/experience/__init__.py", "nemo_rl/experience/blackbox_finalizer.py", + "nemo_rl/experience/finalizer_actor.py", "nemo_rl/experience/row_dump.py", "nemo_rl/experience/interfaces.py", "nemo_rl/experience/metric_utils.py", "nemo_rl/experience/payload.py", "nemo_rl/experience/rollout_manager.py", "nemo_rl/experience/rollouts.py", + "nemo_rl/experience/route_plan.py", "nemo_rl/modelopt/__init__.py", "nemo_rl/modelopt/models/__init__.py", "nemo_rl/modelopt/models/generation/__init__.py", diff --git a/tests/unit/data_plane/test_blackbox_finalizer.py b/tests/unit/data_plane/test_blackbox_finalizer.py index 11b6c1aee84..72399efa7b4 100644 --- a/tests/unit/data_plane/test_blackbox_finalizer.py +++ b/tests/unit/data_plane/test_blackbox_finalizer.py @@ -27,6 +27,9 @@ from __future__ import annotations +from collections import Counter +from dataclasses import replace + import pytest import torch @@ -41,8 +44,15 @@ from nemo_rl.data_plane.tq_token_sink import ( # noqa: E402 STAGING_FIELDS, TQTokenSink, + TQTokenSource, +) +from nemo_rl.data_plane.schema import ( # noqa: E402 + ROUTE_PASSTHROUGH_FLAG, + ROUTE_PLAN_TAG, ) +from nemo_rl.data_plane.worker_mixin import TQWorkerMixin # noqa: E402 from nemo_rl.experience.blackbox_finalizer import BlackboxFinalizer # noqa: E402 +from nemo_rl.experience.route_plan import decode_route_plan # noqa: E402 pytestmark = pytest.mark.nemo_gym @@ -403,3 +413,155 @@ def test_finalize_group_router_replay_without_routes_fails_loudly( [1.0], fallback_weight_version=9, ) + + +# --------------------------------------------------------------------------- +# Deferred router replay: canonical small rows + strict plans, worker assembly +# --------------------------------------------------------------------------- + +_R3_DEFERRED_PARTITION = "rollout_data_fin_r3_deferred_test" +_R3_DEFERRED_STAGING = "rollout_staging_fin_r3_deferred_test" + + +@pytest.fixture() +def r3_deferred_partitions(tq_client): + from nemo_rl.data_plane.tq_token_sink import ROUTED_EXPERTS_FIELD + + tq_client.register_partition( + partition_id=_R3_DEFERRED_STAGING, + fields=list(STAGING_FIELDS) + [ROUTED_EXPERTS_FIELD], + num_samples=64, + consumer_tasks=["finalize", "prev_lp", "train"], + ) + tq_client.register_partition( + partition_id=_R3_DEFERRED_PARTITION, + fields=[ + "input_ids", + "input_lengths", + "generation_logprobs", + "token_mask", + "sample_mask", + "prompt_ids_for_adv", + "total_reward", + ], + num_samples=64, + consumer_tasks=["train"], + ) + yield + tq_client.clear_samples(sample_ids=None, partition_id=_R3_DEFERRED_STAGING) + tq_client.clear_samples(sample_ids=None, partition_id=_R3_DEFERRED_PARTITION) + + +def _stage_deferred_fixture(tq_client, *, rollout_id: str): + fixture = dict(load_fixture("worked_example")) + fixture["rollout_id"] = rollout_id + records, _, receipt, row = build_fixture_artifacts(fixture) + sink = TQTokenSink(tq_client, staging_partition=_R3_DEFERRED_STAGING) + routes_by_call = {} + for idx, record in enumerate(records): + routes = _routes_for_delta(idx, len(record.token_ids_delta)) + routes_by_call[record.call_id] = routes + staged = record.model_copy(update={"extras": {"routed_experts": routes}}) + assert sink.stage(staged).ok + return receipt.model_dump(), row, routes_by_call + + +class _DeferredRouteWorker(TQWorkerMixin): + def __init__(self, client): + self._dp_client = client + self._route_fallback_counts = Counter() + + def _routed_experts_dimensions(self) -> tuple[int, int]: + return 2, 2 + + +def test_deferred_finalizer_publishes_plans_and_worker_replays_routes( + tq_client, r3_deferred_partitions +): + group_id = "grpr3deferred" + rollout_ids = [f"{group_id}_g0", f"{group_id}_g1"] + receipt, expected, routes_by_call = _stage_deferred_fixture( + tq_client, rollout_id=rollout_ids[0] + ) + finalizer = BlackboxFinalizer( + tq_client, + partition_id=_R3_DEFERRED_PARTITION, + staging_partition=_R3_DEFERRED_STAGING, + pad_token_id=PAD, + mixed_weight_version_policy="allow", + min_valid_fraction_per_group=None, + router_replay_enabled=True, + defer_routed_experts_to_policy=True, + ) + + finalized = finalizer.finalize_group( + group_id, + rollout_ids, + [receipt, None], + [1.0, 0.0], + fallback_weight_version=9, + ) + + assert finalized.meta is not None + assert "routed_experts" not in finalized.meta.fields + assert len(finalized.staging_keys) == len(receipt["manifest"]) + plans = [decode_route_plan(tag[ROUTE_PLAN_TAG]) for tag in finalized.meta.tags] + assert plans[0].expected_token_length == len(expected.token_ids) + assert set(plans[0].cleanup_staging_keys) == set(finalized.staging_keys) + assert not plans[1].spans + # Deferred finalization deliberately retains staging through consumption. + source = TQTokenSource(tq_client, staging_partition=_R3_DEFERRED_STAGING) + assert len(source.fetch_for_finalization(finalized.staging_keys)) == len( + finalized.staging_keys + ) + + worker_meta = replace( + finalized.meta, + extra_info={ROUTE_PASSTHROUGH_FLAG: True}, + task_name="train", + ) + materialized = _DeferredRouteWorker(tq_client)._fetch( + worker_meta, + dp_aligned_seq_len=False, + ) + expected_routes = [ + route for call_id in expected.call_ids for route in routes_by_call[call_id] + ] + valid_len = len(expected.token_ids) + assert materialized["routed_experts"][0, :valid_len].tolist() == expected_routes + assert bool(materialized["routed_experts"][1].eq(-1).all()) + + +@pytest.mark.parametrize("bad_routed_len", [-1, 999]) +def test_deferred_finalizer_rejects_invalid_routed_len( + tq_client, r3_deferred_partitions, bad_routed_len +): + from dataclasses import replace as dataclass_replace + + rollout_id = "bad_route_len_g0" + receipt, _, _ = _stage_deferred_fixture(tq_client, rollout_id=rollout_id) + finalizer = BlackboxFinalizer( + tq_client, + partition_id=_R3_DEFERRED_PARTITION, + staging_partition=_R3_DEFERRED_STAGING, + pad_token_id=PAD, + mixed_weight_version_policy="allow", + min_valid_fraction_per_group=None, + router_replay_enabled=True, + defer_routed_experts_to_policy=True, + ) + fetched = finalizer._source.fetch_for_finalization( + [record["staging_key"] for record in receipt["manifest"]] + ) + fetched[0] = dataclass_replace(fetched[0], routed_len=bad_routed_len) + + class _InjectedSource: + def fetch_for_finalization(self, staging_keys): + del staging_keys + return fetched + + finalizer._source = _InjectedSource() + row = finalizer.finalize_rollout(rollout_id, receipt, reward=0.0) + + assert not row.valid + assert (row.rejection_reason or "").startswith("routed_len_mismatch") diff --git a/tests/unit/data_plane/test_preshard_extras.py b/tests/unit/data_plane/test_preshard_extras.py index 0c5b9e0d62f..8d5f6aca149 100644 --- a/tests/unit/data_plane/test_preshard_extras.py +++ b/tests/unit/data_plane/test_preshard_extras.py @@ -133,6 +133,17 @@ def test_shard_meta_for_dp_preserves_partition_id(): assert all(m.partition_id == "train" for m in metas) +def test_shard_meta_for_dp_preserves_tag_row_identity(): + meta = _meta(8) + meta.tags = [{"row": sample_id} for sample_id in meta.sample_ids] + + metas, _ = shard_meta_for_dp(meta, dp_world=4, batch_size=8) + + for shard in metas: + assert shard.tags is not None + assert [tag["row"] for tag in shard.tags] == shard.sample_ids + + def test_shard_meta_for_dp_unsorted_round_trip(): """unsorted_indices must reconstruct the input order from DP-rank concat.""" n, dp = 8, 4 diff --git a/tests/unit/data_plane/test_tq_policy_routes.py b/tests/unit/data_plane/test_tq_policy_routes.py new file mode 100644 index 00000000000..00cd8a805dc --- /dev/null +++ b/tests/unit/data_plane/test_tq_policy_routes.py @@ -0,0 +1,92 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""TQPolicy direct/deferred route field selection tests.""" + +from __future__ import annotations + +import pytest + +from nemo_rl.data_plane import KVBatchMeta +from nemo_rl.data_plane.schema import ( + ROUTE_PASSTHROUGH_FLAG, + ROUTE_PLAN_TAG, + ROUTED_EXPERTS_FIELD, +) +from nemo_rl.models.policy.tq_policy import TQPolicy + + +def _policy() -> TQPolicy: + policy = object.__new__(TQPolicy) + policy._router_replay_enabled = True + return policy + + +def _meta(tags) -> KVBatchMeta: + return KVBatchMeta( + partition_id="canonical", + task_name="train", + sample_ids=[f"row{i}" for i in range(len(tags))], + fields=["input_ids"], + sequence_lengths=[2] * len(tags), + tags=tags, + ) + + +def test_deferred_prev_lp_and_train_omit_canonical_route_field() -> None: + meta = _meta([{ROUTE_PLAN_TAG: {"plan": 0}}, {ROUTE_PLAN_TAG: {"plan": 1}}]) + + result = _policy()._with_route_fields( + meta, + ("input_ids",), + task_name="prev_lp", + want_routes=True, + ) + + assert ROUTED_EXPERTS_FIELD not in result.fields + assert result.extra_info[ROUTE_PASSTHROUGH_FLAG] is True + + +def test_reference_lp_never_requests_or_materializes_routes() -> None: + meta = _meta([{ROUTE_PLAN_TAG: {"plan": 0}}]) + + result = _policy()._with_route_fields( + meta, + ("input_ids",), + task_name="ref_lp", + want_routes=False, + ) + + assert ROUTED_EXPERTS_FIELD not in result.fields + assert ROUTE_PASSTHROUGH_FLAG not in result.extra_info + + +def test_direct_storage_keeps_canonical_route_field() -> None: + result = _policy()._with_route_fields( + _meta([{"weight_version": 1}]), + ("input_ids",), + task_name="train", + want_routes=True, + ) + assert ROUTED_EXPERTS_FIELD in result.fields + assert ROUTE_PASSTHROUGH_FLAG not in result.extra_info + + +def test_mixed_direct_and_deferred_rows_are_rejected() -> None: + with pytest.raises(RuntimeError, match="mixed direct/deferred"): + _policy()._with_route_fields( + _meta([{ROUTE_PLAN_TAG: {}}, {"weight_version": 1}]), + ("input_ids",), + task_name="train", + want_routes=True, + ) diff --git a/tests/unit/data_plane/test_tq_token_sink.py b/tests/unit/data_plane/test_tq_token_sink.py index d543756a084..b94d187665c 100644 --- a/tests/unit/data_plane/test_tq_token_sink.py +++ b/tests/unit/data_plane/test_tq_token_sink.py @@ -85,6 +85,42 @@ def test_fetch_missing_key_raises_keyerror(tq_client, staging_partition): source.fetch(["ghost_rollout/ghost_call"]) +def test_fetch_for_finalization_is_small_typed_and_identity_preserving( + tq_client, staging_partition +): + class RecordingClient: + def __init__(self, client): + self.client = client + self.select_fields = None + + def get_samples(self, **kwargs): + self.select_fields = list(kwargs["select_fields"]) + return self.client.get_samples(**kwargs) + + sink = TQTokenSink(tq_client, staging_partition=staging_partition) + records, _, _, _ = build_fixture_artifacts(load_fixture("single_call")) + assert sink.stage(records[0]).ok + recording_client = RecordingClient(tq_client) + source = TQTokenSource(recording_client, staging_partition=staging_partition) + + fetched = source.fetch_for_finalization([records[0].staging_key]) + + assert recording_client.select_fields == STAGING_FIELDS + assert "routed_experts" not in recording_client.select_fields + assert len(fetched) == 1 + assert fetched[0].staging_key == records[0].staging_key + assert fetched[0].snapshot.call_id == records[0].call_id + assert fetched[0].routed_len == 0 + + +def test_fetch_for_finalization_rejects_duplicate_request_keys( + tq_client, staging_partition +): + source = TQTokenSource(tq_client, staging_partition=staging_partition) + with pytest.raises(KeyError, match="duplicate keys"): + source.fetch_for_finalization(["r/c", "r/c"]) + + def test_stage_failure_reports_not_raises(staging_partition): class ExplodingClient: def put_samples(self, **kwargs): diff --git a/tests/unit/data_plane/test_worker_route_assembly.py b/tests/unit/data_plane/test_worker_route_assembly.py new file mode 100644 index 00000000000..7e6154d2905 --- /dev/null +++ b/tests/unit/data_plane/test_worker_route_assembly.py @@ -0,0 +1,148 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Policy-worker deferred route assembly tests.""" + +from __future__ import annotations + +from collections import Counter + +import torch + +from nemo_rl.data_plane import KVBatchMeta +from nemo_rl.data_plane.schema import ( + ROUTE_PASSTHROUGH_FLAG, + ROUTE_PLAN_TAG, + ROUTED_EXPERTS_FIELD, +) +from nemo_rl.data_plane.worker_mixin import TQWorkerMixin +from nemo_rl.distributed.batched_data_dict import BatchedDataDict +from nemo_rl.experience.route_plan import ( + ROUTE_PLAN_SCHEMA_VERSION, + RouteAssemblyPlan, + RouteSpan, + encode_route_plan, +) + + +class _Rows(dict): + def __init__(self, routed: list[torch.Tensor]) -> None: + super().__init__({ROUTED_EXPERTS_FIELD: routed}) + self.batch_size = (len(routed),) + + +class _RouteClient: + def __init__(self, fragments: dict[str, torch.Tensor]) -> None: + self.fragments = fragments + self.calls: list[list[str]] = [] + + def get_samples(self, *, sample_ids, partition_id, select_fields): + assert partition_id == "staging" + assert select_fields == [ROUTED_EXPERTS_FIELD] + self.calls.append(list(sample_ids)) + return _Rows([self.fragments[key] for key in sample_ids]) + + +class _Worker(TQWorkerMixin): + def __init__(self, client: _RouteClient) -> None: + self._dp_client = client + self._route_fallback_counts = Counter() + + def _routed_experts_dimensions(self) -> tuple[int, int]: + return 1, 2 + + +def _plan( + spans: tuple[RouteSpan, ...], *, expected: int, cleanup: tuple[str, ...] +) -> dict: + return encode_route_plan( + RouteAssemblyPlan( + schema_version=ROUTE_PLAN_SCHEMA_VERSION, + staging_partition="staging", + spans=spans, + cleanup_staging_keys=cleanup, + expected_token_length=expected, + ) + ) + + +def _meta(plans: list[dict], lengths: list[int]) -> tuple[KVBatchMeta, BatchedDataDict]: + meta = KVBatchMeta( + partition_id="canonical", + task_name="train", + sample_ids=[f"row{i}" for i in range(len(plans))], + fields=["input_ids", "input_lengths"], + sequence_lengths=lengths, + tags=[{ROUTE_PLAN_TAG: plan} for plan in plans], + extra_info={ROUTE_PASSTHROUGH_FLAG: True}, + ) + data = BatchedDataDict( + { + "input_ids": torch.zeros((len(plans), max(lengths) + 1), dtype=torch.long), + "input_lengths": torch.tensor(lengths, dtype=torch.long), + } + ) + return meta, data + + +def test_worker_coalesces_keys_and_replays_full_tail_and_placeholder() -> None: + fragments = { + "r/c0": torch.tensor([[[10, 11]], [[12, 13]]], dtype=torch.int16), + "r/c1": torch.tensor([[[22, 23]]], dtype=torch.int16), + } + client = _RouteClient(fragments) + worker = _Worker(client) + plans = [ + _plan( + ( + RouteSpan("r/c0", 0, 2, 2), + RouteSpan("r/c1", 1, 1, 1), + ), + expected=4, + cleanup=("r/c0", "r/c1", "r/off_chain"), + ), + _plan((), expected=1, cleanup=("r/rejected",)), + ] + meta, data = _meta(plans, [4, 1]) + + result = worker._maybe_assemble_routed_experts(meta, data) + + assert client.calls == [["r/c0", "r/c1"]] + routed = result[ROUTED_EXPERTS_FIELD] + assert routed.dtype == torch.int16 + assert routed.shape == (2, 5, 1, 2) + assert routed[0, :4, 0].tolist() == [ + [10, 11], + [12, 13], + [-1, -1], + [22, 23], + ] + assert bool(routed[0, 4].eq(-1).all()) + assert bool(routed[1].eq(-1).all()) + assert not worker._route_fallback_counts + + +def test_wrong_model_shape_falls_back_for_entire_rollout() -> None: + client = _RouteClient({"r/c0": torch.tensor([[[10]], [[11]]], dtype=torch.int16)}) + worker = _Worker(client) + plan = _plan( + (RouteSpan("r/c0", 0, 2, 2),), + expected=2, + cleanup=("r/c0",), + ) + meta, data = _meta([plan], [2]) + + routed = worker._maybe_assemble_routed_experts(meta, data)[ROUTED_EXPERTS_FIELD] + + assert bool(routed.eq(-1).all()) + assert worker._route_fallback_counts == Counter({"fragment_model_shape": 1}) diff --git a/tests/unit/experience/test_finalizer_actor.py b/tests/unit/experience/test_finalizer_actor.py new file mode 100644 index 00000000000..cf03c09c725 --- /dev/null +++ b/tests/unit/experience/test_finalizer_actor.py @@ -0,0 +1,79 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Metadata-only finalizer actor boundary tests.""" + +from __future__ import annotations + +import pytest +import torch + +from nemo_rl.data_plane import KVBatchMeta +from nemo_rl.experience.blackbox_finalizer import FinalizedGroup +from nemo_rl.experience.finalizer_actor import ( + FinalizationRequest, + assert_metadata_only, +) + + +def _request() -> FinalizationRequest: + return FinalizationRequest( + group_id="group", + rollout_ids=("group_g0",), + receipts=( + { + "rollout_id": "group_g0", + "manifest": [ + { + "call_id": "call", + "staging_key": "group_g0/call", + "delta_len": 2, + } + ], + }, + ), + rewards=(1.0,), + fallback_weight_version=4, + ) + + +def test_finalizer_request_and_result_are_metadata_only() -> None: + assert_metadata_only(_request()) + result = FinalizedGroup( + meta=KVBatchMeta( + partition_id="canonical", + task_name="train", + sample_ids=["group_g0"], + fields=["input_ids"], + sequence_lengths=[3], + tags=[{"weight_version": 4}], + ), + group_min_wv=4, + group_max_wv=4, + staging_keys=["group_g0/call"], + metrics={"finalize/total_ms": 1.0}, + ) + assert_metadata_only(result) + + +@pytest.mark.parametrize( + "payload", + [ + torch.ones(2), + {"input_ids": [1, 2]}, + {"routed_experts": [[[[1, 2]]]]}, + ], +) +def test_metadata_guard_rejects_tensor_and_heavy_row_payloads(payload) -> None: + with pytest.raises(TypeError): + assert_metadata_only(payload) diff --git a/tests/unit/experience/test_route_plan.py b/tests/unit/experience/test_route_plan.py new file mode 100644 index 00000000000..1b3e3fc048d --- /dev/null +++ b/tests/unit/experience/test_route_plan.py @@ -0,0 +1,102 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Strict deferred-route plan contract tests.""" + +from __future__ import annotations + +import pytest + +from nemo_rl.experience.route_plan import ( + ROUTE_PLAN_SCHEMA_VERSION, + RouteAssemblyPlan, + RouteSpan, + classify_route_span, + decode_route_plan, + encode_route_plan, +) + + +def _plan() -> RouteAssemblyPlan: + return RouteAssemblyPlan( + schema_version=ROUTE_PLAN_SCHEMA_VERSION, + staging_partition="staging", + spans=( + RouteSpan("r/c0", carry_len=2, generation_len=2, staged_route_len=4), + RouteSpan("r/c1", carry_len=3, generation_len=1, staged_route_len=1), + RouteSpan("r/c2", carry_len=0, generation_len=0, staged_route_len=0), + RouteSpan("r/c3", carry_len=1, generation_len=1, staged_route_len=0), + ), + cleanup_staging_keys=("r/c0", "r/c1", "r/c2", "r/c3", "r/off_chain"), + expected_token_length=10, + ) + + +def test_route_plan_round_trip_is_strict_and_lossless() -> None: + plan = _plan() + assert decode_route_plan(encode_route_plan(plan)) == plan + + +def test_route_span_classification_uses_shared_length_table() -> None: + spans = _plan().spans + assert [classify_route_span(span) for span in spans] == [ + "full", + "tail", + "sentinel", + "sentinel", + ] + + +@pytest.mark.parametrize( + ("mutation", "match"), + [ + ({"schema_version": 999}, "unsupported route plan schema"), + ({"expected_token_length": -1}, "must be non-negative"), + ({"expected_token_length": 9}, "spans contribute 10 tokens"), + ], +) +def test_route_plan_rejects_invalid_top_level_values(mutation, match) -> None: + encoded = encode_route_plan(_plan()) + encoded.update(mutation) + with pytest.raises(ValueError, match=match): + decode_route_plan(encoded) + + +def test_route_plan_rejects_unknown_or_missing_fields() -> None: + encoded = encode_route_plan(_plan()) + encoded["compat_guess"] = True + with pytest.raises(ValueError, match="fields must be exactly"): + decode_route_plan(encoded) + + del encoded["compat_guess"] + del encoded["spans"][0]["staged_route_len"] + with pytest.raises(ValueError, match="fields must be exactly"): + decode_route_plan(encoded) + + +def test_route_plan_rejects_keys_outside_full_cleanup_manifest() -> None: + encoded = encode_route_plan(_plan()) + encoded["cleanup_staging_keys"].remove("r/c1") + with pytest.raises(ValueError, match="outside cleanup_staging_keys"): + decode_route_plan(encoded) + + +def test_placeholder_plan_can_carry_length_without_route_reads() -> None: + placeholder = RouteAssemblyPlan( + schema_version=ROUTE_PLAN_SCHEMA_VERSION, + staging_partition="staging", + spans=(), + cleanup_staging_keys=("r/rejected",), + expected_token_length=7, + ) + assert decode_route_plan(encode_route_plan(placeholder)) == placeholder diff --git a/tests/unit/single_controller/test_finalizer_lifecycle.py b/tests/unit/single_controller/test_finalizer_lifecycle.py new file mode 100644 index 00000000000..a8f67220914 --- /dev/null +++ b/tests/unit/single_controller/test_finalizer_lifecycle.py @@ -0,0 +1,246 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Lifecycle tests for actor-pool finalization and deferred-route ownership.""" + +from __future__ import annotations + +import asyncio +from types import SimpleNamespace +from typing import Any +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from nemo_rl.algorithms.single_controller import SingleControllerActor +from nemo_rl.data_plane import KVBatchMeta +from nemo_rl.data_plane.schema import ROUTE_PLAN_TAG +from nemo_rl.experience.blackbox_finalizer import FinalizedGroup +from nemo_rl.experience.finalizer_actor import FinalizationRequest +from nemo_rl.experience.route_plan import ( + ROUTE_PLAN_SCHEMA_VERSION, + RouteAssemblyPlan, + RouteSpan, + encode_route_plan, +) + + +class _RemoteFinalize: + def __init__( + self, + *, + result: FinalizedGroup | None = None, + error: BaseException | None = None, + ) -> None: + self._result = result + self._error = error + self.calls: list[FinalizationRequest] = [] + + def remote(self, request: FinalizationRequest) -> Any: + self.calls.append(request) + + async def _result(): + if self._error is not None: + raise self._error + assert self._result is not None + return self._result + + return _result() + + +class _DataPlaneClient: + def __init__(self) -> None: + self.clear_calls: list[dict[str, Any]] = [] + + async def clear_samples(self, *, sample_ids: list[str], partition_id: str) -> None: + self.clear_calls.append( + {"sample_ids": list(sample_ids), "partition_id": partition_id} + ) + + +def _request() -> FinalizationRequest: + return FinalizationRequest( + group_id="group", + rollout_ids=("group_g0",), + receipts=( + { + "manifest": [ + {"staging_key": "group_g0/call"}, + {"staging_key": "group_g0/call"}, + ] + }, + ), + rewards=(1.0,), + fallback_weight_version=3, + ) + + +def _controller(actor: object) -> Any: + controller_cls = SingleControllerActor.__ray_metadata__.modified_class + ctrl = object.__new__(controller_cls) + ctrl._available_finalizers = asyncio.Queue() + ctrl._available_finalizers.put_nowait(actor) + ctrl._active_finalizers = 0 + ctrl._finalizer_waiters = 0 + ctrl._finalizer_unknown_outcomes = 0 + ctrl._finalizer_metrics_by_group = {} + ctrl._buffer = MagicMock() + ctrl._buffer.commit_finalized = AsyncMock() + ctrl._dp_client = _DataPlaneClient() + ctrl._partition_id = "canonical" + ctrl._master_config = SimpleNamespace( + token_capture=SimpleNamespace(staging_partition="staging") + ) + return ctrl + + +def test_successful_actor_finalization_returns_actor_and_transfers_ownership() -> None: + meta = KVBatchMeta( + partition_id="canonical", + task_name="train", + sample_ids=["group_g0"], + fields=["input_ids"], + sequence_lengths=[3], + tags=[{"weight_version": 3}], + ) + result = FinalizedGroup( + meta=meta, + group_min_wv=3, + group_max_wv=3, + staging_keys=["group_g0/call"], + metrics={"finalize/group_ms": 1.0}, + ) + finalize = _RemoteFinalize(result=result) + actor = SimpleNamespace(finalize=finalize) + ctrl = _controller(actor) + request = _request() + + asyncio.run(ctrl._finalize_with_actor(request)) + + assert finalize.calls == [request] + assert ctrl._available_finalizers.get_nowait() is actor + assert ctrl._active_finalizers == 0 + assert ctrl._finalizer_unknown_outcomes == 0 + ctrl._buffer.commit_finalized.assert_awaited_once_with( + "group", + meta, + 3, + 3, + staging_keys=["group_g0/call"], + ) + assert ctrl._finalizer_metrics_by_group["group"]["finalize/group_ms"] == 1.0 + + +def test_actor_rpc_failure_is_fatal_and_does_not_retry_or_requeue_actor() -> None: + finalize = _RemoteFinalize(error=RuntimeError("actor died after submission")) + actor = SimpleNamespace(finalize=finalize) + ctrl = _controller(actor) + request = _request() + + with pytest.raises(RuntimeError, match="actor died after submission"): + asyncio.run(ctrl._finalize_with_actor(request)) + + assert finalize.calls == [request] + assert ctrl._available_finalizers.empty() + assert ctrl._active_finalizers == 0 + assert ctrl._finalizer_unknown_outcomes == 1 + ctrl._buffer.commit_finalized.assert_not_awaited() + ctrl._buffer.abort.assert_not_called() + assert ctrl._dp_client.clear_calls == [] + + +def test_missing_actor_metadata_cleans_known_canonical_and_staging_ownership() -> None: + result = FinalizedGroup( + meta=None, + group_min_wv=3, + group_max_wv=3, + staging_keys=["group_g0/call"], + metrics={}, + ) + actor = SimpleNamespace(finalize=_RemoteFinalize(result=result)) + ctrl = _controller(actor) + + with pytest.raises(RuntimeError, match="no metadata for non-dropped group"): + asyncio.run(ctrl._finalize_with_actor(_request())) + + assert ctrl._dp_client.clear_calls == [ + {"sample_ids": ["group_g0"], "partition_id": "canonical"}, + {"sample_ids": ["group_g0/call"], "partition_id": "staging"}, + ] + ctrl._buffer.abort.assert_called_once_with("group") + assert ctrl._available_finalizers.get_nowait() is actor + + +def test_dropped_actor_group_cleans_known_canonical_and_staging_ownership() -> None: + result = FinalizedGroup( + meta=None, + group_min_wv=3, + group_max_wv=3, + staging_keys=["group_g0/call"], + metrics={}, + dropped=True, + ) + actor = SimpleNamespace(finalize=_RemoteFinalize(result=result)) + ctrl = _controller(actor) + + with pytest.raises(RuntimeError, match="group group dropped"): + asyncio.run(ctrl._finalize_with_actor(_request())) + + assert ctrl._dp_client.clear_calls == [ + {"sample_ids": ["group_g0"], "partition_id": "canonical"}, + {"sample_ids": ["group_g0/call"], "partition_id": "staging"}, + ] + ctrl._buffer.abort.assert_called_once_with("group") + assert ctrl._available_finalizers.get_nowait() is actor + + +def test_post_train_cleanup_clears_canonical_rows_and_route_plan_staging_keys() -> None: + plan = RouteAssemblyPlan( + schema_version=ROUTE_PLAN_SCHEMA_VERSION, + staging_partition="staging", + spans=( + RouteSpan( + staging_key="group_g0/call", + carry_len=2, + generation_len=1, + staged_route_len=3, + ), + ), + cleanup_staging_keys=("group_g0/call", "group_g0/fork"), + expected_token_length=3, + ) + meta = KVBatchMeta( + partition_id="canonical", + task_name="train", + sample_ids=["group_g0", "group_g1"], + fields=["input_ids"], + tags=[ + {ROUTE_PLAN_TAG: encode_route_plan(plan)}, + {ROUTE_PLAN_TAG: encode_route_plan(plan)}, + ], + ) + ctrl = _controller(SimpleNamespace()) + + asyncio.run(ctrl._cleanup_consumed_metas([meta])) + + assert ctrl._dp_client.clear_calls == [ + { + "sample_ids": ["group_g0", "group_g1"], + "partition_id": "canonical", + }, + { + "sample_ids": ["group_g0/call", "group_g0/fork"], + "partition_id": "staging", + }, + ] diff --git a/tests/unit/single_controller/test_rollout_pump.py b/tests/unit/single_controller/test_rollout_pump.py index 19c6d2a98a7..ff78ce1caea 100644 --- a/tests/unit/single_controller/test_rollout_pump.py +++ b/tests/unit/single_controller/test_rollout_pump.py @@ -107,6 +107,7 @@ async def generate_and_push( grpo=GRPOConfig.model_construct(max_num_epochs=1) ) ctrl._rollout_manager = _RecordingRolloutManager(buffer) + ctrl._finalizer_actors = [] # The sampler owns admission + target_step stamping (the dispatch counter # lives on the sampler, not the actor). ctrl._sampler = make_sampler(buffer) @@ -221,6 +222,7 @@ async def _main() -> None: grpo=GRPOConfig.model_construct(max_num_epochs=1) ) ctrl._rollout_manager = manager + ctrl._finalizer_actors = [] # Over-sampled windowed policy: admit never gates (buffer unused here). ctrl._sampler = WindowedSampler(None, max_staleness_versions=1) ctrl._dataloader = [ @@ -307,6 +309,7 @@ async def _main() -> None: grpo=GRPOConfig.model_construct(max_num_epochs=1) ) ctrl._rollout_manager = _NeverCalledRolloutManager() + ctrl._finalizer_actors = [] # Over-sampled windowed policy: admit never gates (buffer unused here). ctrl._sampler = WindowedSampler(None, max_staleness_versions=1) ctrl._dataloader = [ @@ -334,6 +337,81 @@ async def _main() -> None: asyncio.run(_main()) +def test_actor_path_releases_generation_permit_before_finalization() -> None: + class _SplitRolloutManager: + def __init__(self) -> None: + self.generated = 0 + self.two_generated = asyncio.Event() + + async def generate_for_finalization( + self, + prompt: Any, + *, + target_step: int | None = None, + inflight_registry: dict[str, tuple[asyncio.Task[None], int]] | None = None, + ) -> Any: + del prompt, target_step, inflight_registry + self.generated += 1 + if self.generated == 2: + self.two_generated.set() + return SimpleNamespace(group_id=f"g{self.generated}") + + async def _main() -> None: + manager = _SplitRolloutManager() + release_finalizers = asyncio.Event() + finalizers_started = 0 + + async def _delayed_finalize(request: Any) -> None: + nonlocal finalizers_started + del request + finalizers_started += 1 + await release_finalizers.wait() + + controller_cls = SingleControllerActor.__ray_metadata__.modified_class + ctrl = object.__new__(controller_cls) + ctrl._async_cfg = SimpleNamespace(max_inflight_prompts=1, diagnostics=False) + ctrl._master_config = SimpleNamespace( + grpo=GRPOConfig.model_construct(max_num_epochs=1) + ) + ctrl._rollout_manager = manager + ctrl._finalizer_actors = [object()] + ctrl._finalize_with_actor = _delayed_finalize + ctrl._sampler = WindowedSampler(None, max_staleness_versions=1) + ctrl._dataloader = [ + BatchedDataDict( + { + "message_log": [ + [{"role": "user", "content": "first"}], + [{"role": "user", "content": "second"}], + ] + } + ) + ] + ctrl._rollout_permitted = asyncio.Event() + ctrl._rollout_permitted.set() + ctrl._rollout_exhausted = asyncio.Event() + ctrl._buffer_capacity = asyncio.Semaphore(2) + ctrl._inflight_rollouts = 0 + ctrl._inflight_by_group_id = {} + ctrl._dispatched_rollouts = set() + ctrl._trainer_version = 0 + ctrl._current_epoch = 0 + + pump = asyncio.create_task(ctrl._rollout_pump()) + await asyncio.wait_for(manager.two_generated.wait(), timeout=1.0) + await asyncio.sleep(0) + assert finalizers_started == 2 + assert ctrl._inflight_rollouts == 0 + release_finalizers.set() + await asyncio.wait_for(pump, timeout=1.0) + + # Successful commits transfer both buffer permits to the train pump. + assert ctrl._buffer_capacity._value == 0 + assert ctrl._rollout_exhausted.is_set() + + asyncio.run(_main()) + + @pytest.mark.vllm def test_rollout_pump_writes_expected_tq_data( multi_step_setup_vllm_async, # noqa: F811 @@ -416,6 +494,7 @@ def test_rollout_pump_writes_expected_tq_data( rollout_manager=rollout_manager, tq_buffer=tq_buffer, partition_id=_PARTITION_ID, + finalizer_actors=[], ) ctrl = SingleControllerActor.remote( master_config=master_config, diff --git a/tests/unit/single_controller/test_single_controller_setup.py b/tests/unit/single_controller/test_single_controller_setup.py index df2864a1f3f..e8da9845ca1 100644 --- a/tests/unit/single_controller/test_single_controller_setup.py +++ b/tests/unit/single_controller/test_single_controller_setup.py @@ -224,6 +224,14 @@ def test_multiple_dataloader_not_supported(self): ("min_groups", "must be >="), ("global_batch_size", "must equal policy.train_global_batch_size"), ("buffer_capacity", "required capacity"), + ( + "deferred_routes_without_capture", + "defer_routed_experts_to_policy requires", + ), + ( + "actor_pool_without_capture", + "finalizer_actor_pool_enabled requires", + ), ], ) def test_invalid_config_fails_before_setup_factories( @@ -239,6 +247,10 @@ def test_invalid_config_fails_before_setup_factories( mc.policy["train_global_batch_size"] = 7 elif invalid_case == "buffer_capacity": mc.async_rl.max_buffered_rollouts = 7 + elif invalid_case == "deferred_routes_without_capture": + mc.token_capture.defer_routed_experts_to_policy = True + elif invalid_case == "actor_pool_without_capture": + mc.token_capture.finalizer_actor_pool_enabled = True else: # pragma: no cover raise AssertionError(f"unknown test case {invalid_case}") diff --git a/tests/unit/single_controller/test_tq_replay_buffer.py b/tests/unit/single_controller/test_tq_replay_buffer.py index 8f61e2bb462..852b5730614 100644 --- a/tests/unit/single_controller/test_tq_replay_buffer.py +++ b/tests/unit/single_controller/test_tq_replay_buffer.py @@ -25,8 +25,15 @@ import nemo_rl.algorithms.async_utils.replay_buffer as _replay_buffer_module from nemo_rl.algorithms.async_utils.replay_buffer import TQReplayBuffer from nemo_rl.data_plane import KVBatchMeta +from nemo_rl.data_plane.schema import ROUTE_PLAN_TAG from nemo_rl.distributed.batched_data_dict import BatchedDataDict from nemo_rl.experience.interfaces import PromptGroupRecord +from nemo_rl.experience.route_plan import ( + ROUTE_PLAN_SCHEMA_VERSION, + RouteAssemblyPlan, + RouteSpan, + encode_route_plan, +) # Each record yields _N_GENS training rows. _N_GENS = 2 @@ -490,6 +497,41 @@ def test_commit_finalized_raises_for_evicted_slot(self): with pytest.raises(ValueError, match="no live slot"): _run(buf.commit_finalized("ghost", meta, group_min_wv=0, group_max_wv=0)) + def test_commit_finalized_verifies_full_plan_manifest_ownership(self): + dp = MultiPartitionFakeDataPlaneClient() + buf = self._make_capture_buffer(dp) + group_id = buf.reserve(weight_version=1, rollout_ids=["r0"]) + plan = encode_route_plan( + RouteAssemblyPlan( + schema_version=ROUTE_PLAN_SCHEMA_VERSION, + staging_partition="rollout_staging", + spans=(RouteSpan("r0/on_chain", 0, 2, 2),), + cleanup_staging_keys=("r0/on_chain", "r0/off_chain"), + expected_token_length=2, + ) + ) + meta = KVBatchMeta( + partition_id="rollout_data", + task_name="train", + sample_ids=["r0"], + fields=["input_ids"], + tags=[{ROUTE_PLAN_TAG: plan}], + ) + + with pytest.raises(ValueError, match="ownership does not match"): + _run( + buf.commit_finalized( + group_id, + meta, + group_min_wv=1, + group_max_wv=1, + staging_keys=["r0/on_chain"], + ) + ) + + assert buf.size() == 1 + assert buf.ready_list == [False] + def test_abort_drops_unready_slot_only(self): dp = MultiPartitionFakeDataPlaneClient() buf = self._make_capture_buffer(dp) @@ -551,6 +593,38 @@ def test_remove_without_staging_partition_skips_staging_clear(self): partitions_cleared = {p for p, _ in dp.clear_calls_by_partition} assert partitions_cleared == {"rollout_data"} + def test_cleanup_failure_retains_buffer_ownership(self): + class FailingStagingCleanupClient(MultiPartitionFakeDataPlaneClient): + def clear_samples(self, sample_ids, partition_id): + if partition_id == "rollout_staging": + raise RuntimeError("injected staging cleanup failure") + return super().clear_samples(sample_ids, partition_id) + + dp = FailingStagingCleanupClient() + buf = self._make_capture_buffer(dp) + group_id = buf.reserve(weight_version=1, rollout_ids=["r0"]) + meta = KVBatchMeta( + partition_id="rollout_data", + task_name="train", + sample_ids=["r0"], + fields=None, + ) + _run( + buf.commit_finalized( + group_id, + meta, + group_min_wv=1, + group_max_wv=1, + staging_keys=["r0/c1"], + ) + ) + + with pytest.raises(RuntimeError, match="retained replay-buffer ownership"): + _run(buf.remove([0], remove_in_dp=True)) + + assert buf.size() == 1 + assert buf._staging_keys_list == [["r0/c1"]] + class TestTQReplayBufferEvictedCommit: def test_commit_on_evicted_slot_writes_nothing(self): diff --git a/tests/unit/single_controller/test_train_pump.py b/tests/unit/single_controller/test_train_pump.py index 8c2eb928c2f..4fa887aea63 100644 --- a/tests/unit/single_controller/test_train_pump.py +++ b/tests/unit/single_controller/test_train_pump.py @@ -342,6 +342,7 @@ def test_train_pump_drives_mcore_training_step( rollout_manager=rollout_manager, tq_buffer=tq_buffer, partition_id=_PARTITION_ID, + finalizer_actors=[], ) ctrl = _RecordingSingleControllerActor.remote( metric_log_handle=log, diff --git a/ultra_launch.sh b/ultra_launch.sh index bf9c5a3d819..70a07d7490d 100755 --- a/ultra_launch.sh +++ b/ultra_launch.sh @@ -698,7 +698,6 @@ VLLM_FLASHINFER_MOE_BACKEND=latency \ NRL_VLLM_ASYNC_TIMEOUT_SECONDS=1800 \ NRL_WG_USE_RAY_REF=1 \ HF_HOME=${HF_HOME:-} \ -HF_TOKEN=${HF_TOKEN:-} \ NRL_USE_FASTOKENS=${NRL_USE_FASTOKENS:-1} \ uv run ${NRL_ENTRYPOINT:-./examples/nemo_gym/run_grpo_nemo_gym.py} \ --config ${CONFIG_PATH} \ @@ -777,6 +776,10 @@ echo "" # ============================================================================= # Record code provenance in the run directory # ============================================================================= +TRAIN_CMD_REDACTED="$( + printf '%s\n' "${TRAIN_CMD}" \ + | sed -E 's/(HF_TOKEN|WANDB_API_KEY|NGC_API_KEY)=[^[:space:]]+/\1=/g' +)" { echo "timestamp: $(date -Iseconds)" echo "branch: $(git -C "${PROJECT_ROOT}" rev-parse --abbrev-ref HEAD 2>/dev/null || echo unknown)" @@ -789,7 +792,7 @@ echo "" echo "container: ${CONTAINER}" echo "config: ${CONFIG_PATH}" echo "genrm_base_url_file: ${GENRM_BASE_URL_FILE}" - echo "command: ${TRAIN_CMD}" + echo "command: ${TRAIN_CMD_REDACTED}" } > "${RUN_DIR}/provenance.txt" # ============================================================================= @@ -800,7 +803,7 @@ if [[ "${DRY_RUN}" == "1" ]]; then echo "DRY_RUN=1 — printing TRAIN_CMD and exiting without submission." echo "" echo "--- TRAIN_CMD ---" - echo "${TRAIN_CMD}" + echo "${TRAIN_CMD_REDACTED}" echo "--- end ---" exit 0 fi From c68a6cd44b498ed7ac5b459bde1c02ea222e1538 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Tue, 18 Aug 2026 13:42:32 -0700 Subject: [PATCH 21/52] refactor(sc): require finalizer actor pool Signed-off-by: Pranav Thombre --- docs/guides/nano-swe-token-capture.md | 1 + .../configs/ultra/nano_swe_teacher_sc.yaml | 5 +- nemo_rl/algorithms/single_controller.py | 5 +- .../single_controller_utils/config.py | 16 +--- .../single_controller_utils/setup.py | 36 ++----- nemo_rl/experience/rollout_manager.py | 96 ------------------- tests/unit/experience/test_rollout_manager.py | 78 +++------------ .../test_single_controller_setup.py | 51 ++++++++-- 8 files changed, 76 insertions(+), 212 deletions(-) diff --git a/docs/guides/nano-swe-token-capture.md b/docs/guides/nano-swe-token-capture.md index 8a13f17ea3d..fc8514c5b1f 100644 --- a/docs/guides/nano-swe-token-capture.md +++ b/docs/guides/nano-swe-token-capture.md @@ -128,6 +128,7 @@ The pieces, by repo: | Worker-side capture + prefix splice | `nemo_rl/models/generation/vllm/vllm_worker_async.py` | | Staging sink/source over TransferQueue | `nemo_rl/data_plane/tq_token_sink.py` | | Receipt → training row | `nemo_rl/experience/blackbox_finalizer.py` | +| Metadata-only finalizer actor pool | `nemo_rl/experience/finalizer_actor.py` | ## Quick start diff --git a/examples/configs/ultra/nano_swe_teacher_sc.yaml b/examples/configs/ultra/nano_swe_teacher_sc.yaml index 7436b591c3d..600eed67015 100644 --- a/examples/configs/ultra/nano_swe_teacher_sc.yaml +++ b/examples/configs/ultra/nano_swe_teacher_sc.yaml @@ -116,8 +116,5 @@ token_capture: capture_dir: null # Keep route bytes out of canonical rows and assemble them on policy workers. defer_routed_experts_to_policy: false - # Independent rollback control for the bounded CPU finalizer actor pool and - # generation/finalization overlap. - finalizer_actor_pool_enabled: false - # Fixed validation pool; actors have no automatic restart/task retry. + # Fixed CPU finalizer pool; actors have no automatic restart/task retry. num_finalizer_workers: 2 diff --git a/nemo_rl/algorithms/single_controller.py b/nemo_rl/algorithms/single_controller.py index bfaecaa2b6c..b356b87cb1e 100644 --- a/nemo_rl/algorithms/single_controller.py +++ b/nemo_rl/algorithms/single_controller.py @@ -483,9 +483,8 @@ async def _rollout_pump(self) -> None: 1. Acquire _buffer_capacity slot (backpressure) 2. Acquire sem (cap concurrent in-flight rollouts) 3. Wait for _rollout_permitted (paused during weight sync) - 4. Call rollout_manager.generate_and_push(prompt) — local async - RolloutManager reserves a slot, runs the rollout, then commits the - group via TQReplayBuffer (→ dp_client.put_samples + mark ready) + 4. Run the rollout, then either commit it directly or submit its + metadata-only request to the finalizer actor pool. 5. Decrement _inflight_rollouts """ sem = asyncio.Semaphore(self._async_cfg.max_inflight_prompts) diff --git a/nemo_rl/algorithms/single_controller_utils/config.py b/nemo_rl/algorithms/single_controller_utils/config.py index ed1ee8f966a..eed8032ce62 100644 --- a/nemo_rl/algorithms/single_controller_utils/config.py +++ b/nemo_rl/algorithms/single_controller_utils/config.py @@ -103,11 +103,7 @@ class TokenCaptureConfig(BaseModel, extra="allow"): # Keep routed_experts out of canonical rows and assemble them on policy # workers from strict staged-fragment plans. defer_routed_experts_to_policy: bool = False - # Move finalization into a bounded CPU Ray actor pool. Disabled is the - # serial rollback path; enabled releases generation capacity before the - # actor RPC and overlaps finalization with the next rollout group. - finalizer_actor_pool_enabled: bool = False - # Fixed pool size for validation; actors are never automatically replaced. + # Fixed CPU finalizer pool size; actors are never automatically replaced. num_finalizer_workers: PositiveInt = 2 @@ -187,15 +183,7 @@ def validate_single_controller_config(master_config: MasterConfig) -> None: "token_capture.enabled=true" ) if ( - token_capture_config.finalizer_actor_pool_enabled - and not token_capture_config.enabled - ): - raise ValueError( - "token_capture.finalizer_actor_pool_enabled requires " - "token_capture.enabled=true" - ) - if ( - token_capture_config.finalizer_actor_pool_enabled + token_capture_config.enabled and token_capture_config.num_finalizer_workers > async_config.max_buffered_rollouts ): diff --git a/nemo_rl/algorithms/single_controller_utils/setup.py b/nemo_rl/algorithms/single_controller_utils/setup.py index 7f55b8d1353..db551f0fd0b 100644 --- a/nemo_rl/algorithms/single_controller_utils/setup.py +++ b/nemo_rl/algorithms/single_controller_utils/setup.py @@ -725,33 +725,16 @@ def _build_generation_then_trainer( token_capture_cfg.staging_partition if token_capture_cfg.enabled else None ), ) - finalizer = None finalizer_actors: list[Any] = [] if token_capture_cfg.enabled: - from nemo_rl.experience.blackbox_finalizer import BlackboxFinalizer - - if token_capture_cfg.finalizer_actor_pool_enabled: - from nemo_rl.experience.finalizer_actor import ( - FinalizerActorConfig, - create_finalizer_actors, - ) + from nemo_rl.experience.finalizer_actor import ( + FinalizerActorConfig, + create_finalizer_actors, + ) - finalizer_actors = create_finalizer_actors( - dp_config, - FinalizerActorConfig( - partition_id=partition_id, - staging_partition=token_capture_cfg.staging_partition, - pad_token_id=pad_id, - mixed_weight_version_policy=token_capture_cfg.mixed_weight_version_policy, - min_valid_fraction_per_group=token_capture_cfg.min_valid_fraction_per_group, - router_replay_enabled=router_replay_enabled(policy_config), - defer_routed_experts_to_policy=token_capture_cfg.defer_routed_experts_to_policy, - ), - num_workers=token_capture_cfg.num_finalizer_workers, - ) - else: - finalizer = BlackboxFinalizer( - dp_client, + finalizer_actors = create_finalizer_actors( + dp_config, + FinalizerActorConfig( partition_id=partition_id, staging_partition=token_capture_cfg.staging_partition, pad_token_id=pad_id, @@ -759,7 +742,9 @@ def _build_generation_then_trainer( min_valid_fraction_per_group=token_capture_cfg.min_valid_fraction_per_group, router_replay_enabled=router_replay_enabled(policy_config), defer_routed_experts_to_policy=token_capture_cfg.defer_routed_experts_to_policy, - ) + ), + num_workers=token_capture_cfg.num_finalizer_workers, + ) rollout_manager = RolloutManager( tokenizer=tokenizer, task_to_env=env_handles, @@ -771,7 +756,6 @@ def _build_generation_then_trainer( use_nemo_gym=use_nemo_gym, mask_env_flagged_samples=should_mask_flagged_samples(master_config.env), tq_buffer=tq_buffer, - finalizer=finalizer, ) # Print setup timing metrics diff --git a/nemo_rl/experience/rollout_manager.py b/nemo_rl/experience/rollout_manager.py index b0e1d46e186..945cca7984b 100644 --- a/nemo_rl/experience/rollout_manager.py +++ b/nemo_rl/experience/rollout_manager.py @@ -754,16 +754,10 @@ def __init__( use_nemo_gym: bool = False, mask_env_flagged_samples: bool = True, tq_buffer: Optional[TQReplayBuffer] = None, - finalizer: Optional[Any] = None, ) -> None: assert num_generations_per_prompt >= 1, ( "num_generations_per_prompt must be >= 1" ) - if finalizer is not None: - assert use_nemo_gym, ( - "token capture (finalizer) is only supported on the NeMo-Gym path" - ) - if not use_nemo_gym: rollout_cls = AsyncRolloutImpl assert policy_generation is not None, ( @@ -789,7 +783,6 @@ def __init__( self._tokenizer = tokenizer self._num_generations_per_prompt = num_generations_per_prompt self._tq_buffer = tq_buffer - self._finalizer = finalizer # The NeMo-Gym env handle doubles as the gate control-plane proxy # (gate_metrics / fail_rollouts) on the capture path. self._env_handles = task_to_env @@ -842,13 +835,6 @@ async def generate_and_push( assert self._tq_buffer is not None, ( "generate_and_push requires tq_buffer to be set at __init__" ) - if self._finalizer is not None: - await self._generate_and_finalize( - input_sample, - target_step=target_step, - inflight_registry=inflight_registry, - ) - return start_version = self._weight_version group_id = self._tq_buffer.reserve( weight_version=start_version, target_step=target_step @@ -883,84 +869,6 @@ async def generate_and_push( ) raise - async def _generate_and_finalize( - self, - input_sample: DatumSpec, - *, - target_step: Optional[int] = None, - inflight_registry: Optional[dict[str, tuple[asyncio.Task[None], int]]] = None, - ) -> None: - """Token-capture dispatch: receipts in, canonical rows via the finalizer. - - Mints the group's rollout ids up front — sample ids and gate-registered - rollout ids are the same strings (``{group_id}_g{i}``) — reserves the - slot with them so cleanup can name what it owns before a receipt - exists, and commits via ``commit_finalized`` with the group's - min/max call weight versions. - """ - start_version = self._weight_version - group_id = str(uuid.uuid4()) - rollout_ids = [ - f"{group_id}_g{i}" for i in range(self._num_generations_per_prompt) - ] - self._tq_buffer.reserve( - weight_version=start_version, - target_step=target_step, - group_id=group_id, - rollout_ids=rollout_ids, - ) - try: - if inflight_registry is not None: - current_task = asyncio.current_task() - assert current_task is not None - inflight_registry[group_id] = (current_task, start_version) - try: - record = await self.run_rollout(input_sample, rollout_ids=rollout_ids) - finally: - if inflight_registry is not None: - inflight_registry.pop(group_id, None) - receipts = [c.env_extras.get("ng_receipt") for c in record.completions] - rewards = [float(c.reward) for c in record.completions] - finalized = await asyncio.to_thread( - self._finalizer.finalize_group, - group_id, - rollout_ids, - receipts, - rewards, - fallback_weight_version=start_version, - ) - record.rollout_metrics.update(finalized.metrics) - if finalized.dropped: - # min_valid_fraction_per_group rejected the group: nothing - # was published, so drop the slot like a failed dispatch. - self._tq_buffer.abort(group_id) - raise RuntimeError( - f"token capture: group {group_id} dropped " - "(min_valid_fraction_per_group)" - ) - await self._tq_buffer.commit_finalized( - group_id, - finalized.meta, - finalized.group_min_wv, - finalized.group_max_wv, - staging_keys=finalized.staging_keys, - ) - except BaseException: - self._tq_buffer.abort(group_id) - # Best-effort gate cleanup: no receipt will ever seal these ids. - nemo_gym_env = self._env_handles.get("nemo_gym") - if nemo_gym_env is not None: - try: - await nemo_gym_env.fail_rollouts.remote( - rollout_ids, reason="dispatch_failed" - ) - except Exception as error: - raise RuntimeError( - "failed to close Gym rollout ownership after dispatch " - f"failure for group {group_id!r}" - ) from error - raise - async def generate_for_finalization( self, input_sample: DatumSpec, @@ -979,10 +887,6 @@ async def generate_for_finalization( assert self._tq_buffer is not None, ( "generate_for_finalization requires tq_buffer to be set at __init__" ) - if self._finalizer is not None: - raise RuntimeError( - "generate_for_finalization cannot be combined with a local finalizer" - ) start_version = self._weight_version group_id = str(uuid.uuid4()) rollout_ids = tuple( diff --git a/tests/unit/experience/test_rollout_manager.py b/tests/unit/experience/test_rollout_manager.py index e80f0d3d223..b65a68ef48a 100644 --- a/tests/unit/experience/test_rollout_manager.py +++ b/tests/unit/experience/test_rollout_manager.py @@ -142,7 +142,6 @@ def _make_manager(buffer: _FakeBuffer, impl: _FakeImpl) -> RolloutManager: mgr._tokenizer = None mgr._num_generations_per_prompt = 1 mgr._tq_buffer = buffer - mgr._finalizer = None mgr._env_handles = {} mgr._weight_version = 0 return mgr @@ -306,7 +305,6 @@ async def _second_run(_sample): second_mgr._tokenizer = None second_mgr._num_generations_per_prompt = 1 second_mgr._tq_buffer = buf - second_mgr._finalizer = None second_mgr._env_handles = {} second_mgr._weight_version = 0 @@ -991,35 +989,10 @@ def _last_assistant_token_ids(msg_log): ) -class _FakeFinalizedGroup: - def __init__(self, *, dropped=False): - self.meta = None if dropped else "meta-sentinel" - self.group_min_wv = 3 - self.group_max_wv = 4 - self.staging_keys = [] - self.metrics = {"finalize/invalid_row_rate": 0.0} - self.dropped = dropped - - -class _FakeFinalizer: - def __init__(self, *, dropped=False): - self.calls: list[tuple] = [] - self._dropped = dropped - - def finalize_group( - self, group_id, rollout_ids, receipts, rewards, *, fallback_weight_version - ): - self.calls.append( - (group_id, rollout_ids, receipts, rewards, fallback_weight_version) - ) - return _FakeFinalizedGroup(dropped=self._dropped) - - class _FakeCaptureBuffer(_FakeBuffer): def __init__(self): super().__init__() self.reserve_rollout_ids: list[list[str] | None] = [] - self.commit_finalized_calls: list[tuple] = [] def reserve( self, *, weight_version, target_step=None, group_id=None, rollout_ids=None @@ -1032,14 +1005,6 @@ def reserve( rollout_ids=rollout_ids, ) - async def commit_finalized( - self, group_id, meta, group_min_wv, group_max_wv, *, staging_keys=None - ): - self.commit_finalized_calls.append( - (group_id, meta, group_min_wv, group_max_wv, staging_keys) - ) - return meta - class _FakeGymEnvHandle: """NemoGym actor stand-in exposing fail_rollouts.remote.""" @@ -1080,12 +1045,11 @@ def _receipt_record(rollout_ids, receipts): ) -def _make_capture_manager(buf, finalizer, *, on_run=None, num_generations=2): +def _make_capture_manager(buf, *, on_run=None, num_generations=2): mgr = object.__new__(RolloutManager) mgr._tokenizer = None mgr._num_generations_per_prompt = num_generations mgr._tq_buffer = buf - mgr._finalizer = finalizer mgr._env_handles = {"nemo_gym": _FakeGymEnvHandle()} mgr._weight_version = 7 @@ -1105,50 +1069,38 @@ async def run_rollout(self, _sample, *, rollout_ids=None): return mgr -class TestGenerateAndFinalizeFlow: - def test_mints_ids_finalizes_and_commits(self): +class TestGenerateForFinalizationFlow: + def test_mints_ids_and_returns_metadata_request(self): buf = _FakeCaptureBuffer() - finalizer = _FakeFinalizer() - mgr = _make_capture_manager(buf, finalizer) + mgr = _make_capture_manager(buf) - _run(mgr.generate_and_push({"prompt": "p"}, target_step=5)) + request = _run(mgr.generate_for_finalization({"prompt": "p"}, target_step=5)) # Rollout ids were minted from the reserved group id and threaded - # end to end: reserve -> impl -> finalizer. + # end to end: reserve -> impl -> metadata-only actor request. (group_id,) = buf._slots expected_ids = [f"{group_id}_g0", f"{group_id}_g1"] assert buf.reserve_rollout_ids == [expected_ids] assert mgr._impl.seen_rollout_ids == expected_ids - (fin_group_id, fin_ids, receipts, rewards, fallback_wv) = finalizer.calls[0] - assert fin_group_id == group_id - assert fin_ids == expected_ids - assert [r["rollout_id"] for r in receipts] == expected_ids - assert rewards == [0.5, 0.5] - assert fallback_wv == 7 - # commit_finalized carried the group's min/max call versions. - assert buf.commit_finalized_calls == [(group_id, "meta-sentinel", 3, 4, [])] - # The legacy commit path was not used and nothing failed at the gate. + assert request.group_id == group_id + assert request.rollout_ids == tuple(expected_ids) + assert [r["rollout_id"] for r in request.receipts] == expected_ids + assert request.rewards == (0.5, 0.5) + assert request.fallback_weight_version == 7 + # Finalization and commit are exclusively owned by the controller's + # actor-pool path; the manager leaves the reservation unready. assert buf.commit_calls == [] assert mgr._env_handles["nemo_gym"].failed == [] - def test_dropped_group_aborts_slot(self): - buf = _FakeCaptureBuffer() - mgr = _make_capture_manager(buf, _FakeFinalizer(dropped=True)) - with pytest.raises(RuntimeError, match="min_valid_fraction"): - _run(mgr.generate_and_push({"prompt": "p"})) - assert buf.commit_finalized_calls == [] - assert len(buf.abort_calls) >= 1 - def test_failed_dispatch_aborts_and_fails_gate_rollouts(self): buf = _FakeCaptureBuffer() async def _boom(_sample): raise RuntimeError("rollout exploded") - mgr = _make_capture_manager(buf, _FakeFinalizer(), on_run=_boom) + mgr = _make_capture_manager(buf, on_run=_boom) with pytest.raises(RuntimeError, match="rollout exploded"): - _run(mgr.generate_and_push({"prompt": "p"})) - assert buf.commit_finalized_calls == [] + _run(mgr.generate_for_finalization({"prompt": "p"})) assert len(buf.abort_calls) == 1 (failed_ids, reason) = mgr._env_handles["nemo_gym"].failed[0] (group_id,) = [buf.abort_calls[0]] diff --git a/tests/unit/single_controller/test_single_controller_setup.py b/tests/unit/single_controller/test_single_controller_setup.py index e8da9845ca1..9baf13f77ad 100644 --- a/tests/unit/single_controller/test_single_controller_setup.py +++ b/tests/unit/single_controller/test_single_controller_setup.py @@ -228,10 +228,6 @@ def test_multiple_dataloader_not_supported(self): "deferred_routes_without_capture", "defer_routed_experts_to_policy requires", ), - ( - "actor_pool_without_capture", - "finalizer_actor_pool_enabled requires", - ), ], ) def test_invalid_config_fails_before_setup_factories( @@ -249,8 +245,6 @@ def test_invalid_config_fails_before_setup_factories( mc.async_rl.max_buffered_rollouts = 7 elif invalid_case == "deferred_routes_without_capture": mc.token_capture.defer_routed_experts_to_policy = True - elif invalid_case == "actor_pool_without_capture": - mc.token_capture.finalizer_actor_pool_enabled = True else: # pragma: no cover raise AssertionError(f"unknown test case {invalid_case}") @@ -296,6 +290,7 @@ def test_returns_actor_args(self, patched_factories): assert actor_args.partition_id == "rollout_data" assert actor_args.tq_buffer._partition_id == "rollout_data" assert actor_args.tq_buffer._require_routed_experts is False + assert actor_args.finalizer_actors == [] def test_router_replay_requires_routes_in_tq_buffer(self, patched_factories): mc = _make_master_config(colocated=True) @@ -443,6 +438,50 @@ def test_nemo_gym_wires_env_handle(self, patched_factories): ) assert actor_args.env_handles["nemo_gym"] is fake_gym_actor + def test_token_capture_always_creates_finalizer_actor_pool(self, patched_factories): + mc = _make_master_config(colocated=True, backend="vllm") + mc.policy["generation"].update( + { + "model_name": "test-model", + "stop_strings": None, + "stop_token_ids": None, + "top_k": None, + "vllm_cfg": {"async_engine": True}, + } + ) + mc.logger = {"log_dir": "/tmp/test-token-capture"} + mc.token_capture.enabled = True + mc.token_capture.num_finalizer_workers = 3 + patched_factories["setup_response_data"].return_value = ( + list(range(8)), + None, + ) + fake_actors = [MagicMock(name=f"finalizer_{index}") for index in range(3)] + + with ( + patch.object(sc_setup_mod, "_should_use_nemo_gym", return_value=True), + patch.object( + sc_setup_mod, "spinup_nemo_gym_actor", return_value=MagicMock() + ), + patch.object(sc_setup_mod, "router_replay_enabled", return_value=False), + patch( + "nemo_rl.experience.finalizer_actor.create_finalizer_actors", + return_value=fake_actors, + ) as mock_create_finalizer_actors, + ): + actor_args, _ = setup_single_controller(mc, MagicMock(pad_token_id=9)) + + (actor_dp_config, actor_config), actor_kwargs = ( + mock_create_finalizer_actors.call_args + ) + assert actor_dp_config == mc.data_plane + assert actor_config.partition_id == "rollout_data" + assert actor_config.staging_partition == mc.token_capture.staging_partition + assert actor_config.pad_token_id == 9 + assert actor_kwargs == {"num_workers": 3} + assert actor_args.finalizer_actors == fake_actors + assert not hasattr(actor_args.rollout_manager, "_finalizer") + def test_setup_timing_populated_for_colocated_vllm(self, patched_factories): """Colocated vLLM records gen+policy+collective+total+worker fields.""" mc = _make_master_config(colocated=True, backend="vllm") From 7a6b3fac9e064c228f98574e3a4d28172fbb1d11 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Tue, 18 Aug 2026 14:07:10 -0700 Subject: [PATCH 22/52] docs(swe): document finalizer pool smoke launch Signed-off-by: Pranav Thombre --- docs/guides/nano-swe-token-capture.md | 161 +++++++++++++++++++------- 1 file changed, 121 insertions(+), 40 deletions(-) diff --git a/docs/guides/nano-swe-token-capture.md b/docs/guides/nano-swe-token-capture.md index fc8514c5b1f..109bb96415f 100644 --- a/docs/guides/nano-swe-token-capture.md +++ b/docs/guides/nano-swe-token-capture.md @@ -15,7 +15,8 @@ only what token capture adds. ## Verified result -Run on 6 GB200 NVL72 nodes: +Pool-only smoke run on 6 GB200 NVL72 nodes (Slurm job `6294776`, +2026-08-18): | | | |---|---| @@ -23,9 +24,11 @@ Run on 6 GB200 NVL72 nodes: | Config | `examples/configs/ultra/nano_swe_teacher_sc.yaml` | | Model | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` | | Shape | train 4 nodes (TP2·PP2·CP4, EP1) / gen 2 nodes (vLLM TP2 → 4 engines) | -| Capture flags | `token_capture.enabled=true`, `NG_TIC_FP_CANONICAL=1` | -| Outcome | 15/15 steps, `token_in_rate ≈ 0.9999`, zero `empty_manifest`, zero `capture_failed`, `valid_seqs = GBS` | -| Launched via | `swe_nano_sc_capture.sh` (batch) / `swe_nano_sc_capture_interactive.sh` (attach + iterate) | +| Smoke sizing | 2 prompts × 4 generations = GBS 8, one training step | +| Finalization | fixed pool of 2 CPU Ray actors; routed-expert assembly deferred to policy workers | +| Outcome | exit 0, step 1/1, `global_valid_seqs=8`, `invalid_row_rate=0`, routed-expert coverage 1.0, zero capture failures | +| Timing | 33m29s wall clock; 765.4s setup on warm shared caches | +| Launched via | `swe_nano_sc_capture.sh` (batch) | ## How it works @@ -80,12 +83,18 @@ The call flow for a single SWE rollout, end to end: manifest of committed calls (call ids, staging keys, digests, weight versions) and a `terminal_call_id` naming the chain that is the training row. -7. **Finalize.** Trainer-side, the `BlackboxFinalizer` fetches the staged - deltas the manifest names from TQ, re-verifies each (digest, lengths, - mask shape, weight-version tags), linearizes the terminal chain into one - exact token row, and publishes it to the training partition — a rejected - rollout becomes a masked placeholder so the GRPO group keeps its shape. - Staged rows are cleared after publish. +7. **Finalize.** The controller constructs a metadata-only + `FinalizationRequest`, releases the rollout concurrency permit, and submits + it to a fixed pool of CPU Ray actors. Each actor owns a connect-only TQ + client and a `BlackboxFinalizer`; it fetches the staged deltas named by the + manifest, re-verifies them (digest, lengths, mask shape, weight-version + tags), linearizes the terminal chain into one exact token row, and publishes + it to the training partition. A rejected rollout becomes a masked + placeholder so the GRPO group keeps its shape. The pool is the only + token-capture finalization path; there is no inline controller finalizer. + In the deferred-route posture below, staged rows remain until policy workers + assemble routes and the training step consumes them; direct mode clears them + immediately after publication. 8. **Train.** Once a global batch of rows is buffered, the SC takes an optimizer step and syncs weights to the engines; the bumped `weight_version` is stamped on subsequent calls so refit boundaries are @@ -111,8 +120,8 @@ gate ingests coords into its lineage index; when the rollout ends it returns a token-free RolloutReceipt (a manifest of call_ids and staging keys) on the /run response ▼ -finalizer (trainer side): fetches the staged deltas by key, verifies -digests, rebuilds the exact training row, publishes it, clears staging +fixed CPU Ray finalizer pool: accepts metadata only, fetches staged deltas +by key, verifies digests, rebuilds the exact row, and publishes it to TQ ``` The heavy bytes (token arrays, logprobs) move exactly once, worker→TQ, @@ -132,30 +141,92 @@ The pieces, by repo: ## Quick start -Batch, one command from a networked shell at the repo root: +### Prepare the checkout + +Run from a networked shell at this repository's root. Before submitting, +change every per-user write setting in `swe_nano.env`: + +| Variable | Required value | +|---|---| +| `CODE_DIR` | Absolute path to **this checkout**; the launcher mounts it into the container | +| `WORKSPACE_DIR` | Writable results, Ray-log, and checkpoint root | +| `HF_HOME` | Writable Hugging Face cache (~60 GB for the model) | +| `PERSISTENT_CACHE` | Writable vLLM, Triton, and Inductor cache | +| `NRL_MEGATRON_CHECKPOINT_DIR` | Writable Megatron conversion cache, normally below `PERSISTENT_CACHE` | +| `SLURM_ACCOUNT` | Slurm account you can charge | + +The container, SWE data, sandbox SIFs, and model name in the shared read-only +block can be reused. Keep `USE_SNAPSHOT=0` to execute the live files in +`CODE_DIR`; set a unique `SC_EXP_NAME` and staging partition for every active +run. Export `HF_TOKEN` if the model is not already cached. + +### Reproduce the one-step, six-node smoke + +The following is the pool-only posture validated by job `6294776`. The batch +wrapper itself supplies `token_capture.enabled=true` and pins Gym rollout +attempts to one: ```bash -DRY_RUN=0 SC_EXP_NAME=my-capture-run NG_TIC_FP_CANONICAL=1 \ - WALLTIME=3:59:00 bash swe_nano_sc_capture.sh +CAPTURE_OVERRIDES=( + grpo.max_num_steps=1 + grpo.num_prompts_per_step=2 + policy.train_global_batch_size=8 + token_capture.num_finalizer_workers=2 + token_capture.defer_routed_experts_to_policy=true + token_capture.staging_partition=rollout_staging_my_capture_smoke + +policy.router_replay.enabled=true + async_rl.sampler.name=windowed + +async_rl.sampler.max_staleness_versions=1 + +env.nemo_gym.model_endpoint_readiness_timeout_seconds=1800 + policy.generation.vllm_cfg.reasoning_parser_plugin=/opt/nemo-rl/nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py +) + +# Resolve and inspect the six-node driver command without submitting. +DRY_RUN=1 SC_EXP_NAME=my-capture-smoke NG_TIC_FP_CANONICAL=1 \ + WALLTIME=1:49:00 bash swe_nano_sc_capture.sh "${CAPTURE_OVERRIDES[@]}" + +# Submit the same command. +DRY_RUN=0 SC_EXP_NAME=my-capture-smoke NG_TIC_FP_CANONICAL=1 \ + WALLTIME=1:49:00 bash swe_nano_sc_capture.sh "${CAPTURE_OVERRIDES[@]}" ``` -Without `DRY_RUN=0` it prints the resolved driver command and exits, which is -worth doing once. Interactive mode — allocate once, keep Ray up, run the -driver by hand, edit, re-run — follows the same attach/run-cmd workflow as -the TransferQueue recipe: +`num_prompts_per_step × num_generations_per_prompt` must equal +`train_global_batch_size`; this config has four generations per prompt, hence +`2 × 4 = 8`. The finalizer pool is mandatory whenever capture is enabled, +and `num_finalizer_workers` must be positive. There is no pool enable/disable +or legacy-inline-finalizer flag. + +The 1h49 allocation is suitable when the model and compilation caches are +warm and automatically selects the `short` QOS in `ultra_launch.sh`. For a +cold cache, or when the short-QOS node quota is unavailable, use +`WALLTIME=3:59:00`; `swe_nano.env` leaves `SLURM_QOS` empty for that case. +`WALLTIME` must be a Slurm time string—a value such as `3h` is invalid. + +For a longer run, change `grpo.max_num_steps` and use the longer walltime. Do +not reuse a staging partition concurrently with another run. + +### Interactive iteration + +The interactive launcher accepts the same override array: ```bash -NG_TIC_FP_CANONICAL=1 bash swe_nano_sc_capture_interactive.sh +SC_EXP_NAME=my-capture-smoke-interactive NG_TIC_FP_CANONICAL=1 \ + WALLTIME=3:59:00 bash swe_nano_sc_capture_interactive.sh \ + "${CAPTURE_OVERRIDES[@]}" ``` -The non-capture baseline remains `swe_nano_sc.sh` / -`swe_nano_sc_interactive.sh`. Batch and interactive build byte-for-byte the -same driver command. +It allocates six nodes, keeps Ray alive, and prints commands equivalent to: + +```bash +bash -attach.sh +source -run-cmd.sh +``` -`WALLTIME` must be a Slurm time string (`3:59:00`). A bare `3h` fails with -`sbatch: error: Invalid --time specification` printed at the very *end* of -the launch output — easy to miss. Budget ~1 h 40 of setup (venv rebuild + -checkpoint load) before the first step. +Attach to the head node and source the generated run command after Ray is up; +edit and re-source it to iterate in the same allocation. The non-capture +baseline remains `swe_nano_sc.sh` / `swe_nano_sc_interactive.sh`. If batch and +interactive runs overlap, first change the staging-partition value in +`CAPTURE_OVERRIDES` so each live run has its own partition. ## What the capture launchers add, and why @@ -164,13 +235,15 @@ its absence broke a run: | Setting | Without it | |---|---| -| `token_capture.enabled=true` | Capture never engages. The `token_capture` block (with `enabled: false` and all defaults documented) lives in `nano_swe_teacher_sc.yaml`; the launcher flips the one flag. | +| `token_capture.enabled=true` | Capture never engages. The launcher flips the config's default. Enabling it always constructs the fixed CPU finalizer actor pool sized by `token_capture.num_finalizer_workers`; there is no inline fallback. | +| `token_capture.defer_routed_experts_to_policy=true` + `+policy.router_replay.enabled=true` | The validated R3 posture keeps routed-expert tensors out of controller RPCs and canonical rows, then reconstructs them on policy workers from staged-fragment plans. Set both together. | +| `async_rl.sampler.name=windowed` + `max_staleness_versions=1` | The smoke does not exercise the intended bounded-staleness sampling policy. | | `NG_TIC_FP_CANONICAL=1` | Token-in silently degrades to ~0: reasoning models echo history with `` blocks stripped, the gate's fingerprint of the served turn never matches, and every call falls back to text mode. With it, `token_in_rate ≈ 0.9999`. | | `NRL_DRIVER_PYTHONPATH=/opt/nemo-rl/3rdparty/Gym-workspace/Gym` | Driver `ModuleNotFoundError: nemo_gym` — the driver imports the staging record schema, and the baked driver venv has no nemo_gym. | | `NRL_DRIVER_PIP_INSTALL=orjson` | Driver `ModuleNotFoundError: orjson` — Gym's `token_id_capture/__init__` eagerly imports the store. | | `VllmAsyncGenerationWorker` in `NRL_FORCE_REBUILD_VENVS_LIST` | Worker `ModuleNotFoundError: orjson` — venv caching is spec-unaware and silently reuses a non-capture worker venv built by an earlier job. | | capture env set *after* sourcing `swe_nano.env` | `swe_nano.env` exports `NRL_FORCE_REBUILD_VENVS_LIST` unconditionally and clobbers an env-prefix value — which is why these are dedicated launchers rather than an env prefix on `swe_nano_sc.sh`. | -| `CALL_TIMING=0` (optional) | Per-call latency JSONL is on by default in these launchers (`NRL_CALL_TIMING_DIR`/`NG_CALL_TIMING_DIR`); set 0 to disable. All probes are env-gated and dormant without the dir. | +| `CALL_TIMING=0` (optional, batch) | Per-call latency JSONL is on by default in the batch launcher (`NRL_CALL_TIMING_DIR`/`NG_CALL_TIMING_DIR`); set 0 to disable. All probes are env-gated and dormant without the dir. | ## Verifying capture is really engaged @@ -191,9 +264,13 @@ Config echo is not evidence. Check, in order: above). `unattributed_calls` > 0 means the harness is not forwarding `ng_rollout_id` (attribution fix missing from the Gym pin). -2. **No finalize rejections.** `finalize: ... rejected (empty_manifest)` on - every rollout means calls reach the worker without gate context — the run - generates but nothing is trainable. +2. **Finalizer-pool health.** In `step_metrics`, require + `finalize/invalid_row_rate=0` and, for the R3 command above, + `finalize/routed_experts_row_coverage=1`. The one-step smoke reported + `finalize/queue_depth=0` and `finalize/active_actor_count=1`; queue depth can + be nonzero under heavier load. A `finalizer actor RPC failed after + submission` message is fatal because the publication outcome is unknown + and actors are deliberately not retried. 3. **TQ staging traffic**: `PUT_DATA` on the staging partition fires per model call (tens of thousands per run), not just per training batch. @@ -209,14 +286,18 @@ Config echo is not evidence. Check, in order: - **Receipt-mode W&B rollout metrics are not yet comparable to legacy**: `gen_tokens_per_sample` counts the carried prompt tail and `truncation_rate` is constant on the capture arm. -- **Weight-version mixing** across spliced chains at a refit boundary is - tag-checked per call; a per-row guard in the finalizer is a pending - hardening item. -- **Router replay (R3)**: the capture path stages routed experts beside the - token delta (`routed_experts_delta` extras) so the per-token arrays never - transit HTTP — but vLLM 0.20.0's engine currently crashes with - `enable_return_routed_experts=true` on this model; blocked on an engine - fix. +- **Weight-version mixing** across a spliced chain is tag-checked per call. + The recipe defaults to `mixed_weight_version_policy=allow` and stamps the + row with the group's oldest version for staleness accounting; set it to + `reject` to emit a placeholder for a mixed-version rollout. +- **Router replay (R3)** is enabled in the verified pool-only smoke. Routed + experts are staged beside token deltas and reconstructed on policy workers; + keep `defer_routed_experts_to_policy=true` paired with + `policy.router_replay.enabled=true`. +- **Shutdown noise after a completed smoke** can include forced Ray/Gym actor + teardown because the asynchronous rollout pump may have work beyond the + final requested train step. Grade the run from the completed `train step + 1/1`, metrics, and Slurm exit code rather than teardown warnings alone. ## Related From b46db918e31f793e33086684bf06891d2924a93d Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 19 Aug 2026 04:52:57 -0700 Subject: [PATCH 23/52] fix(token-capture): align RL with rebased Gym contracts Signed-off-by: Pranav Thombre --- 3rdparty/Gym-workspace/Gym | 2 +- docs/guides/nano-swe-token-capture.md | 46 +-- .../configs/ultra/nano_swe_teacher_sc.yaml | 12 +- nemo_rl/algorithms/single_controller.py | 6 +- .../single_controller_utils/config.py | 10 +- .../single_controller_utils/setup.py | 20 +- nemo_rl/data_plane/schema.py | 14 + nemo_rl/data_plane/tq_token_sink.py | 281 ++++++++++++++---- nemo_rl/data_plane/worker_mixin.py | 93 +++++- nemo_rl/environments/nemo_gym.py | 115 ++++--- nemo_rl/experience/blackbox_finalizer.py | 247 ++++++++++++--- nemo_rl/experience/finalizer_actor.py | 1 + nemo_rl/experience/rollout_manager.py | 3 +- nemo_rl/experience/route_plan.py | 44 ++- .../generation/vllm/vllm_worker_async.py | 66 +++- .../data_plane/test_blackbox_finalizer.py | 97 ++++-- tests/unit/data_plane/test_tq_token_sink.py | 26 +- .../data_plane/test_worker_route_assembly.py | 67 ++++- .../data_plane/token_capture_test_fixtures.py | 137 +++++++++ .../test_nemo_gym_token_capture.py | 90 ++++++ tests/unit/experience/test_route_plan.py | 26 +- .../test_vllm_token_capture_hosting.py | 29 +- .../test_finalizer_lifecycle.py | 2 + .../test_tq_replay_buffer.py | 2 +- ultra_launch.sh | 2 +- 25 files changed, 1174 insertions(+), 264 deletions(-) create mode 100644 tests/unit/data_plane/token_capture_test_fixtures.py create mode 100644 tests/unit/environments/test_nemo_gym_token_capture.py diff --git a/3rdparty/Gym-workspace/Gym b/3rdparty/Gym-workspace/Gym index 96a3cb0faf4..1b34208461e 160000 --- a/3rdparty/Gym-workspace/Gym +++ b/3rdparty/Gym-workspace/Gym @@ -1 +1 @@ -Subproject commit 96a3cb0faf4bf25fb8b5da074e10839f42f5b0c8 +Subproject commit 1b34208461eca1451477005cd94d9190bfd87275 diff --git a/docs/guides/nano-swe-token-capture.md b/docs/guides/nano-swe-token-capture.md index 109bb96415f..5f4590bb458 100644 --- a/docs/guides/nano-swe-token-capture.md +++ b/docs/guides/nano-swe-token-capture.md @@ -45,12 +45,14 @@ The call flow for a single SWE rollout, end to end: 1. **Dispatch.** The `SingleControllerActor`'s rollout pump pulls a prompt group from the dataset and hands it to the NemoGym environment actor, which POSTs `/run` to the `swe_agents_train` Gym server. The run body - carries a fresh `ng_rollout_id`; the gate **registers** the rollout - (control plane) before any model call happens. + carries a fresh `ng_rollout_id`; RL **registers** the rollout with the gate + (control plane) before dispatch and mounts the returned rollout-local data + capability into the agent sandbox without serializing the secret. 2. **Sandbox + agent.** The SWE harness materializes the SWE-bench instance in a sandbox and starts the OpenHands agent. The rollout id is snapshotted - into the agent's config so every LLM request it makes can be attributed - back to this rollout (`metadata.ng_rollout_id` on the request body). + into the agent's config so every LLM request uses the rollout-prefixed + route and its data-capability header. The capability file is mode `0600` + and is removed when the request finishes. 3. **Agent turn loop.** Each turn, OpenHands POSTs `/v1/chat/completions` with the *full rendered history* to the policy model server, which runs in **gate mode**: @@ -59,11 +61,13 @@ The call flow for a single SWE rollout, end to end: one continues (its *parent*) in the lineage index; - on a match it attaches the parent's **exact cumulative token ids** (`required_prefix_token_ids`) plus a capture context - (`rollout_id`, `call_id`, `parent_call_id`, `prev_len`), and forwards - the request to a vLLM engine (rollout→engine affinity keeps a - rollout's calls on the engine holding its KV cache); + (`rollout_id`, `model_call_id`, `parent_call_id`, `prev_len`), and + forwards the request to a vLLM engine; - no match (a true root, or a rewritten/condensed history) starts a new chain — a fallback, never an error. + Gate state and lineage are file-backed under the shared capture directory, + so registration, consecutive calls, and sealing may land on different + serving workers without losing ancestry. 4. **Generate + stage.** The vLLM worker splices the supplied prefix verbatim, renders only the new tail through the chat template, and generates. Before acknowledging the call it **stages the call's token @@ -80,8 +84,8 @@ The call flow for a single SWE rollout, end to end: 6. **Verify + seal.** When the agent finishes, the harness runs the SWE-bench verifier to score the patch (reward 0/1). The `/run` response returns the reward plus a **token-free `RolloutReceipt`**: the ordered - manifest of committed calls (call ids, staging keys, digests, weight - versions) and a `terminal_call_id` naming the chain that is the + manifest of committed calls (model-call ids, staging keys, digests, weight + versions) and a `terminal_model_call_id` naming the chain that is the training row. 7. **Finalize.** The controller constructs a metadata-only `FinalizationRequest`, releases the rollout concurrency permit, and submits @@ -104,7 +108,7 @@ The token path in that flow, compressed: ``` agent (nv-OpenHands) gate (vllm_model, gate mode) - /run body carries ng_rollout_id ───► registers the rollout, admits calls, + prefixed path + data capability ───► admits registered calls, fingerprints incoming history → resolves the parent call → sends the parent's exact prefix token ids @@ -117,7 +121,7 @@ vLLM worker: splices the prefix verbatim, renders only the new tail, │ coords (≈4 B/token) ▼ gate ingests coords into its lineage index; when the rollout ends it -returns a token-free RolloutReceipt (a manifest of call_ids and staging +returns a token-free RolloutReceipt (a manifest of model_call_ids and staging keys) on the /run response ▼ fixed CPU Ray finalizer pool: accepts metadata only, fetches staged deltas @@ -132,7 +136,7 @@ The pieces, by repo: | Component | Where | |---|---| | Gate mode, prefix serving, lineage | Gym `responses_api_models/vllm_model/app.py` + `nemo_gym/token_id_capture/gate.py` | -| Rollout attribution (`ng_rollout_id`) | Gym capture middleware + `swe_agents` harness forwarding | +| Rollout attribution + capability custody | Gym capture middleware + `swe_agents` sandbox mount/header forwarding | | Wire schema (deltas, coords, receipts) | Gym `nemo_gym/token_id_capture/staging/records.py` | | Worker-side capture + prefix splice | `nemo_rl/models/generation/vllm/vllm_worker_async.py` | | Staging sink/source over TransferQueue | `nemo_rl/data_plane/tq_token_sink.py` | @@ -174,6 +178,7 @@ CAPTURE_OVERRIDES=( token_capture.num_finalizer_workers=2 token_capture.defer_routed_experts_to_policy=true token_capture.staging_partition=rollout_staging_my_capture_smoke + +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 @@ -194,7 +199,9 @@ DRY_RUN=0 SC_EXP_NAME=my-capture-smoke NG_TIC_FP_CANONICAL=1 \ `train_global_batch_size`; this config has four generations per prompt, hence `2 × 4 = 8`. The finalizer pool is mandatory whenever capture is enabled, and `num_finalizer_workers` must be positive. There is no pool enable/disable -or legacy-inline-finalizer flag. +or legacy-inline-finalizer flag. The smoke deliberately runs two Gym policy +model workers; their gate state and lineage are shared, so it does not rely on +single-worker request affinity. The 1h49 allocation is suitable when the model and compilation caches are warm and automatically selects the `short` QOS in `ultra_launch.sh`. For a @@ -255,14 +262,15 @@ Config echo is not evidence. Check, in order: look stalled: ``` - gate_metrics={'registered': 128.0, 'token_in': 7042.0, - 'fallback_no_match': 0.0, 'capture_failed': 0.0, ...} + gate_metrics={'registered': 128.0, 'admitted': 7048.0, + 'token_in': 7042.0, 'text': 6.0, + 'capture_failed': 0.0, ...} ``` - `token_in / (token_in + fallback_*)` should be ≥ 0.99. A rate near 0 with - large `fallback_no_match` means canonical fingerprints are off (see - above). `unattributed_calls` > 0 means the harness is not forwarding - `ng_rollout_id` (attribution fix missing from the Gym pin). + `token_in / admitted` should be ≥ 0.99 after the root calls. A rate near 0 + with a large `text` count means canonical fingerprints are off (see above). + `capability_rejected` or `unattributed_calls` above zero means the harness + is not forwarding the rollout capability/path correctly. 2. **Finalizer-pool health.** In `step_metrics`, require `finalize/invalid_row_rate=0` and, for the R3 command above, diff --git a/examples/configs/ultra/nano_swe_teacher_sc.yaml b/examples/configs/ultra/nano_swe_teacher_sc.yaml index 600eed67015..ab9c5c22ac8 100644 --- a/examples/configs/ultra/nano_swe_teacher_sc.yaml +++ b/examples/configs/ultra/nano_swe_teacher_sc.yaml @@ -100,19 +100,13 @@ token_capture: # TQ does not currently provide an operational staging TTL/reaper. registration_ttl_s: 3600.0 staging_ttl_s: 3600.0 - # Gym LineageIndex capacity (each in-flight rollout's cumulative tokens - # live in the gate process). null = derived from the training config: - # rollouts = 2 x max in-flight, tokens = rollouts x max sequence length. - # Size explicitly for agentic workloads with deep per-rollout call trees. - lineage_max_rollouts: null - lineage_max_tokens: null - # Bearer token for the gate's /ng-control/* routes. null = minted per run. + # Bearer token for Gym's token-capture control routes. null = minted per run. control_auth_token: null # Hard deadline per control-plane call (gate death must surface as failed # dispatches + placeholders, not a silent retry stall). control_timeout_s: 60.0 - # Directory for the Gym base capture layer the gate rides on (stays - # essentially empty on the gate path). null = /gym_token_capture. + # Process-shared gate state, lineage, and capture root. null resolves to + # /gym_token_capture. capture_dir: null # Keep route bytes out of canonical rows and assemble them on policy workers. defer_routed_experts_to_policy: false diff --git a/nemo_rl/algorithms/single_controller.py b/nemo_rl/algorithms/single_controller.py index b356b87cb1e..2e3db2681e2 100644 --- a/nemo_rl/algorithms/single_controller.py +++ b/nemo_rl/algorithms/single_controller.py @@ -974,12 +974,10 @@ async def _log_gate_metrics(self) -> None: return if not counters: return - calls = counters["token_in"] + sum( - v for k, v in counters.items() if k.startswith("fallback_") - ) + calls = counters.get("admitted", 0) gate_metrics: dict[str, float] = {k: float(v) for k, v in counters.items()} if calls: - gate_metrics["token_in_rate"] = counters["token_in"] / calls + gate_metrics["token_in_rate"] = counters.get("token_in", 0) / calls self._logger.log_metrics(gate_metrics, step=self._train_steps, prefix="gate") print(f"gate_metrics={gate_metrics}", flush=True) diff --git a/nemo_rl/algorithms/single_controller_utils/config.py b/nemo_rl/algorithms/single_controller_utils/config.py index eed8032ce62..aeda29e84c8 100644 --- a/nemo_rl/algorithms/single_controller_utils/config.py +++ b/nemo_rl/algorithms/single_controller_utils/config.py @@ -80,15 +80,7 @@ class TokenCaptureConfig(BaseModel, extra="allow"): # Gate-side cleanup backstops. registration_ttl_s: float = 3600.0 staging_ttl_s: float = 3600.0 - # Gym LineageIndex capacity (finding M: it holds each in-flight rollout's - # full cumulative token sequence, and eviction of a live rollout silently - # degrades token-in to fallbacks). None = derived at setup from the - # training config: rollouts ≈ 2 × max in-flight; tokens ≈ rollouts × max - # sequence length. Set explicitly for agentic workloads whose per-rollout - # call trees hold more than one context of tokens. - lineage_max_rollouts: Optional[int] = None - lineage_max_tokens: Optional[int] = None - # Bearer token for the gate's /ng-control/* routes (finding S). None = + # Bearer token for Gym's token-capture control routes. None = # minted per run at setup; set explicitly only for multi-controller # setups that must share one gate. control_auth_token: Optional[str] = None diff --git a/nemo_rl/algorithms/single_controller_utils/setup.py b/nemo_rl/algorithms/single_controller_utils/setup.py index db551f0fd0b..0917f428f9b 100644 --- a/nemo_rl/algorithms/single_controller_utils/setup.py +++ b/nemo_rl/algorithms/single_controller_utils/setup.py @@ -439,11 +439,9 @@ def setup_single_controller( "nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker" ] = PY_EXECUTABLES.VLLM_GYM - # Fill the derived gate-hosting fields (see TokenCaptureConfig): - # a per-run control-plane bearer token, the base capture dir the - # gate rides on, and LineageIndex capacity sized from the training - # config (finding M: eviction of a live rollout must not happen - # under normal operation). + # Fill the derived gate-hosting fields (see TokenCaptureConfig): a + # per-run control-plane bearer token and the process-shared capture + # directory used by every Gym worker. if token_capture_cfg.control_auth_token is None: # Deferred import: only needed on the capture path. import secrets @@ -456,18 +454,6 @@ def setup_single_controller( "gym_token_capture", ) ) - if token_capture_cfg.lineage_max_rollouts is None: - group_size = grpo_config.num_generations_per_prompt - in_flight = ( - master_config.async_rl.max_buffered_rollouts - + master_config.async_rl.max_inflight_prompts - ) * group_size - token_capture_cfg.lineage_max_rollouts = 2 * in_flight - if token_capture_cfg.lineage_max_tokens is None: - token_capture_cfg.lineage_max_tokens = ( - token_capture_cfg.lineage_max_rollouts - * int(master_config.policy["max_total_sequence_length"]) - ) set_seed(grpo_config.seed) diff --git a/nemo_rl/data_plane/schema.py b/nemo_rl/data_plane/schema.py index 0d6a97df01f..68aed992bc5 100644 --- a/nemo_rl/data_plane/schema.py +++ b/nemo_rl/data_plane/schema.py @@ -63,6 +63,8 @@ ROUTED_EXPERTS_FIELD = "routed_experts" ROUTED_LEN_FIELD = "routed_len" +ROUTED_EXPERTS_ENCODING_FIELD = "routed_experts_encoding" +ROUTED_EXTRAS_METADATA_FIELD = "extras_metadata_json" # Deferred route storage. Canonical rows carry one strict encoded route plan # per tag; policy workers omit the absent canonical route column and assemble @@ -87,6 +89,18 @@ "total_reward", SAMPLE_MASK, ROUTED_LEN_FIELD, + ROUTED_EXPERTS_ENCODING_FIELD, + "schema_version", + "digest_version", + "extras_digest_version", + "parent_call_id_present", + "capture_mode", + "prev_len", + "delta_len", + "cum_len", + "weight_version", + "chain_hash_present", + "cumulative_hash_present", } ) diff --git a/nemo_rl/data_plane/tq_token_sink.py b/nemo_rl/data_plane/tq_token_sink.py index 330510d7321..10c0ece0631 100644 --- a/nemo_rl/data_plane/tq_token_sink.py +++ b/nemo_rl/data_plane/tq_token_sink.py @@ -22,19 +22,19 @@ staging keys. Each staged row carries three jagged columns (``token_ids_delta``, -``token_mask_delta``, ``generation_logprobs_delta``) plus scalar ``prev_len`` -and ``weight_version`` columns so a fetched row round-trips to a complete -``StagedCallSnapshot`` (parent pointers are lineage state and are rejoined -from the receipt manifest by the finalizer). Masks/logprobs are float32 on -the wire, matching ``compute_staging_digest``'s float32-bit-pattern scheme, -so the finalizer's digest recomputation over fetched values is byte-exact. +``token_mask_delta``, ``generation_logprobs_delta``), the complete receipt +identity/lineage metadata, and all digest inputs so it round-trips to a +complete ``StagedCallSnapshot``. Masks/logprobs are float32 on the wire, +matching ``compute_staging_digest``'s float32-bit-pattern scheme, so the +finalizer's digest recomputation over fetched values is byte-exact. """ from __future__ import annotations +import json import logging from dataclasses import dataclass -from typing import Any, Optional +from typing import Any import ray import torch @@ -46,17 +46,60 @@ StageResult, ) -from nemo_rl.data_plane.schema import ROUTED_EXPERTS_FIELD, ROUTED_LEN_FIELD +from nemo_rl.data_plane.schema import ( + ROUTED_EXPERTS_ENCODING_FIELD, + ROUTED_EXPERTS_FIELD, + ROUTED_EXTRAS_METADATA_FIELD, + ROUTED_LEN_FIELD, +) STAGING_FIELDS = [ "token_ids_delta", "token_mask_delta", "generation_logprobs_delta", + "schema_version", + "digest_version", + "extras_digest_version", + "rollout_id_utf8", + "model_call_id_utf8", + "parent_call_id_utf8", + "parent_call_id_present", + "capture_mode", "prev_len", + "delta_len", + "cum_len", "weight_version", + "digest_bytes", + "extras_digest_bytes", + "chain_hash_bytes", + "chain_hash_present", + "cumulative_hash_bytes", + "cumulative_hash_present", + ROUTED_EXTRAS_METADATA_FIELD, + ROUTED_EXPERTS_ENCODING_FIELD, ROUTED_LEN_FIELD, ] +_ROUTE_ENCODING_NONE = 0 +_ROUTE_ENCODING_ENVELOPE = 1 +_ROUTE_ENCODING_LIST = 2 +_MODE_TO_CODE = {"text": 0, "token_in": 1} +_CODE_TO_MODE = {code: mode for mode, code in _MODE_TO_CODE.items()} + + +def _bytes_tensor(value: bytes) -> torch.Tensor: + """Encode non-empty bytes as one jagged TQ row.""" + if not value: + raise ValueError("staging byte fields must be non-empty") + return torch.tensor([list(value)], dtype=torch.uint8) + + +def _optional_digest_fields(value: str | None) -> tuple[torch.Tensor, torch.Tensor]: + return ( + _bytes_tensor(bytes.fromhex(value) if value is not None else bytes(32)), + torch.tensor([value is not None], dtype=torch.bool), + ) + @dataclass(frozen=True) class FetchedStagedCall: @@ -101,58 +144,111 @@ def stage(self, record: StagedCallRecord) -> StageResult: [record.token_mask_delta], dtype=torch.float32 ), "generation_logprobs_delta": torch.tensor( - [record.generation_logprobs_delta], dtype=torch.float32 + [record.generation_log_probs_delta], dtype=torch.float32 + ), + "schema_version": torch.tensor( + [record.schema_version], dtype=torch.int64 + ), + "digest_version": torch.tensor( + [record.digest_version], dtype=torch.int64 + ), + "extras_digest_version": torch.tensor( + [record.extras_digest_version], dtype=torch.int64 + ), + "rollout_id_utf8": _bytes_tensor(record.rollout_id.encode("utf-8")), + "model_call_id_utf8": _bytes_tensor( + record.model_call_id.encode("utf-8") + ), + "parent_call_id_utf8": _bytes_tensor( + (record.parent_call_id or "\0").encode("utf-8") + ), + "parent_call_id_present": torch.tensor( + [record.parent_call_id is not None], dtype=torch.bool + ), + "capture_mode": torch.tensor( + [_MODE_TO_CODE[record.mode]], dtype=torch.int64 ), "prev_len": torch.tensor([record.prev_len], dtype=torch.int64), + "delta_len": torch.tensor([record.delta_len], dtype=torch.int64), + "cum_len": torch.tensor([record.cum_len], dtype=torch.int64), "weight_version": torch.tensor( [record.weight_version], dtype=torch.int64 ), + "digest_bytes": _bytes_tensor(bytes.fromhex(record.digest)), + "extras_digest_bytes": _bytes_tensor( + bytes.fromhex(record.extras_digest) + ), } - routed = (record.extras or {}).get("routed_experts") + chain_hash, chain_hash_present = _optional_digest_fields(record.chain_hash) + cumulative_hash, cumulative_hash_present = _optional_digest_fields( + record.cumulative_hash + ) + field_dict.update( + { + "chain_hash_bytes": chain_hash, + "chain_hash_present": chain_hash_present, + "cumulative_hash_bytes": cumulative_hash, + "cumulative_hash_present": cumulative_hash_present, + } + ) + extras_metadata = dict(record.extras) if record.extras is not None else None + routed = ( + extras_metadata.pop("routed_experts", None) + if extras_metadata is not None + else None + ) + field_dict[ROUTED_EXTRAS_METADATA_FIELD] = _bytes_tensor( + json.dumps( + extras_metadata, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + ).encode("utf-8") + ) routed_len = 0 + routed_encoding = _ROUTE_ENCODING_NONE if routed is not None: - # Worker sends full prompt+generation coverage of this call's - # spliced sequence; the delta-aligned slice is [prev_len:]. - # A length mismatch stages the delta WITHOUT extras (replay - # degrades to Megatron's own router for this call) rather - # than poisoning the rollout. delta_len = len(record.token_ids_delta) if isinstance(routed, str): - # The worker ships routes as the nrlre1 base64 envelope - # (#3292). int16 covers every practical expert count - # (<32k) and the -1 sentinel. from nemo_rl.utils.routed_experts_codec import ( decode_routed_experts, ) - experts = decode_routed_experts(routed, torch.int16) + dtype_name = routed.split(":", 3)[1] + dtype = { + "int8": torch.int8, + "int16": torch.int16, + "int32": torch.int32, + }.get(dtype_name) + if dtype is None: + raise ValueError( + f"unsupported routed_experts dtype {dtype_name!r}" + ) + experts = decode_routed_experts(routed, dtype) + routed_encoding = _ROUTE_ENCODING_ENVELOPE else: - # int16 covers every practical expert count (<32k) and - # the -1 sentinel; halves staged bytes vs int32. experts = torch.tensor(routed, dtype=torch.int16) - if experts.shape[0] == record.prev_len + delta_len: - experts = experts[record.prev_len :] - if experts.shape[0] == delta_len: - field_dict[ROUTED_EXPERTS_FIELD] = experts.unsqueeze(0) - routed_len = int(experts.shape[0]) - else: - logging.getLogger(__name__).warning( - "routed_experts length %d does not cover delta %d " - "(prev_len %d) for %s — staging without extras", - experts.shape[0], - delta_len, - record.prev_len, - key, + routed_encoding = _ROUTE_ENCODING_LIST + if experts.dim() != 3 or experts.shape[0] != delta_len: + raise ValueError( + "routed_experts must already be delta-aligned: " + f"got shape {tuple(experts.shape)} for delta_len={delta_len}" ) + field_dict[ROUTED_EXPERTS_FIELD] = experts.unsqueeze(0) + routed_len = int(experts.shape[0]) + field_dict[ROUTED_EXPERTS_ENCODING_FIELD] = torch.tensor( + [routed_encoding], dtype=torch.int64 + ) field_dict[ROUTED_LEN_FIELD] = torch.tensor([routed_len], dtype=torch.int64) fields = TensorDict(field_dict, batch_size=[1]) tags = [ { "rollout_id": record.rollout_id, - "call_id": record.call_id, + "model_call_id": record.model_call_id, "parent_call_id": record.parent_call_id, "prev_len": record.prev_len, - "new_len": record.new_len, + "delta_len": record.delta_len, + "cum_len": record.cum_len, "weight_version": record.weight_version, "digest": record.digest, "schema_version": record.schema_version, @@ -254,8 +350,12 @@ def fetch(self, staging_keys: list[str]) -> list[StagedCallSnapshot]: # recomputation is the byte-exact backstop if that ever breaks. snapshots: list[StagedCallSnapshot] = [] for index, key in enumerate(staging_keys): - _, _, call_id = key.rpartition("/") - snapshots.append(_row_to_snapshot(call_id, _select_row(rows, index))) + snapshot = _row_to_snapshot(_select_row(rows, index), include_routes=True) + if snapshot.staging_key != key: + raise KeyError( + f"staged row identity mismatch: requested {key!r}, got {snapshot.staging_key!r}" + ) + snapshots.append(snapshot) return snapshots def fetch_for_finalization( @@ -287,14 +387,16 @@ def fetch_for_finalization( ) fetched: list[FetchedStagedCall] = [] for index, key in enumerate(staging_keys): - _, separator, call_id = key.rpartition("/") - if not separator or not call_id: - raise KeyError(f"invalid staging key without call id: {key!r}") row = _select_row(rows, index) + snapshot = _row_to_snapshot(row, include_routes=False) + if snapshot.staging_key != key: + raise KeyError( + f"staged row identity mismatch: requested {key!r}, got {snapshot.staging_key!r}" + ) fetched.append( FetchedStagedCall( staging_key=key, - snapshot=_row_to_snapshot(call_id, row), + snapshot=snapshot, routed_len=_row_scalar_int(row, ROUTED_LEN_FIELD), ) ) @@ -315,33 +417,94 @@ def _select_row(rows: TensorDict, index: int) -> dict[str, torch.Tensor]: return row -def _row_to_snapshot(call_id: str, row: Any) -> StagedCallSnapshot: +def _row_to_snapshot(row: Any, *, include_routes: bool) -> StagedCallSnapshot: def _leaf(name: str) -> torch.Tensor: value = row[name] tensor = value[0] if value.dim() > 1 or value.numel() > 1 else value return tensor.reshape(-1) - prev_len = int(_leaf("prev_len")[0].item()) - weight_version: Optional[int] = int(_leaf("weight_version")[0].item()) - extras: Optional[dict[str, Any]] = None + def _text(name: str) -> str: + return bytes(int(value) for value in _leaf(name).tolist()).decode("utf-8") + + def _digest(name: str) -> str: + value = bytes(int(item) for item in _leaf(name).tolist()) + if len(value) != 32: + raise ValueError(f"{name} must contain exactly 32 bytes") + return value.hex() + + def _optional_digest(name: str, present_name: str) -> str | None: + return _digest(name) if bool(_leaf(present_name)[0].item()) else None + + parent_call_id = ( + _text("parent_call_id_utf8") + if bool(_leaf("parent_call_id_present")[0].item()) + else None + ) + mode_code = int(_leaf("capture_mode")[0].item()) try: - routed = row[ROUTED_EXPERTS_FIELD] - except KeyError: - routed = None - if routed is not None: - # [1, delta_len, ...] -> [delta_len, ...]; keep the nested list wire - # shape so the snapshot stays a plain-pydantic record. - experts = routed[0] if routed.dim() > 2 or routed.shape[0] == 1 else routed - extras = {"routed_experts": experts.tolist()} - return StagedCallSnapshot( - call_id=call_id, - prev_len=prev_len, + mode = _CODE_TO_MODE[mode_code] + except KeyError as error: + raise ValueError(f"unknown capture_mode code {mode_code}") from error + extras = json.loads(_text(ROUTED_EXTRAS_METADATA_FIELD)) + routed_encoding = int(_leaf(ROUTED_EXPERTS_ENCODING_FIELD)[0].item()) + if routed_encoding != _ROUTE_ENCODING_NONE and include_routes: + try: + routed = row[ROUTED_EXPERTS_FIELD] + except KeyError as error: + raise KeyError( + "staged row metadata names routed_experts but its field is absent" + ) from error + experts = routed[0] if routed.dim() > 3 or routed.shape[0] == 1 else routed + if extras is None: + extras = {} + if not isinstance(extras, dict): + raise TypeError("staged extras metadata must decode to an object or null") + if routed_encoding == _ROUTE_ENCODING_ENVELOPE: + from nemo_rl.utils.routed_experts_codec import encode_routed_experts + + extras[ROUTED_EXPERTS_FIELD] = encode_routed_experts(experts) + elif routed_encoding == _ROUTE_ENCODING_LIST: + extras[ROUTED_EXPERTS_FIELD] = experts.tolist() + else: + raise ValueError(f"unknown routed_experts_encoding {routed_encoding}") + elif routed_encoding not in ( + _ROUTE_ENCODING_NONE, + _ROUTE_ENCODING_ENVELOPE, + _ROUTE_ENCODING_LIST, + ): + raise ValueError(f"unknown routed_experts_encoding {routed_encoding}") + + values = dict( + schema_version=int(_leaf("schema_version")[0].item()), + digest_version=int(_leaf("digest_version")[0].item()), + extras_digest_version=int(_leaf("extras_digest_version")[0].item()), + rollout_id=_text("rollout_id_utf8"), + model_call_id=_text("model_call_id_utf8"), + parent_call_id=parent_call_id, + mode=mode, + prev_len=int(_leaf("prev_len")[0].item()), + delta_len=int(_leaf("delta_len")[0].item()), + cum_len=int(_leaf("cum_len")[0].item()), + weight_version=int(_leaf("weight_version")[0].item()), + digest=_digest("digest_bytes"), token_ids_delta=[int(t) for t in _leaf("token_ids_delta").tolist()], token_mask_delta=[float(m) for m in _leaf("token_mask_delta").tolist()], - logprobs_delta=[float(p) for p in _leaf("generation_logprobs_delta").tolist()], - weight_version=weight_version, + generation_log_probs_delta=[ + float(p) for p in _leaf("generation_logprobs_delta").tolist() + ], extras=extras, + extras_digest=_digest("extras_digest_bytes"), + chain_hash=_optional_digest("chain_hash_bytes", "chain_hash_present"), + cumulative_hash=_optional_digest( + "cumulative_hash_bytes", "cumulative_hash_present" + ), ) + if include_routes or routed_encoding == _ROUTE_ENCODING_NONE: + return StagedCallSnapshot(**values) + # Metadata-only deferred finalization deliberately leaves the heavy route + # fragment in TQ. The finalizer verifies its digest binding before + # publishing a route assembly plan. + return StagedCallSnapshot.model_construct(**values) def _row_scalar_int(row: Any, field_name: str) -> int: diff --git a/nemo_rl/data_plane/worker_mixin.py b/nemo_rl/data_plane/worker_mixin.py index 6f6231473ce..d3c81fc8b99 100644 --- a/nemo_rl/data_plane/worker_mixin.py +++ b/nemo_rl/data_plane/worker_mixin.py @@ -27,9 +27,11 @@ from __future__ import annotations +import json import logging import time from collections import Counter +from dataclasses import dataclass from typing import TYPE_CHECKING, Any, Literal, Optional import torch @@ -44,7 +46,9 @@ MICRO_BATCH_LENGTHS, ROUTE_PASSTHROUGH_FLAG, ROUTE_PLAN_TAG, + ROUTED_EXPERTS_ENCODING_FIELD, ROUTED_EXPERTS_FIELD, + ROUTED_EXTRAS_METADATA_FIELD, Layout, ) from nemo_rl.distributed.batched_data_dict import BatchedDataDict, SequencePackingArgs @@ -56,6 +60,13 @@ from nemo_rl.data_plane.interfaces import DataPlaneClient +@dataclass(frozen=True) +class _DeferredRouteFragment: + routes: torch.Tensor + encoding: int + extras_metadata_json: torch.Tensor + + def _broadcast_batched_data_dict( data: Optional[BatchedDataDict[Any]], *, @@ -317,25 +328,77 @@ def _fetch_route_fragments( *, keys: list[str], partition_id: str, - ) -> dict[str, torch.Tensor]: + ) -> dict[str, _DeferredRouteFragment]: """Fetch a unique key set in one request and preserve request identity.""" if not keys: return {} rows = self._require_dp_client().get_samples( sample_ids=keys, partition_id=partition_id, - select_fields=[ROUTED_EXPERTS_FIELD], + select_fields=[ + ROUTED_EXPERTS_FIELD, + ROUTED_EXPERTS_ENCODING_FIELD, + ROUTED_EXTRAS_METADATA_FIELD, + ], ) n_rows = int(rows.batch_size[0]) if len(rows.batch_size) else 0 if n_rows != len(keys): raise KeyError(f"requested {len(keys)} route rows, got {n_rows}") - column = rows.get(ROUTED_EXPERTS_FIELD) - return {key: column[index] for index, key in enumerate(keys)} + route_column = rows.get(ROUTED_EXPERTS_FIELD) + encoding_column = rows.get(ROUTED_EXPERTS_ENCODING_FIELD) + metadata_column = rows.get(ROUTED_EXTRAS_METADATA_FIELD) + if route_column is None or encoding_column is None or metadata_column is None: + raise KeyError("deferred route row is missing integrity metadata") + return { + key: _DeferredRouteFragment( + routes=route_column[index], + encoding=int(encoding_column[index].reshape(-1)[0].item()), + extras_metadata_json=metadata_column[index].reshape(-1), + ) + for index, key in enumerate(keys) + } + + @staticmethod + def _verify_route_fragment_integrity( + fragment: _DeferredRouteFragment, + *, + extras_digest_version: int, + expected_extras_digest: str, + ) -> bool: + """Rebuild deferred extras and verify the receipt-bound Gym digest.""" + from nemo_gym.token_id_capture.staging.digest import ( + EXTRAS_DIGEST_VERSION, + compute_extras_digest, + ) + from nemo_rl.utils.routed_experts_codec import encode_routed_experts + + if extras_digest_version != EXTRAS_DIGEST_VERSION: + return False + try: + raw_metadata = bytes( + int(value) for value in fragment.extras_metadata_json.tolist() + ) + decoded = json.loads(raw_metadata.decode("utf-8")) + if decoded is None: + extras: dict[str, Any] = {} + elif isinstance(decoded, dict): + extras = decoded + else: + return False + if fragment.encoding == 1: + extras[ROUTED_EXPERTS_FIELD] = encode_routed_experts(fragment.routes) + elif fragment.encoding == 2: + extras[ROUTED_EXPERTS_FIELD] = fragment.routes.tolist() + else: + return False + return compute_extras_digest(extras) == expected_extras_digest + except (TypeError, ValueError): + return False def _route_fragments_by_row( self, plans: list[Any], - ) -> tuple[list[dict[str, torch.Tensor]], int, float]: + ) -> tuple[list[dict[str, _DeferredRouteFragment]], int, float]: """Use one normal-path batch read; isolate error retries per rollout.""" from nemo_rl.experience.route_plan import decode_route_plan @@ -365,7 +428,7 @@ def _route_fragments_by_row( "deferred route batch fetch failed; isolating by rollout: %s", batch_error, ) - per_row: list[dict[str, torch.Tensor]] = [] + per_row: list[dict[str, _DeferredRouteFragment]] = [] for plan in decoded: row_keys = list( dict.fromkeys( @@ -456,23 +519,31 @@ def _maybe_assemble_routed_experts( if fragment is None: reason = "missing_fragment" break - if fragment.dim() != 3: + if not self._verify_route_fragment_integrity( + fragment, + extras_digest_version=span.extras_digest_version, + expected_extras_digest=span.extras_digest, + ): + reason = "fragment_integrity" + break + routes = fragment.routes + if routes.dim() != 3: reason = "fragment_rank" break - if int(fragment.shape[0]) != span.staged_route_len: + if int(routes.shape[0]) != span.staged_route_len: reason = "fragment_length" break - if tuple(fragment.shape[1:]) != (num_moe_layers, top_k): + if tuple(routes.shape[1:]) != (num_moe_layers, top_k): reason = "fragment_model_shape" break if mode == "full": routed[row_index, position : position + contribution] = ( - fragment.to(torch.int16) + routes.to(torch.int16) ) else: tail_start = position + span.carry_len routed[row_index, tail_start : position + contribution] = ( - fragment[-span.generation_len :].to(torch.int16) + routes[-span.generation_len :].to(torch.int16) ) position += contribution if reason is None and plan.spans and position != canonical_len: diff --git a/nemo_rl/environments/nemo_gym.py b/nemo_rl/environments/nemo_gym.py index d4f6578f7b4..dcfe8fd10ea 100644 --- a/nemo_rl/environments/nemo_gym.py +++ b/nemo_rl/environments/nemo_gym.py @@ -12,6 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. import asyncio +import hashlib import math import os import subprocess @@ -141,6 +142,9 @@ class NemoGymConfig(TypedDict): # model call, so the TQ sample id IS the capture key end to end. _POLICY_SERVER_NAME = "policy_model" _NG_ROLLOUT_ID_BODY_KEY = "_ng_rollout_id" +_NG_CAPTURE_CAPABILITY_BODY_KEY = "_ng_capture_capability" +_TOKEN_CAPTURE_CONTROL_PREFIX = "/training-token-capture/control" +_TOKEN_CAPTURE_CONTROL_ENV = "NEMO_GYM_TOKEN_CAPTURE_CONTROL_TOKEN" def _detect_invalid_tool_call_and_malformed_thinking( @@ -488,6 +492,7 @@ def _spinup(self) -> None: self._server_client = None self._control_headers: Dict[str, str] = {} self._control_timeout_s = 60.0 + self._control_owner_id = f"nemo-rl-{os.getpid()}-{id(self):x}" if self._token_capture_enabled: if self.rollout_max_attempts_to_avoid_lp_nan != 1: raise ValueError( @@ -501,35 +506,29 @@ def _spinup(self) -> None: .setdefault("vllm_model", {}) ) policy_overrides["return_token_id_information"] = False - policy_overrides["token_capture_gate"] = { + capture_dir = os.path.abspath(token_capture["capture_dir"]) + initial_global_config_dict["token_id_capture"] = { "enabled": True, - "registration_ttl_s": token_capture["registration_ttl_s"], - # Finding M: the LineageIndex holds each in-flight rollout's - # cumulative tokens; capacity is sized from the training - # config at setup, never left at the eval-sized defaults. - "lineage_max_rollouts": token_capture["lineage_max_rollouts"], - "lineage_max_tokens": token_capture["lineage_max_tokens"], - # Finding S: control routes are bearer-authed, - # default-required; the token is minted per run at setup. - "control_auth_token": token_capture["control_auth_token"], + "all_agents": True, + "rebuild_response": False, + "dir": capture_dir, + # Both stores are process-shared. Every uvicorn worker builds + # its own handle over these same paths, so token-in ancestry + # remains valid when consecutive calls land on different + # workers. + "lineage_store": ("nemo_gym.token_id_capture.lineage:FileLineageStore"), + "lineage_store_kwargs": {"root": os.path.join(capture_dir, "lineage")}, + "gate": { + "enabled": True, + "state_store_path": os.path.join(capture_dir, "gate-state.json"), + "registration_ttl_s": token_capture["registration_ttl_s"], + "control_auth_token_env": _TOKEN_CAPTURE_CONTROL_ENV, + }, } - # #2124-c1 workaround: the capture middleware mints model_call_id - # (and sets the capture context the gate keys on) only when the - # base token capture is active, which requires a capture dir. - # The base's own capture_tokens no-ops on the gate path (the - # worker strips token fields before responding), so the dir - # stays essentially empty. - initial_global_config_dict["token_id_capture_enabled"] = True - initial_global_config_dict["token_id_capture_dir"] = token_capture[ - "capture_dir" - ] - # Agents apply the /ng-rollout/ correlation prefix for token - # capture only when they opt in (the global enabled switch alone - # gates the infrastructure, not the prefix). In an RL training - # run every agent must correlate, and the SC cannot enumerate - # agent servers configured via config_paths — so flip the - # run-wide opt-in the Gym branch adds for exactly this case. - initial_global_config_dict["token_id_capture_all_agents"] = True + # Gym resolves the credential inside each serving process. Keep + # only the variable name in serialized config and inherit the + # secret through the server process environment. + os.environ[_TOKEN_CAPTURE_CONTROL_ENV] = token_capture["control_auth_token"] self._control_headers = { "Authorization": f"Bearer {token_capture['control_auth_token']}" } @@ -594,10 +593,29 @@ async def _control(self, method: str, path: str, **kwargs: Any) -> dict: ) return await response.json() - async def register_rollouts(self, rollout_ids: list[str]) -> None: - """Create-only registration before dispatch (§ 3.1).""" + def _control_operation_id(self, operation: str, rollout_id: str) -> str: + payload = f"{self._control_owner_id}\0{operation}\0{rollout_id}".encode() + return f"{operation}-{hashlib.sha256(payload).hexdigest()}" + + async def register_rollouts(self, rollout_ids: list[str]) -> dict[str, str]: + """Register each rollout and return its worker-facing data capability.""" + capabilities: dict[str, str] = {} for rollout_id in rollout_ids: - await self._control("PUT", f"/ng-control/rollouts/{rollout_id}") + registration = await self._control( + "PUT", + f"{_TOKEN_CAPTURE_CONTROL_PREFIX}/rollouts/{rollout_id}", + json={ + "owner_id": self._control_owner_id, + "operation_id": self._control_operation_id("register", rollout_id), + }, + ) + capability = registration.get("data_capability") + if not isinstance(capability, str) or not capability: + raise RuntimeError( + f"gate registration for {rollout_id} returned no data capability" + ) + capabilities[rollout_id] = capability + return capabilities async def fail_rollouts(self, rollout_ids: list[str], *, reason: str) -> None: """Best-effort gate cleanup for a cancelled/failed dispatch (§ 7).""" @@ -605,8 +623,12 @@ async def fail_rollouts(self, rollout_ids: list[str], *, reason: str) -> None: try: await self._control( "POST", - f"/ng-control/rollouts/{rollout_id}/fail", - json={"reason": reason}, + f"{_TOKEN_CAPTURE_CONTROL_PREFIX}/rollouts/{rollout_id}/fail", + json={ + "owner_id": self._control_owner_id, + "operation_id": self._control_operation_id("fail", rollout_id), + "reason": reason, + }, ) except (RuntimeError, OSError) as error: # The registration TTL is the backstop when the gate is @@ -614,7 +636,7 @@ async def fail_rollouts(self, rollout_ids: list[str], *, reason: str) -> None: print(f"fail_rollout({rollout_id}) failed: {error}", flush=True) async def gate_metrics(self) -> dict: - return await self._control("GET", "/ng-control/metrics") + return await self._control("GET", f"{_TOKEN_CAPTURE_CONTROL_PREFIX}/metrics") async def run_rollouts( self, @@ -645,7 +667,9 @@ async def run_rollouts( rollout_ids = [ example[_NG_ROLLOUT_ID_BODY_KEY] for example in nemo_gym_examples ] - await self.register_rollouts(rollout_ids) + capabilities = await self.register_rollouts(rollout_ids) + for example, rollout_id in zip(nemo_gym_examples, rollout_ids): + example[_NG_CAPTURE_CAPABILITY_BODY_KEY] = capabilities[rollout_id] timer.start("_run_rollouts_total") nemo_gym_result_iterator = self.rch.run_examples( @@ -737,11 +761,30 @@ async def _postprocess_receipt_mode( f"Hit a non-successful response when querying NeMo Gym for rollouts: {nemo_gym_result}" ) rollout_id = nemo_gym_row[_NG_ROLLOUT_ID_BODY_KEY] + terminal_logical_request_id = nemo_gym_result.get("terminal_logical_request_id") + if not isinstance(terminal_logical_request_id, str) or not ( + terminal_logical_request_id + ): + await self.fail_rollouts( + [rollout_id], reason="missing_terminal_logical_request_id" + ) + return { + "message_log": [], + "input_message_log": [], + "full_result": nemo_gym_result, + "rollout_id": rollout_id, + "receipt": None, + } try: receipt = await self._control( "POST", - f"/ng-control/rollouts/{rollout_id}/seal", - json={"reward": float(nemo_gym_result.get("reward") or 0.0)}, + f"{_TOKEN_CAPTURE_CONTROL_PREFIX}/rollouts/{rollout_id}/seal", + json={ + "owner_id": self._control_owner_id, + "operation_id": self._control_operation_id("seal", rollout_id), + "terminal_logical_request_id": terminal_logical_request_id, + "reward": float(nemo_gym_result.get("reward") or 0.0), + }, ) except (RuntimeError, OSError) as error: # An unsealable rollout finalizes as a placeholder; the gate's diff --git a/nemo_rl/experience/blackbox_finalizer.py b/nemo_rl/experience/blackbox_finalizer.py index f6e0932faee..bd195ada519 100644 --- a/nemo_rl/experience/blackbox_finalizer.py +++ b/nemo_rl/experience/blackbox_finalizer.py @@ -52,9 +52,11 @@ ) from nemo_rl.experience.payload import pack_payload from nemo_rl.experience.row_dump import maybe_dump_train_rows -from nemo_rl.models.generation.interfaces import ( - ROUTED_EXPERTS_MISSING_ROUTE_SENTINEL as _ROUTED_EXPERTS_SENTINEL, -) + +# Keep the finalizer importable in its CPU-only actor without importing the +# generation package (which eagerly loads backend dependencies). This value is +# the shared router-replay missing-route wire sentinel. +_ROUTED_EXPERTS_SENTINEL = -1 @dataclass(frozen=True) @@ -92,6 +94,112 @@ class FinalizedGroup: dropped: bool = False +def _linearize_metadata_only(receipt: Any, snapshots: list[Any]) -> Any: + """Linearize a verified chain while leaving digest-bound route bytes in TQ.""" + from nemo_gym.token_id_capture.staging.rebuild import ( + LinearizedRow, + RebuildError, + WeightVersionSpan, + ) + + records = {record.model_call_id: record for record in receipt.manifest} + staged = {snapshot.model_call_id: snapshot for snapshot in snapshots} + for model_call_id, record in records.items(): + if record.parent_call_id is not None: + parent = records.get(record.parent_call_id) + if parent is None: + raise RebuildError( + "missing_parent", + f"call {model_call_id} names absent parent {record.parent_call_id}", + ) + if parent.cum_len != record.prev_len: + raise RebuildError( + "parent_length_mismatch", + f"call {model_call_id} starts at {record.prev_len}, parent ends at {parent.cum_len}", + ) + visited: set[str] = set() + cursor = record + while cursor is not None: + if cursor.model_call_id in visited: + raise RebuildError( + "lineage_cycle", f"cycle reaches call {cursor.model_call_id}" + ) + visited.add(cursor.model_call_id) + cursor = ( + records.get(cursor.parent_call_id) + if cursor.parent_call_id is not None + else None + ) + + terminal_id = receipt.terminal_model_call_id + if terminal_id is None or terminal_id not in records: + raise RebuildError( + "missing_terminal", "successful receipt has no terminal call" + ) + chain = [] + cursor = records[terminal_id] + while cursor is not None: + chain.append(cursor) + cursor = ( + records.get(cursor.parent_call_id) + if cursor.parent_call_id is not None + else None + ) + chain.reverse() + + token_ids: list[int] = [] + token_mask: list[float] = [] + logprobs: list[float] = [] + model_call_ids: list[str] = [] + weight_versions: list[int] = [] + weight_version_spans = [] + link_spans: list[tuple[str, int, int]] = [] + prompt_len = 0 + for index, record in enumerate(chain): + snapshot = staged[record.model_call_id] + boundary = 0 + for mask in snapshot.token_mask_delta: + if mask != 0.0: + break + boundary += 1 + if boundary == len(snapshot.token_mask_delta) or any( + mask != 1.0 for mask in snapshot.token_mask_delta[boundary:] + ): + raise RebuildError( + "invalid_mask_order", + f"call {record.model_call_id} mask is not carry-then-generation", + ) + start = len(token_ids) + token_ids.extend(snapshot.token_ids_delta) + token_mask.extend(snapshot.token_mask_delta) + logprobs.extend(snapshot.generation_log_probs_delta) + end = len(token_ids) + if index == 0: + prompt_len = boundary + model_call_ids.append(record.model_call_id) + weight_versions.append(record.weight_version) + weight_version_spans.append( + WeightVersionSpan( + model_call_id=record.model_call_id, + start=start, + end=end, + weight_version=record.weight_version, + ) + ) + link_spans.append((record.model_call_id, boundary, record.delta_len - boundary)) + return LinearizedRow( + rollout_id=receipt.rollout_id, + token_ids=token_ids, + token_mask=token_mask, + logprobs=logprobs, + model_call_ids=model_call_ids, + prompt_len=prompt_len, + weight_versions=weight_versions, + weight_version_spans=weight_version_spans, + link_spans=link_spans, + ) + + class BlackboxFinalizer: """Receipts -> verified rows -> N-row publish, off the generation hot path.""" @@ -141,7 +249,11 @@ def finalize_rollout( """ # Deferred: nemo_gym is an optional extra absent in non-gym runs. from nemo_gym.token_id_capture.staging.digest import compute_staging_digest - from nemo_gym.token_id_capture.staging.rebuild import RebuildError, linearize + from nemo_gym.token_id_capture.staging.rebuild import ( + ReceiptVerificationError, + RebuildError, + verify_and_linearize, + ) from nemo_gym.token_id_capture.staging.records import RolloutReceipt def rejected(reason: str, staging_keys: list[str]) -> FinalizedRollout: @@ -177,7 +289,7 @@ def rejected(reason: str, staging_keys: list[str]) -> FinalizedRollout: "duplicate_staging_key", list(dict.fromkeys(staging_keys)), ) - records_by_call = {record.call_id: record for record in parsed.manifest} + records_by_call = {record.model_call_id: record for record in parsed.manifest} if len(records_by_call) != len(parsed.manifest): return rejected("duplicate_manifest_call_id", staging_keys) @@ -201,11 +313,13 @@ def rejected(reason: str, staging_keys: list[str]) -> FinalizedRollout: for record, item in zip(parsed.manifest, fetched): if item.staging_key != record.staging_key: return rejected( - f"staging_key_mismatch:{record.call_id}", staging_keys + f"staging_key_mismatch:{record.model_call_id}", staging_keys ) - if item.snapshot.call_id != record.call_id: - return rejected(f"call_id_mismatch:{record.call_id}", staging_keys) - fetched_by_call[record.call_id] = item + if item.snapshot.model_call_id != record.model_call_id: + return rejected( + f"call_id_mismatch:{record.model_call_id}", staging_keys + ) + fetched_by_call[record.model_call_id] = item if len(fetched_by_call) != len(fetched): return rejected("duplicate_fetched_call_id", staging_keys) @@ -213,47 +327,98 @@ def rejected(reason: str, staging_keys: list[str]) -> FinalizedRollout: if not ( len(snapshot.token_ids_delta) == len(snapshot.token_mask_delta) - == len(snapshot.logprobs_delta) + == len(snapshot.generation_log_probs_delta) ): - return rejected(f"misaligned_delta:{record.call_id}", staging_keys) + return rejected( + f"misaligned_delta:{record.model_call_id}", staging_keys + ) if any(m not in (0.0, 1.0) for m in snapshot.token_mask_delta): - return rejected(f"invalid_token_mask:{record.call_id}", staging_keys) - if any(not math.isfinite(p) for p in snapshot.logprobs_delta): - return rejected(f"non_finite_logprob:{record.call_id}", staging_keys) + return rejected( + f"invalid_token_mask:{record.model_call_id}", staging_keys + ) + if any(not math.isfinite(p) for p in snapshot.generation_log_probs_delta): + return rejected( + f"non_finite_logprob:{record.model_call_id}", staging_keys + ) if record.delta_len != len(snapshot.token_ids_delta) or ( snapshot.prev_len + record.delta_len != record.cum_len ): - return rejected(f"length_mismatch:{record.call_id}", staging_keys) - if snapshot.weight_version != record.weight_version: + return rejected(f"length_mismatch:{record.model_call_id}", staging_keys) + compared_fields = ( + "schema_version", + "digest_version", + "extras_digest_version", + "rollout_id", + "model_call_id", + "parent_call_id", + "mode", + "prev_len", + "delta_len", + "cum_len", + "weight_version", + "digest", + "extras_digest", + "chain_hash", + "cumulative_hash", + ) + mismatch = next( + ( + field_name + for field_name in compared_fields + if getattr(snapshot, field_name) + != ( + rollout_id + if field_name == "rollout_id" + else getattr(record, field_name) + ) + ), + None, + ) + if mismatch is not None: return rejected( - f"weight_version_mismatch:{record.call_id}", staging_keys + f"{mismatch}_mismatch:{record.model_call_id}", staging_keys + ) + try: + digest = compute_staging_digest( + schema_version=snapshot.schema_version, + digest_version=snapshot.digest_version, + extras_digest_version=snapshot.extras_digest_version, + rollout_id=rollout_id, + model_call_id=record.model_call_id, + parent_call_id=record.parent_call_id, + mode=record.mode, + prev_len=snapshot.prev_len, + delta_len=snapshot.delta_len, + cum_len=snapshot.cum_len, + weight_version=snapshot.weight_version, + token_ids_delta=snapshot.token_ids_delta, + token_mask_delta=snapshot.token_mask_delta, + generation_log_probs_delta=(snapshot.generation_log_probs_delta), + extras_digest=snapshot.extras_digest, + chain_hash=snapshot.chain_hash, + cumulative_hash=snapshot.cumulative_hash, + ) + except (TypeError, ValueError, OverflowError) as error: + return rejected( + f"invalid_digest_input:{record.model_call_id}:{error}", + staging_keys, ) - digest = compute_staging_digest( - rollout_id=rollout_id, - call_id=record.call_id, - prev_len=snapshot.prev_len, - token_ids_delta=snapshot.token_ids_delta, - token_mask_delta=snapshot.token_mask_delta, - logprobs_delta=snapshot.logprobs_delta, - ) if digest != record.digest: - return rejected(f"digest_mismatch:{record.call_id}", staging_keys) - - # Parent pointers are lineage state, not storage state: rejoin from - # the manifest before rebuilding (the storage rows carry them too, - # but the receipt is authoritative). - rejoined = [ - snapshot.model_copy(update={"parent_call_id": record.parent_call_id}) - for snapshot, record in zip(snapshots, parsed.manifest) - ] + return rejected(f"digest_mismatch:{record.model_call_id}", staging_keys) + try: - row = linearize( - rollout_id, - rejoined, - parsed.manifest, - terminal_hint=parsed.terminal_call_id, + row = ( + _linearize_metadata_only(parsed, snapshots) + if self._defer_routed_experts_to_policy + else verify_and_linearize(parsed, snapshots) ) - except (RebuildError, NotImplementedError) as error: + except ( + KeyError, + ValueError, + ReceiptVerificationError, + RebuildError, + NotImplementedError, + ) as error: return rejected(f"rebuild_failed:{error}", staging_keys) weight_versions = [record.weight_version for record in parsed.manifest] min_wv, max_wv = min(weight_versions), max(weight_versions) @@ -290,6 +455,8 @@ def rejected(reason: str, staging_keys: list[str]) -> FinalizedRollout: carry_len=int(carry_len), generation_len=int(generation_len), staged_route_len=item.routed_len, + extras_digest_version=record.extras_digest_version, + extras_digest=record.extras_digest, ) classify_route_span(span) route_spans.append(span) diff --git a/nemo_rl/experience/finalizer_actor.py b/nemo_rl/experience/finalizer_actor.py index ab85af2ab85..3502eee0807 100644 --- a/nemo_rl/experience/finalizer_actor.py +++ b/nemo_rl/experience/finalizer_actor.py @@ -33,6 +33,7 @@ "token_mask_delta", "generation_logprobs", "generation_logprobs_delta", + "generation_log_probs_delta", "logprobs_delta", "routed_experts", } diff --git a/nemo_rl/experience/rollout_manager.py b/nemo_rl/experience/rollout_manager.py index 945cca7984b..4d1faae6903 100644 --- a/nemo_rl/experience/rollout_manager.py +++ b/nemo_rl/experience/rollout_manager.py @@ -801,7 +801,8 @@ async def gate_metrics(self) -> Optional[dict[str, int]]: Returns: Cumulative gate counters (token_in, fallback_*, capture_failed, - registered/sealed/failed/expired) from ``/ng-control/metrics``, + registered/sealed/failed/expired) from Gym's token-capture control + metrics route, or None when no NemoGym env handle is wired. """ env = self._env_handles.get("nemo_gym") if self._env_handles else None diff --git a/nemo_rl/experience/route_plan.py b/nemo_rl/experience/route_plan.py index 8ecb092b7ac..ff31c39fa91 100644 --- a/nemo_rl/experience/route_plan.py +++ b/nemo_rl/experience/route_plan.py @@ -19,7 +19,9 @@ from dataclasses import dataclass from typing import Any, Literal -ROUTE_PLAN_SCHEMA_VERSION = 1 +ROUTE_PLAN_SCHEMA_VERSION = 2 +_EXTRAS_DIGEST_VERSION = 1 +_SHA256_HEX_LENGTH = 64 @dataclass(frozen=True) @@ -30,6 +32,8 @@ class RouteSpan: carry_len: int generation_len: int staged_route_len: int + extras_digest_version: int + extras_digest: str @dataclass(frozen=True) @@ -116,6 +120,25 @@ def _validate_plan(plan: RouteAssemblyPlan) -> None: span.staged_route_len, where=f"route_plan.spans[{index}].staged_route_len", ) + if ( + type(span.extras_digest_version) is not int + or span.extras_digest_version != _EXTRAS_DIGEST_VERSION + ): + raise ValueError( + f"route_plan.spans[{index}].extras_digest_version must be " + f"{_EXTRAS_DIGEST_VERSION}" + ) + if ( + not isinstance(span.extras_digest, str) + or len(span.extras_digest) != _SHA256_HEX_LENGTH + or any( + character not in "0123456789abcdef" for character in span.extras_digest + ) + ): + raise ValueError( + f"route_plan.spans[{index}].extras_digest must be a lowercase " + "SHA-256 hex digest" + ) if span.staging_key not in cleanup_keys: raise ValueError( f"route_plan.spans[{index}] key {span.staging_key!r} is outside " @@ -143,6 +166,8 @@ def encode_route_plan(plan: RouteAssemblyPlan) -> dict[str, Any]: "carry_len": span.carry_len, "generation_len": span.generation_len, "staged_route_len": span.staged_route_len, + "extras_digest_version": span.extras_digest_version, + "extras_digest": span.extras_digest, } for span in plan.spans ], @@ -175,7 +200,14 @@ def decode_route_plan(value: Any) -> RouteAssemblyPlan: raise TypeError(f"route_plan.spans[{index}] must be a dict") _require_exact_keys( span_value, - {"staging_key", "carry_len", "generation_len", "staged_route_len"}, + { + "staging_key", + "carry_len", + "generation_len", + "staged_route_len", + "extras_digest_version", + "extras_digest", + }, where=f"route_plan.spans[{index}]", ) spans.append( @@ -196,6 +228,14 @@ def decode_route_plan(value: Any) -> RouteAssemblyPlan: span_value["staged_route_len"], where=f"route_plan.spans[{index}].staged_route_len", ), + extras_digest_version=_require_int( + span_value["extras_digest_version"], + where=f"route_plan.spans[{index}].extras_digest_version", + ), + extras_digest=_require_string( + span_value["extras_digest"], + where=f"route_plan.spans[{index}].extras_digest", + ), ) ) cleanup_value = value["cleanup_staging_keys"] diff --git a/nemo_rl/models/generation/vllm/vllm_worker_async.py b/nemo_rl/models/generation/vllm/vllm_worker_async.py index 9ad9428f11f..61275293a64 100644 --- a/nemo_rl/models/generation/vllm/vllm_worker_async.py +++ b/nemo_rl/models/generation/vllm/vllm_worker_async.py @@ -407,16 +407,60 @@ def _begin_request_capture(self, request: Any, prompt_token_ids: list[int]) -> N context = getattr(request, "ng_capture", None) if capture is None or not context: return + from nemo_gym.token_id_capture.staging.records import CaptureAdmission + call = capture.begin_call( - rollout_id=context["rollout_id"], - call_id=context["call_id"], - parent_call_id=context.get("parent_call_id"), - prev_len=int(context.get("prev_len") or 0), - mode=context.get("mode") or "text", + CaptureAdmission.model_validate(context), stream=bool(getattr(request, "stream", False)), ) self._capture_calls[id(request)] = (call, list(prompt_token_ids)) + @staticmethod + def _delta_align_routed_experts( + payload: dict[str, Any], *, prev_len: int, prompt_len: int, generated_len: int + ) -> None: + """Normalize optional vLLM routes to the exact staged token delta.""" + choices = payload.get("choices") or [] + if len(choices) != 1 or not isinstance(choices[0], dict): + return + choice = dict(choices[0]) + message = dict(choice.get("message") or {}) + routed = message.get("routed_experts") + if routed is None: + return + try: + from nemo_rl.utils.routed_experts_codec import ( + decode_routed_experts, + encode_routed_experts, + ) + + if isinstance(routed, str): + dtype_name = routed.split(":", 3)[1] + dtype = { + "int8": torch.int8, + "int16": torch.int16, + "int32": torch.int32, + }.get(dtype_name) + if dtype is None: + raise ValueError(f"unsupported routed_experts dtype {dtype_name!r}") + else: + dtype = torch.int16 + experts = decode_routed_experts(routed, dtype) + expected_full_len = prompt_len + generated_len + if experts.dim() != 3 or experts.shape[0] != expected_full_len: + raise ValueError( + f"route length {experts.shape[0]} does not match engine sequence " + f"length {expected_full_len}" + ) + message["routed_experts"] = encode_routed_experts(experts[prev_len:]) + except (IndexError, TypeError, ValueError) as error: + LOGGER.warning( + "dropping invalid routed_experts from staged capture: %s", error + ) + message.pop("routed_experts", None) + choice["message"] = message + payload["choices"] = [choice] + def _finish_request_capture(self, request: Any, content: dict) -> dict: """Stage the finished call and ride its coords on the response. @@ -436,6 +480,18 @@ def _finish_request_capture(self, request: Any, content: dict) -> dict: # preprocess-time engine prompt off the payload (see # nemo_gym.token_id_capture.adapters.vllm.extract_prompt_ids). payload["prompt_token_ids"] = prompt_token_ids + adapter = self.token_capture.adapter + if adapter is not None: + try: + generated_token_ids, _ = adapter.extract_generation(payload) + except Exception: # capture core will report the authoritative failure + generated_token_ids = [] + self._delta_align_routed_experts( + payload, + prev_len=call.admission.prev_len, + prompt_len=len(prompt_token_ids), + generated_len=len(generated_token_ids), + ) coords = self.token_capture.complete_call_from_response(call, payload) for choice in content.get("choices") or []: choice.pop("logprobs", None) diff --git a/tests/unit/data_plane/test_blackbox_finalizer.py b/tests/unit/data_plane/test_blackbox_finalizer.py index 72399efa7b4..07526c55292 100644 --- a/tests/unit/data_plane/test_blackbox_finalizer.py +++ b/tests/unit/data_plane/test_blackbox_finalizer.py @@ -35,10 +35,12 @@ nemo_gym = pytest.importorskip("nemo_gym.token_id_capture.staging") -from nemo_gym.token_id_capture.staging.conformance.kit import ( # noqa: E402 - build_fixture_artifacts, - f32, - load_fixture, +from nemo_gym.token_id_capture.staging.digest import ( # noqa: E402 + compute_extras_digest, + compute_staging_digest, +) +from nemo_gym.token_id_capture.staging.records import ( # noqa: E402 + StagedCallRecord, ) from nemo_rl.data_plane.tq_token_sink import ( # noqa: E402 @@ -53,6 +55,10 @@ from nemo_rl.data_plane.worker_mixin import TQWorkerMixin # noqa: E402 from nemo_rl.experience.blackbox_finalizer import BlackboxFinalizer # noqa: E402 from nemo_rl.experience.route_plan import decode_route_plan # noqa: E402 +from tests.unit.data_plane.token_capture_test_fixtures import ( # noqa: E402 + build_fixture_artifacts, + f32, +) pytestmark = pytest.mark.nemo_gym @@ -103,11 +109,7 @@ def _finalizer(tq_client, **overrides) -> BlackboxFinalizer: def _stage_fixture(tq_client, name: str, *, rollout_id: str | None = None): """Stage one golden fixture's rows (optionally re-keyed to rollout_id) and return (receipt_dict, expected LinearizedRow).""" - fixture = load_fixture(name) - if rollout_id is not None: - fixture = dict(fixture) - fixture["rollout_id"] = rollout_id - records, _, receipt, row = build_fixture_artifacts(fixture) + records, receipt, row = build_fixture_artifacts(name, rollout_id=rollout_id) sink = TQTokenSink(tq_client, staging_partition=STAGING_PARTITION) for record in records: assert sink.stage(record).ok @@ -140,7 +142,7 @@ def test_finalize_rollout_rejections(tq_client, partitions): finalizer.finalize_rollout("rej_a", poisoned, reward=0.0).rejection_reason == "capture_poisoned" ) - empty = dict(receipt, manifest=[], terminal_call_id=None) + empty = dict(receipt, manifest=[], terminal_model_call_id=None) assert ( finalizer.finalize_rollout("rej_a", empty, reward=0.0).rejection_reason == "empty_manifest" @@ -303,21 +305,67 @@ def _routes_for_delta(call_idx: int, n_tokens: int) -> list: ] +def _record_with_routes(record: StagedCallRecord, routes: list) -> StagedCallRecord: + extras = {"routed_experts": routes} + extras_digest = compute_extras_digest(extras) + digest = compute_staging_digest( + schema_version=record.schema_version, + digest_version=record.digest_version, + extras_digest_version=record.extras_digest_version, + rollout_id=record.rollout_id, + model_call_id=record.model_call_id, + parent_call_id=record.parent_call_id, + mode=record.mode, + prev_len=record.prev_len, + delta_len=record.delta_len, + cum_len=record.cum_len, + weight_version=record.weight_version, + token_ids_delta=record.token_ids_delta, + token_mask_delta=record.token_mask_delta, + generation_log_probs_delta=record.generation_log_probs_delta, + extras_digest=extras_digest, + chain_hash=record.chain_hash, + cumulative_hash=record.cumulative_hash, + ) + return StagedCallRecord.model_validate( + record.model_dump() + | {"extras": extras, "extras_digest": extras_digest, "digest": digest} + ) + + +def _receipt_with_staged_records(receipt, records): + manifest_by_id = {record.model_call_id: record for record in receipt.manifest} + return receipt.model_copy( + update={ + "manifest": [ + manifest_by_id[record.model_call_id].model_copy( + update={ + "digest": record.digest, + "extras_digest": record.extras_digest, + } + ) + for record in records + ] + } + ) + + def _stage_fixture_with_routes(tq_client, name: str, *, rollout_id: str): """Stage the golden fixture with per-call routed_experts extras attached. Returns (receipt_dict, expected LinearizedRow, routes_by_call). """ - fixture = dict(load_fixture(name)) - fixture["rollout_id"] = rollout_id - records, _, receipt, row = build_fixture_artifacts(fixture) + records, receipt, row = build_fixture_artifacts(name, rollout_id=rollout_id) sink = TQTokenSink(tq_client, staging_partition=_R3_STAGING) routes_by_call = {} + staged_records = [] for idx, record in enumerate(records): routes = _routes_for_delta(idx, len(record.token_ids_delta)) - routes_by_call[record.call_id] = routes - staged = record.model_copy(update={"extras": {"routed_experts": routes}}) + routes_by_call[record.model_call_id] = routes + staged = _record_with_routes(record, routes) + staged_records.append(staged) assert sink.stage(staged).ok + receipt = _receipt_with_staged_records(receipt, staged_records) return receipt.model_dump(), row, routes_by_call @@ -389,9 +437,9 @@ def test_finalize_group_router_replay_without_routes_fails_loudly( ): group_id = "grpr3b" rollout_id = f"{group_id}_g0" - fixture = dict(load_fixture("worked_example")) - fixture["rollout_id"] = rollout_id - records, _, receipt, _ = build_fixture_artifacts(fixture) + records, receipt, _ = build_fixture_artifacts( + "worked_example", rollout_id=rollout_id + ) sink = TQTokenSink(tq_client, staging_partition=_R3_STAGING) for record in records: assert sink.stage(record).ok # no extras staged @@ -453,16 +501,19 @@ def r3_deferred_partitions(tq_client): def _stage_deferred_fixture(tq_client, *, rollout_id: str): - fixture = dict(load_fixture("worked_example")) - fixture["rollout_id"] = rollout_id - records, _, receipt, row = build_fixture_artifacts(fixture) + records, receipt, row = build_fixture_artifacts( + "worked_example", rollout_id=rollout_id + ) sink = TQTokenSink(tq_client, staging_partition=_R3_DEFERRED_STAGING) routes_by_call = {} + staged_records = [] for idx, record in enumerate(records): routes = _routes_for_delta(idx, len(record.token_ids_delta)) - routes_by_call[record.call_id] = routes - staged = record.model_copy(update={"extras": {"routed_experts": routes}}) + routes_by_call[record.model_call_id] = routes + staged = _record_with_routes(record, routes) + staged_records.append(staged) assert sink.stage(staged).ok + receipt = _receipt_with_staged_records(receipt, staged_records) return receipt.model_dump(), row, routes_by_call diff --git a/tests/unit/data_plane/test_tq_token_sink.py b/tests/unit/data_plane/test_tq_token_sink.py index b94d187665c..3e3ba63aeec 100644 --- a/tests/unit/data_plane/test_tq_token_sink.py +++ b/tests/unit/data_plane/test_tq_token_sink.py @@ -26,12 +26,6 @@ nemo_gym = pytest.importorskip("nemo_gym.token_id_capture.staging") -from nemo_gym.token_id_capture.staging.conformance import ( # noqa: E402 - build_fixture_artifacts, - fixture_names, - load_fixture, - run_sink_source_conformance, -) from nemo_gym.token_id_capture.staging.protocols import ( # noqa: E402 StagingSink as TokenSinkProtocol, ) @@ -44,6 +38,10 @@ TQTokenSink, TQTokenSource, ) +from tests.unit.data_plane.token_capture_test_fixtures import ( # noqa: E402 + build_fixture_artifacts, + fixture_names, +) STAGING_PARTITION = "rollout_staging_test" @@ -76,7 +74,13 @@ def test_tq_sink_source_passes_conformance(tq_client, staging_partition, fixture assert fixture_name in fixture_names() sink = TQTokenSink(tq_client, staging_partition=staging_partition) source = TQTokenSource(tq_client, staging_partition=staging_partition) - run_sink_source_conformance(load_fixture(fixture_name), sink, source) + records, _, _ = build_fixture_artifacts(fixture_name) + for record in records: + assert sink.stage(record).ok + snapshots = source.fetch([record.staging_key for record in records]) + assert [snapshot.model_dump() for snapshot in snapshots] == [ + record.model_dump() for record in records + ] def test_fetch_missing_key_raises_keyerror(tq_client, staging_partition): @@ -98,7 +102,7 @@ def get_samples(self, **kwargs): return self.client.get_samples(**kwargs) sink = TQTokenSink(tq_client, staging_partition=staging_partition) - records, _, _, _ = build_fixture_artifacts(load_fixture("single_call")) + records, _, _ = build_fixture_artifacts("single_call") assert sink.stage(records[0]).ok recording_client = RecordingClient(tq_client) source = TQTokenSource(recording_client, staging_partition=staging_partition) @@ -109,7 +113,7 @@ def get_samples(self, **kwargs): assert "routed_experts" not in recording_client.select_fields assert len(fetched) == 1 assert fetched[0].staging_key == records[0].staging_key - assert fetched[0].snapshot.call_id == records[0].call_id + assert fetched[0].snapshot.model_call_id == records[0].model_call_id assert fetched[0].routed_len == 0 @@ -127,7 +131,7 @@ def put_samples(self, **kwargs): raise RuntimeError("controller down") sink = TQTokenSink(ExplodingClient(), staging_partition=staging_partition) - records, _, _, _ = build_fixture_artifacts(load_fixture("single_call")) + records, _, _ = build_fixture_artifacts("single_call") result = sink.stage(records[0]) assert not result.ok assert result.staging_key == records[0].staging_key @@ -137,7 +141,7 @@ def put_samples(self, **kwargs): def test_sink_clear_drops_rows(tq_client, staging_partition): sink = TQTokenSink(tq_client, staging_partition=staging_partition) source = TQTokenSource(tq_client, staging_partition=staging_partition) - records, _, _, _ = build_fixture_artifacts(load_fixture("single_call")) + records, _, _ = build_fixture_artifacts("single_call") for record in records: assert sink.stage(record).ok keys = [record.staging_key for record in records] diff --git a/tests/unit/data_plane/test_worker_route_assembly.py b/tests/unit/data_plane/test_worker_route_assembly.py index 7e6154d2905..015f618f1d9 100644 --- a/tests/unit/data_plane/test_worker_route_assembly.py +++ b/tests/unit/data_plane/test_worker_route_assembly.py @@ -18,12 +18,15 @@ from collections import Counter import torch +from nemo_gym.token_id_capture.staging.digest import compute_extras_digest from nemo_rl.data_plane import KVBatchMeta from nemo_rl.data_plane.schema import ( ROUTE_PASSTHROUGH_FLAG, ROUTE_PLAN_TAG, + ROUTED_EXPERTS_ENCODING_FIELD, ROUTED_EXPERTS_FIELD, + ROUTED_EXTRAS_METADATA_FIELD, ) from nemo_rl.data_plane.worker_mixin import TQWorkerMixin from nemo_rl.distributed.batched_data_dict import BatchedDataDict @@ -33,11 +36,22 @@ RouteSpan, encode_route_plan, ) +from nemo_rl.utils.routed_experts_codec import encode_routed_experts class _Rows(dict): def __init__(self, routed: list[torch.Tensor]) -> None: - super().__init__({ROUTED_EXPERTS_FIELD: routed}) + super().__init__( + { + ROUTED_EXPERTS_FIELD: routed, + ROUTED_EXPERTS_ENCODING_FIELD: [ + torch.tensor([1], dtype=torch.int64) for _ in routed + ], + ROUTED_EXTRAS_METADATA_FIELD: [ + torch.tensor(list(b"{}"), dtype=torch.uint8) for _ in routed + ], + } + ) self.batch_size = (len(routed),) @@ -48,7 +62,11 @@ def __init__(self, fragments: dict[str, torch.Tensor]) -> None: def get_samples(self, *, sample_ids, partition_id, select_fields): assert partition_id == "staging" - assert select_fields == [ROUTED_EXPERTS_FIELD] + assert select_fields == [ + ROUTED_EXPERTS_FIELD, + ROUTED_EXPERTS_ENCODING_FIELD, + ROUTED_EXTRAS_METADATA_FIELD, + ] self.calls.append(list(sample_ids)) return _Rows([self.fragments[key] for key in sample_ids]) @@ -76,6 +94,27 @@ def _plan( ) +def _span( + client: _RouteClient, + staging_key: str, + carry_len: int, + generation_len: int, + staged_route_len: int, +) -> RouteSpan: + routes = client.fragments[staging_key] + extras_digest = compute_extras_digest( + {ROUTED_EXPERTS_FIELD: encode_routed_experts(routes)} + ) + return RouteSpan( + staging_key, + carry_len, + generation_len, + staged_route_len, + extras_digest_version=1, + extras_digest=extras_digest, + ) + + def _meta(plans: list[dict], lengths: list[int]) -> tuple[KVBatchMeta, BatchedDataDict]: meta = KVBatchMeta( partition_id="canonical", @@ -105,8 +144,8 @@ def test_worker_coalesces_keys_and_replays_full_tail_and_placeholder() -> None: plans = [ _plan( ( - RouteSpan("r/c0", 0, 2, 2), - RouteSpan("r/c1", 1, 1, 1), + _span(client, "r/c0", 0, 2, 2), + _span(client, "r/c1", 1, 1, 1), ), expected=4, cleanup=("r/c0", "r/c1", "r/off_chain"), @@ -136,7 +175,7 @@ def test_wrong_model_shape_falls_back_for_entire_rollout() -> None: client = _RouteClient({"r/c0": torch.tensor([[[10]], [[11]]], dtype=torch.int16)}) worker = _Worker(client) plan = _plan( - (RouteSpan("r/c0", 0, 2, 2),), + (_span(client, "r/c0", 0, 2, 2),), expected=2, cleanup=("r/c0",), ) @@ -146,3 +185,21 @@ def test_wrong_model_shape_falls_back_for_entire_rollout() -> None: assert bool(routed.eq(-1).all()) assert worker._route_fallback_counts == Counter({"fragment_model_shape": 1}) + + +def test_tampered_fragment_falls_back_for_entire_rollout() -> None: + client = _RouteClient( + {"r/c0": torch.tensor([[[10, 11]], [[12, 13]]], dtype=torch.int16)} + ) + worker = _Worker(client) + span = _span(client, "r/c0", 0, 2, 2) + client.fragments["r/c0"][0, 0, 0] = 999 + meta, data = _meta( + [_plan((span,), expected=2, cleanup=("r/c0",))], + [2], + ) + + routed = worker._maybe_assemble_routed_experts(meta, data)[ROUTED_EXPERTS_FIELD] + + assert bool(routed.eq(-1).all()) + assert worker._route_fallback_counts == Counter({"fragment_integrity": 1}) diff --git a/tests/unit/data_plane/token_capture_test_fixtures.py b/tests/unit/data_plane/token_capture_test_fixtures.py new file mode 100644 index 00000000000..0c492e72d47 --- /dev/null +++ b/tests/unit/data_plane/token_capture_test_fixtures.py @@ -0,0 +1,137 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Small v2 Gym staging fixtures shared by the RL data-plane tests.""" + +from __future__ import annotations + +from nemo_gym.token_id_capture.staging.digest import ( + compute_extras_digest, + compute_staging_digest, +) +from nemo_gym.token_id_capture.staging.rebuild import verify_and_linearize +from nemo_gym.token_id_capture.staging.records import ( + CallRecord, + RolloutReceipt, + StagedCallRecord, + StagedCallSnapshot, +) + + +def f32(value: float) -> float: + """Round one value through the float32 wire representation.""" + import struct + + return struct.unpack(">f", struct.pack(">f", value))[0] + + +def _record( + *, + rollout_id: str, + model_call_id: str, + parent_call_id: str | None, + prev_len: int, + token_ids: list[int], + token_mask: list[float], + logprobs: list[float], + weight_version: int, +) -> StagedCallRecord: + token_mask = [f32(value) for value in token_mask] + logprobs = [f32(value) for value in logprobs] + mode = "text" if parent_call_id is None else "token_in" + delta_len = len(token_ids) + cum_len = prev_len + delta_len + extras_digest = compute_extras_digest(None) + values = { + "rollout_id": rollout_id, + "model_call_id": model_call_id, + "parent_call_id": parent_call_id, + "mode": mode, + "prev_len": prev_len, + "delta_len": delta_len, + "cum_len": cum_len, + "weight_version": weight_version, + "token_ids_delta": token_ids, + "token_mask_delta": token_mask, + "generation_log_probs_delta": logprobs, + "extras": None, + "extras_digest": extras_digest, + } + return StagedCallRecord( + **values, + digest=compute_staging_digest( + schema_version=2, + digest_version=2, + extras_digest_version=1, + **{key: value for key, value in values.items() if key != "extras"}, + ), + ) + + +def _manifest(record: StagedCallRecord) -> CallRecord: + return CallRecord( + model_call_id=record.model_call_id, + parent_call_id=record.parent_call_id, + mode=record.mode, + prev_len=record.prev_len, + delta_len=record.delta_len, + cum_len=record.cum_len, + weight_version=record.weight_version, + digest=record.digest, + extras_digest=record.extras_digest, + staging_key=record.staging_key, + chain_hash=record.chain_hash, + cumulative_hash=record.cumulative_hash, + ) + + +def fixture_names() -> tuple[str, ...]: + return ("worked_example", "single_call", "mixed_weight_versions") + + +def build_fixture_artifacts( + name: str, *, rollout_id: str | None = None +) -> tuple[list[StagedCallRecord], RolloutReceipt, object]: + if name not in fixture_names(): + raise KeyError(name) + rollout_id = ( + rollout_id + or { + "worked_example": "g7_r0", + "single_call": "single_r0", + "mixed_weight_versions": "mixed_r0", + }[name] + ) + root = _record( + rollout_id=rollout_id, + model_call_id="c1", + parent_call_id=None, + prev_len=0, + token_ids=[10, 11, 12, 13], + token_mask=[0.0, 0.0, 1.0, 1.0], + logprobs=[0.0, 0.0, -0.1, -0.2], + weight_version=4, + ) + records = [root] + if name != "single_call": + records.append( + _record( + rollout_id=rollout_id, + model_call_id="c2", + parent_call_id="c1", + prev_len=root.cum_len, + token_ids=[20, 21, 22], + token_mask=[0.0, 1.0, 1.0], + logprobs=[0.0, -0.3, -0.4], + weight_version=5 if name == "mixed_weight_versions" else 4, + ) + ) + receipt = RolloutReceipt( + rollout_id=rollout_id, + terminal_model_call_id=records[-1].model_call_id, + manifest=[_manifest(record) for record in records], + ) + snapshots = [ + StagedCallSnapshot.model_validate(record.model_dump()) for record in records + ] + return records, receipt, verify_and_linearize(receipt, snapshots) diff --git a/tests/unit/environments/test_nemo_gym_token_capture.py b/tests/unit/environments/test_nemo_gym_token_capture.py new file mode 100644 index 00000000000..a87b040f622 --- /dev/null +++ b/tests/unit/environments/test_nemo_gym_token_capture.py @@ -0,0 +1,90 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import asyncio +from unittest.mock import AsyncMock + +from nemo_rl.environments.nemo_gym import NemoGym + + +def _capture_env() -> NemoGym: + env_cls = NemoGym.__ray_metadata__.modified_class + env = object.__new__(env_cls) + env._control_owner_id = "owner-1" + return env + + +def test_register_rollouts_returns_capabilities_and_uses_idempotent_ownership() -> None: + env = _capture_env() + env._control = AsyncMock( + side_effect=[ + {"data_capability": "cap-r0"}, + {"data_capability": "cap-r1"}, + ] + ) + + capabilities = asyncio.run(env.register_rollouts(["r0", "r1"])) + + assert capabilities == {"r0": "cap-r0", "r1": "cap-r1"} + for rollout_id, call in zip(("r0", "r1"), env._control.await_args_list): + assert call.args == ( + "PUT", + f"/training-token-capture/control/rollouts/{rollout_id}", + ) + assert call.kwargs["json"]["owner_id"] == "owner-1" + assert call.kwargs["json"]["operation_id"].startswith("register-") + + +def test_receipt_postprocess_fails_closed_without_a_terminal_logical_id() -> None: + env = _capture_env() + env.fail_rollouts = AsyncMock() + + result = asyncio.run( + env._postprocess_receipt_mode( + {"_ng_rollout_id": "r0"}, + {"reward": 1.0}, + ) + ) + + env.fail_rollouts.assert_awaited_once_with( + ["r0"], reason="missing_terminal_logical_request_id" + ) + assert result["receipt"] is None + + +def test_receipt_postprocess_seals_the_trusted_logical_terminal() -> None: + env = _capture_env() + env._control = AsyncMock(return_value={"rollout_id": "r0", "manifest": []}) + + result = asyncio.run( + env._postprocess_receipt_mode( + {"_ng_rollout_id": "r0"}, + { + "reward": 1.0, + "terminal_logical_request_id": "logical-terminal", + }, + ) + ) + + call = env._control.await_args + assert call.args == ( + "POST", + "/training-token-capture/control/rollouts/r0/seal", + ) + assert call.kwargs["json"]["owner_id"] == "owner-1" + assert call.kwargs["json"]["operation_id"].startswith("seal-") + assert call.kwargs["json"]["terminal_logical_request_id"] == ("logical-terminal") + assert result["receipt"] == {"rollout_id": "r0", "manifest": []} diff --git a/tests/unit/experience/test_route_plan.py b/tests/unit/experience/test_route_plan.py index 1b3e3fc048d..af292e0cd5d 100644 --- a/tests/unit/experience/test_route_plan.py +++ b/tests/unit/experience/test_route_plan.py @@ -26,16 +26,34 @@ encode_route_plan, ) +_DIGEST = "0" * 64 + + +def _span( + staging_key: str, + carry_len: int, + generation_len: int, + staged_route_len: int, +) -> RouteSpan: + return RouteSpan( + staging_key, + carry_len, + generation_len, + staged_route_len, + extras_digest_version=1, + extras_digest=_DIGEST, + ) + def _plan() -> RouteAssemblyPlan: return RouteAssemblyPlan( schema_version=ROUTE_PLAN_SCHEMA_VERSION, staging_partition="staging", spans=( - RouteSpan("r/c0", carry_len=2, generation_len=2, staged_route_len=4), - RouteSpan("r/c1", carry_len=3, generation_len=1, staged_route_len=1), - RouteSpan("r/c2", carry_len=0, generation_len=0, staged_route_len=0), - RouteSpan("r/c3", carry_len=1, generation_len=1, staged_route_len=0), + _span("r/c0", carry_len=2, generation_len=2, staged_route_len=4), + _span("r/c1", carry_len=3, generation_len=1, staged_route_len=1), + _span("r/c2", carry_len=0, generation_len=0, staged_route_len=0), + _span("r/c3", carry_len=1, generation_len=1, staged_route_len=0), ), cleanup_staging_keys=("r/c0", "r/c1", "r/c2", "r/c3", "r/off_chain"), expected_token_length=10, diff --git a/tests/unit/models/generation/test_vllm_token_capture_hosting.py b/tests/unit/models/generation/test_vllm_token_capture_hosting.py index 65111211205..4bc4424be23 100644 --- a/tests/unit/models/generation/test_vllm_token_capture_hosting.py +++ b/tests/unit/models/generation/test_vllm_token_capture_hosting.py @@ -34,6 +34,7 @@ RolloutTokenCapture, ) from nemo_gym.token_id_capture.staging.records import ( # noqa: E402 + CaptureAdmission, StagedCallRecord, StageResult, ) @@ -125,9 +126,13 @@ def test_weight_version_is_stamped_from_worker_state(monkeypatch): ) asyncio.run(VllmAsyncGenerationWorkerImpl.set_rollout_weight_version(worker, 4)) - first = worker.token_capture.begin_call(rollout_id="r", call_id="c1", mode="text") + first = worker.token_capture.begin_call( + CaptureAdmission(rollout_id="r", model_call_id="c1", mode="text") + ) asyncio.run(VllmAsyncGenerationWorkerImpl.set_rollout_weight_version(worker, 5)) - second = worker.token_capture.begin_call(rollout_id="r", call_id="c2", mode="text") + second = worker.token_capture.begin_call( + CaptureAdmission(rollout_id="r", model_call_id="c2", mode="text") + ) assert (first.weight_version, second.weight_version) == (4, 5) @@ -148,6 +153,10 @@ def _generation_with_mock_group(*, async_engine: bool = True) -> VllmGeneration: def test_generation_setup_token_capture_fans_out(monkeypatch): gen = _generation_with_mock_group() + monkeypatch.setattr( + "nemo_rl.models.generation.vllm.vllm_generation.ray.get", + lambda futures: futures, + ) gen.setup_token_capture({"backend": "simple"}, "rollout_staging") gen.worker_group.run_all_workers_single_data.assert_called_once_with( "setup_token_capture", @@ -163,8 +172,12 @@ def test_generation_setup_token_capture_requires_async_engine(): gen.setup_token_capture({}, "rollout_staging") -def test_generation_set_rollout_weight_version_fans_out(): +def test_generation_set_rollout_weight_version_fans_out(monkeypatch): gen = _generation_with_mock_group() + monkeypatch.setattr( + "nemo_rl.models.generation.vllm.vllm_generation.ray.get", + lambda futures: futures, + ) gen.set_rollout_weight_version(7) gen.worker_group.run_all_workers_single_data.assert_called_once_with( "set_rollout_weight_version", @@ -187,6 +200,9 @@ def _worker_with_capture(sink: _MemorySink): worker = _fake_worker() worker._capture_calls = {} + worker._delta_align_routed_experts = ( + VllmAsyncGenerationWorkerImpl._delta_align_routed_experts + ) worker.token_capture = RolloutTokenCapture( sink=sink, weight_version_fn=lambda: worker._rollout_weight_version, @@ -218,7 +234,7 @@ def test_request_capture_round_trip_stages_and_rides_coords(): request = _FakeRequest( ng_capture={ "rollout_id": "r0", - "call_id": "c1", + "model_call_id": "c1", "parent_call_id": None, "prev_len": 0, "mode": "text", @@ -250,10 +266,11 @@ def test_request_capture_token_in_prev_len_chains(): request = _FakeRequest( ng_capture={ "rollout_id": "r0", - "call_id": "c2", + "model_call_id": "c2", "parent_call_id": "c1", "prev_len": 3, "mode": "token_in", + "required_prefix_token_ids": [10, 11, 12], }, stream=False, ) @@ -292,7 +309,7 @@ def test_request_capture_abort_fails_the_call_and_drains_state(): request = _FakeRequest( ng_capture={ "rollout_id": "r0", - "call_id": "c1", + "model_call_id": "c1", "parent_call_id": None, "prev_len": 0, "mode": "text", diff --git a/tests/unit/single_controller/test_finalizer_lifecycle.py b/tests/unit/single_controller/test_finalizer_lifecycle.py index a8f67220914..5f061fec181 100644 --- a/tests/unit/single_controller/test_finalizer_lifecycle.py +++ b/tests/unit/single_controller/test_finalizer_lifecycle.py @@ -215,6 +215,8 @@ def test_post_train_cleanup_clears_canonical_rows_and_route_plan_staging_keys() carry_len=2, generation_len=1, staged_route_len=3, + extras_digest_version=1, + extras_digest="0" * 64, ), ), cleanup_staging_keys=("group_g0/call", "group_g0/fork"), diff --git a/tests/unit/single_controller/test_tq_replay_buffer.py b/tests/unit/single_controller/test_tq_replay_buffer.py index 852b5730614..9e48b346296 100644 --- a/tests/unit/single_controller/test_tq_replay_buffer.py +++ b/tests/unit/single_controller/test_tq_replay_buffer.py @@ -505,7 +505,7 @@ def test_commit_finalized_verifies_full_plan_manifest_ownership(self): RouteAssemblyPlan( schema_version=ROUTE_PLAN_SCHEMA_VERSION, staging_partition="rollout_staging", - spans=(RouteSpan("r0/on_chain", 0, 2, 2),), + spans=(RouteSpan("r0/on_chain", 0, 2, 2, 1, "0" * 64),), cleanup_staging_keys=("r0/on_chain", "r0/off_chain"), expected_token_length=2, ) diff --git a/ultra_launch.sh b/ultra_launch.sh index 70a07d7490d..5dc1f011d86 100755 --- a/ultra_launch.sh +++ b/ultra_launch.sh @@ -689,7 +689,7 @@ NRL_VLLM_CACHE_SEED_DIR=${NRL_VLLM_CACHE_SEED_DIR} \ DG_JIT_CACHE_DIR=${NRL_VLLM_LOCAL_CACHE_DIR}/deep_gemm \ TORCHINDUCTOR_CACHE_DIR=${INDUCTOR_CACHE_DIR} \ TRITON_CACHE_DIR=${TRITON_CACHE_DIR} \ -UV_CACHE_DIR=/tmp/nemo-gym-uv-cache-\${SLURM_JOB_ID:-default} \ +UV_CACHE_DIR=${UV_CACHE_DIR:-/tmp/nemo-gym-uv-cache-\${SLURM_JOB_ID:-default}} \ UV_LOCK_TIMEOUT=1800 \ RAY_ENABLE_UV_RUN_RUNTIME_ENV=0 \ UV_HTTP_TIMEOUT=10 \ From 0b22fed29870fa05cde27295682f203e64ac0e26 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 19 Aug 2026 04:58:57 -0700 Subject: [PATCH 24/52] fix(launcher): resolve configured uv cache path safely Signed-off-by: Pranav Thombre --- ultra_launch.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/ultra_launch.sh b/ultra_launch.sh index 5dc1f011d86..4f8d52d8089 100755 --- a/ultra_launch.sh +++ b/ultra_launch.sh @@ -676,6 +676,12 @@ export SETUP_COMMAND # learning rate, etc.) live in CONFIG_PATH. The launcher only passes the # per-run overrides: cluster shape, paths, judge endpoints, logging. # ============================================================================= +if [[ -n "${UV_CACHE_DIR:-}" ]]; then + TRAIN_UV_CACHE_DIR="${UV_CACHE_DIR}" +else + TRAIN_UV_CACHE_DIR='/tmp/nemo-gym-uv-cache-${SLURM_JOB_ID:-default}' +fi + TRAIN_CMD="cd ${CODE_ROOT} && date ; \ ${NRL_DRIVER_PIP_INSTALL:+uv pip install --python /opt/nemo_rl_venv/bin/python ${NRL_DRIVER_PIP_INSTALL} ; }\ ${GENRM_RUNTIME_SETUP}\ @@ -689,7 +695,7 @@ NRL_VLLM_CACHE_SEED_DIR=${NRL_VLLM_CACHE_SEED_DIR} \ DG_JIT_CACHE_DIR=${NRL_VLLM_LOCAL_CACHE_DIR}/deep_gemm \ TORCHINDUCTOR_CACHE_DIR=${INDUCTOR_CACHE_DIR} \ TRITON_CACHE_DIR=${TRITON_CACHE_DIR} \ -UV_CACHE_DIR=${UV_CACHE_DIR:-/tmp/nemo-gym-uv-cache-\${SLURM_JOB_ID:-default}} \ +UV_CACHE_DIR=${TRAIN_UV_CACHE_DIR} \ UV_LOCK_TIMEOUT=1800 \ RAY_ENABLE_UV_RUN_RUNTIME_ENV=0 \ UV_HTTP_TIMEOUT=10 \ From 1875d4299f3016a16f9860989f6c97ef92d1e31b Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 19 Aug 2026 05:15:50 -0700 Subject: [PATCH 25/52] fix(smoke): preserve Ray-compatible driver environment Signed-off-by: Pranav Thombre --- docs/guides/nano-swe-token-capture.md | 1 + swe_nano_sc_capture.sh | 3 +++ swe_nano_sc_capture_interactive.sh | 3 +++ ultra_launch.sh | 3 ++- 4 files changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/guides/nano-swe-token-capture.md b/docs/guides/nano-swe-token-capture.md index 5f4590bb458..3822311ad4c 100644 --- a/docs/guides/nano-swe-token-capture.md +++ b/docs/guides/nano-swe-token-capture.md @@ -248,6 +248,7 @@ its absence broke a run: | `NG_TIC_FP_CANONICAL=1` | Token-in silently degrades to ~0: reasoning models echo history with `` blocks stripped, the gate's fingerprint of the served turn never matches, and every call falls back to text mode. With it, `token_in_rate ≈ 0.9999`. | | `NRL_DRIVER_PYTHONPATH=/opt/nemo-rl/3rdparty/Gym-workspace/Gym` | Driver `ModuleNotFoundError: nemo_gym` — the driver imports the staging record schema, and the baked driver venv has no nemo_gym. | | `NRL_DRIVER_PIP_INSTALL=orjson` | Driver `ModuleNotFoundError: orjson` — Gym's `token_id_capture/__init__` eagerly imports the store. | +| `NRL_DRIVER_UV_RUN_FLAGS="--locked --no-sync"` | `uv run` otherwise replaces the prefetched driver environment and can give the driver a different Python/Ray version from the already-running Ray cluster. Lock mutation is forbidden; worker-specific environments are still rebuilt. | | `VllmAsyncGenerationWorker` in `NRL_FORCE_REBUILD_VENVS_LIST` | Worker `ModuleNotFoundError: orjson` — venv caching is spec-unaware and silently reuses a non-capture worker venv built by an earlier job. | | capture env set *after* sourcing `swe_nano.env` | `swe_nano.env` exports `NRL_FORCE_REBUILD_VENVS_LIST` unconditionally and clobbers an env-prefix value — which is why these are dedicated launchers rather than an env prefix on `swe_nano_sc.sh`. | | `CALL_TIMING=0` (optional, batch) | Per-call latency JSONL is on by default in the batch launcher (`NRL_CALL_TIMING_DIR`/`NG_CALL_TIMING_DIR`); set 0 to disable. All probes are env-gated and dormant without the dir. | diff --git a/swe_nano_sc_capture.sh b/swe_nano_sc_capture.sh index e979206f344..41397528c6c 100644 --- a/swe_nano_sc_capture.sh +++ b/swe_nano_sc_capture.sh @@ -31,6 +31,9 @@ BASE_LOG_DIR="${WORKSPACE_DIR}/ray_logs/${EXP_NAME}" NRL_FORCE_REBUILD_VENVS_LIST="nemo_rl.environments.nemo_gym.NemoGym,nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker" NRL_DRIVER_PYTHONPATH="/opt/nemo-rl/3rdparty/Gym-workspace/Gym" NRL_DRIVER_PIP_INSTALL="orjson" +# Ray is started from the container's prefetched environment. Keep the driver on +# that exact Python/Ray pair; only class-specific worker venvs are rebuilt below. +NRL_DRIVER_UV_RUN_FLAGS="--locked --no-sync" # --- Per-call latency breakdown (CALL_TIMING=0 to disable) -------------------- if [ "${CALL_TIMING:-1}" = "1" ]; then diff --git a/swe_nano_sc_capture_interactive.sh b/swe_nano_sc_capture_interactive.sh index 731f801f3a4..509f4a5602b 100755 --- a/swe_nano_sc_capture_interactive.sh +++ b/swe_nano_sc_capture_interactive.sh @@ -40,6 +40,9 @@ BASE_LOG_DIR="${WORKSPACE_DIR}/ray_logs/${EXP_NAME}" NRL_FORCE_REBUILD_VENVS_LIST="nemo_rl.environments.nemo_gym.NemoGym,nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker" NRL_DRIVER_PYTHONPATH="/opt/nemo-rl/3rdparty/Gym-workspace/Gym" NRL_DRIVER_PIP_INSTALL="orjson" +# Ray is started from the container's prefetched environment. Keep the driver on +# that exact Python/Ray pair; only class-specific worker venvs are rebuilt below. +NRL_DRIVER_UV_RUN_FLAGS="--locked --no-sync" set +a INTERACTIVE=1 DRY_RUN=0 INTERACTIVE_WAIT="${INTERACTIVE_WAIT:-1}" \ diff --git a/ultra_launch.sh b/ultra_launch.sh index 4f8d52d8089..195febcfba4 100755 --- a/ultra_launch.sh +++ b/ultra_launch.sh @@ -62,6 +62,7 @@ set -euo pipefail # INTERACTIVE_WALLTIME= override WALLTIME for the interactive alloc # HF_HOME= HuggingFace cache root (recommended) # HF_TOKEN= HuggingFace API token +# NRL_DRIVER_UV_RUN_FLAGS= Extra flags for the driver-only uv run # WANDB_API_KEY= Weights & Biases API key # WANDB_PROJ=nemotron-3-ultra W&B project # WANDB_ENTITY= W&B entity @@ -705,7 +706,7 @@ NRL_VLLM_ASYNC_TIMEOUT_SECONDS=1800 \ NRL_WG_USE_RAY_REF=1 \ HF_HOME=${HF_HOME:-} \ NRL_USE_FASTOKENS=${NRL_USE_FASTOKENS:-1} \ -uv run ${NRL_ENTRYPOINT:-./examples/nemo_gym/run_grpo_nemo_gym.py} \ +uv run ${NRL_DRIVER_UV_RUN_FLAGS:-} ${NRL_ENTRYPOINT:-./examples/nemo_gym/run_grpo_nemo_gym.py} \ --config ${CONFIG_PATH} \ policy.model_name=${MODEL_PATH} \ cluster.num_nodes=${NUM_ACTOR_NODES} \ From bf897c38f70beda0f7017bc19d331d254d7fb625 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 19 Aug 2026 05:55:56 -0700 Subject: [PATCH 26/52] fix(launcher): forward worker venv roots Signed-off-by: Pranav Thombre --- ultra_launch.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/ultra_launch.sh b/ultra_launch.sh index 195febcfba4..7a8853062f1 100755 --- a/ultra_launch.sh +++ b/ultra_launch.sh @@ -63,6 +63,8 @@ set -euo pipefail # HF_HOME= HuggingFace cache root (recommended) # HF_TOKEN= HuggingFace API token # NRL_DRIVER_UV_RUN_FLAGS= Extra flags for the driver-only uv run +# NEMO_RL_VENV_DIR= Worker venv root forwarded to the driver +# NEMO_GYM_VENV_DIR= Gym service venv root forwarded to the driver # WANDB_API_KEY= Weights & Biases API key # WANDB_PROJ=nemotron-3-ultra W&B project # WANDB_ENTITY= W&B entity @@ -682,6 +684,8 @@ if [[ -n "${UV_CACHE_DIR:-}" ]]; then else TRAIN_UV_CACHE_DIR='/tmp/nemo-gym-uv-cache-${SLURM_JOB_ID:-default}' fi +TRAIN_NEMO_RL_VENV_DIR="${NEMO_RL_VENV_DIR:-}" +TRAIN_NEMO_GYM_VENV_DIR="${NEMO_GYM_VENV_DIR:-}" TRAIN_CMD="cd ${CODE_ROOT} && date ; \ ${NRL_DRIVER_PIP_INSTALL:+uv pip install --python /opt/nemo_rl_venv/bin/python ${NRL_DRIVER_PIP_INSTALL} ; }\ @@ -697,6 +701,8 @@ DG_JIT_CACHE_DIR=${NRL_VLLM_LOCAL_CACHE_DIR}/deep_gemm \ TORCHINDUCTOR_CACHE_DIR=${INDUCTOR_CACHE_DIR} \ TRITON_CACHE_DIR=${TRITON_CACHE_DIR} \ UV_CACHE_DIR=${TRAIN_UV_CACHE_DIR} \ +${TRAIN_NEMO_RL_VENV_DIR:+NEMO_RL_VENV_DIR=${TRAIN_NEMO_RL_VENV_DIR} }\ +${TRAIN_NEMO_GYM_VENV_DIR:+NEMO_GYM_VENV_DIR=${TRAIN_NEMO_GYM_VENV_DIR} }\ UV_LOCK_TIMEOUT=1800 \ RAY_ENABLE_UV_RUN_RUNTIME_ENV=0 \ UV_HTTP_TIMEOUT=10 \ From ab06425e25e288cf8ad697ee5e4ab847608c24b6 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 19 Aug 2026 06:14:24 -0700 Subject: [PATCH 27/52] fix(launcher): forward managed uv Python settings Signed-off-by: Pranav Thombre --- ultra_launch.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/ultra_launch.sh b/ultra_launch.sh index 7a8853062f1..5d933d38556 100755 --- a/ultra_launch.sh +++ b/ultra_launch.sh @@ -65,6 +65,8 @@ set -euo pipefail # NRL_DRIVER_UV_RUN_FLAGS= Extra flags for the driver-only uv run # NEMO_RL_VENV_DIR= Worker venv root forwarded to the driver # NEMO_GYM_VENV_DIR= Gym service venv root forwarded to the driver +# UV_PYTHON= Python request used for uv-managed worker venvs +# UV_PYTHON_INSTALL_DIR= Managed Python root forwarded to the driver # WANDB_API_KEY= Weights & Biases API key # WANDB_PROJ=nemotron-3-ultra W&B project # WANDB_ENTITY= W&B entity @@ -686,6 +688,8 @@ else fi TRAIN_NEMO_RL_VENV_DIR="${NEMO_RL_VENV_DIR:-}" TRAIN_NEMO_GYM_VENV_DIR="${NEMO_GYM_VENV_DIR:-}" +TRAIN_UV_PYTHON="${UV_PYTHON:-}" +TRAIN_UV_PYTHON_INSTALL_DIR="${UV_PYTHON_INSTALL_DIR:-}" TRAIN_CMD="cd ${CODE_ROOT} && date ; \ ${NRL_DRIVER_PIP_INSTALL:+uv pip install --python /opt/nemo_rl_venv/bin/python ${NRL_DRIVER_PIP_INSTALL} ; }\ @@ -703,6 +707,8 @@ TRITON_CACHE_DIR=${TRITON_CACHE_DIR} \ UV_CACHE_DIR=${TRAIN_UV_CACHE_DIR} \ ${TRAIN_NEMO_RL_VENV_DIR:+NEMO_RL_VENV_DIR=${TRAIN_NEMO_RL_VENV_DIR} }\ ${TRAIN_NEMO_GYM_VENV_DIR:+NEMO_GYM_VENV_DIR=${TRAIN_NEMO_GYM_VENV_DIR} }\ +${TRAIN_UV_PYTHON:+UV_PYTHON=${TRAIN_UV_PYTHON} }\ +${TRAIN_UV_PYTHON_INSTALL_DIR:+UV_PYTHON_INSTALL_DIR=${TRAIN_UV_PYTHON_INSTALL_DIR} }\ UV_LOCK_TIMEOUT=1800 \ RAY_ENABLE_UV_RUN_RUNTIME_ENV=0 \ UV_HTTP_TIMEOUT=10 \ From 618c1c93ed764b83beac46445c1155ca3178ffcd Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 19 Aug 2026 06:38:23 -0700 Subject: [PATCH 28/52] build: lock rebased Gym worker environments Signed-off-by: Pranav Thombre --- nemo_rl/utils/venvs.py | 4 +- ultra_launch.sh | 19 +- uv.lock | 819 +---------------------------------------- 3 files changed, 34 insertions(+), 808 deletions(-) diff --git a/nemo_rl/utils/venvs.py b/nemo_rl/utils/venvs.py index fa5b8acb738..3afa42c2e7f 100644 --- a/nemo_rl/utils/venvs.py +++ b/nemo_rl/utils/venvs.py @@ -94,7 +94,9 @@ def create_local_venv( exec_cmd.extend(["echo", f"Finished creating venv {venv_path}"]) # Always run uv sync first to ensure the build requirements are set (for --no-build-isolation packages) - subprocess.run(["uv", "sync", "--directory", git_root], env=env, check=True) + subprocess.run( + ["uv", "sync", "--locked", "--directory", git_root], env=env, check=True + ) subprocess.run(exec_cmd, env=env, check=True) # Return the path to the python executable in the virtual environment diff --git a/ultra_launch.sh b/ultra_launch.sh index 5d933d38556..fae8bcaf145 100755 --- a/ultra_launch.sh +++ b/ultra_launch.sh @@ -67,6 +67,7 @@ set -euo pipefail # NEMO_GYM_VENV_DIR= Gym service venv root forwarded to the driver # UV_PYTHON= Python request used for uv-managed worker venvs # UV_PYTHON_INSTALL_DIR= Managed Python root forwarded to the driver +# NRL_UV_BIN_DIR= Directory containing the uv binary used by workers # WANDB_API_KEY= Weights & Biases API key # WANDB_PROJ=nemotron-3-ultra W&B project # WANDB_ENTITY= W&B entity @@ -561,9 +562,9 @@ fi # ============================================================================= # Container mounts # ============================================================================= -# By default, nemo_rl (Python package) and examples/configs (YAML configs) from -# the code snapshot are overlaid into the container. Everything else uses the -# container's built-in code at /opt/nemo-rl. +# By default, the project metadata/lock, nemo_rl (Python package), and +# examples/configs (YAML configs) from the code snapshot are overlaid into the +# container. Everything else uses the container's built-in code at /opt/nemo-rl. # # To overlay additional components (e.g. a local Megatron-LM checkout), pass # EXTRA_MOUNTS as a comma-separated list of host:container pairs: @@ -590,6 +591,12 @@ if [[ -d "${OVERLAY_SOURCE}/nemo_rl" ]]; then _append_mount "${OVERLAY_SOURCE}/nemo_rl:/opt/nemo-rl/nemo_rl" echo " Mount: nemo_rl → /opt/nemo-rl/nemo_rl" fi +for _project_file in pyproject.toml uv.lock .python-version; do + if [[ -f "${OVERLAY_SOURCE}/${_project_file}" ]]; then + _append_mount "${OVERLAY_SOURCE}/${_project_file}:/opt/nemo-rl/${_project_file}" + echo " Mount: ${_project_file} → /opt/nemo-rl/${_project_file}" + fi +done if [[ -d "${OVERLAY_SOURCE}/examples/configs" ]]; then _append_mount "${OVERLAY_SOURCE}/examples/configs:/opt/nemo-rl/examples/configs" echo " Mount: configs → /opt/nemo-rl/examples/configs" @@ -690,6 +697,11 @@ TRAIN_NEMO_RL_VENV_DIR="${NEMO_RL_VENV_DIR:-}" TRAIN_NEMO_GYM_VENV_DIR="${NEMO_GYM_VENV_DIR:-}" TRAIN_UV_PYTHON="${UV_PYTHON:-}" TRAIN_UV_PYTHON_INSTALL_DIR="${UV_PYTHON_INSTALL_DIR:-}" +if [[ -n "${NRL_UV_BIN_DIR:-}" ]]; then + TRAIN_NRL_PATH="${NRL_UV_BIN_DIR}:\${PATH}" +else + TRAIN_NRL_PATH="" +fi TRAIN_CMD="cd ${CODE_ROOT} && date ; \ ${NRL_DRIVER_PIP_INSTALL:+uv pip install --python /opt/nemo_rl_venv/bin/python ${NRL_DRIVER_PIP_INSTALL} ; }\ @@ -709,6 +721,7 @@ ${TRAIN_NEMO_RL_VENV_DIR:+NEMO_RL_VENV_DIR=${TRAIN_NEMO_RL_VENV_DIR} }\ ${TRAIN_NEMO_GYM_VENV_DIR:+NEMO_GYM_VENV_DIR=${TRAIN_NEMO_GYM_VENV_DIR} }\ ${TRAIN_UV_PYTHON:+UV_PYTHON=${TRAIN_UV_PYTHON} }\ ${TRAIN_UV_PYTHON_INSTALL_DIR:+UV_PYTHON_INSTALL_DIR=${TRAIN_UV_PYTHON_INSTALL_DIR} }\ +${TRAIN_NRL_PATH:+PATH=${TRAIN_NRL_PATH} }\ UV_LOCK_TIMEOUT=1800 \ RAY_ENABLE_UV_RUN_RUNTIME_ENV=0 \ UV_HTTP_TIMEOUT=10 \ diff --git a/uv.lock b/uv.lock index 988cb862fb0..8c7331ac648 100644 --- a/uv.lock +++ b/uv.lock @@ -1,5 +1,5 @@ version = 1 -revision = 3 +revision = 4 requires-python = ">=3.13.14, <3.14" resolution-markers = [ "platform_machine == 'x86_64' and sys_platform == 'linux'", @@ -1825,15 +1825,6 @@ wheels = [ { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_aarch64.whl", hash = "sha256:9fdfbdf2d4ca984ffa34518a4054c62c295fa06e184d7a233848dc06be8f1ec2" }, ] -[package.metadata] -requires-dist = [ - { name = "einops" }, - { name = "ninja" }, - { name = "psutil" }, - { name = "setuptools" }, - { name = "torch" }, -] - [[package]] name = "flash-attn" version = "2.8.1+cu13torch2.10cxx11abitrue" @@ -1852,15 +1843,6 @@ wheels = [ { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_x86_64.whl", hash = "sha256:a14fea157eb61dfee56f605a24a7a6150704517759125b3942dd04bffa05d6cf" }, ] -[package.metadata] -requires-dist = [ - { name = "einops" }, - { name = "ninja" }, - { name = "psutil" }, - { name = "setuptools" }, - { name = "torch" }, -] - [[package]] name = "flash-attn-4" version = "4.0.0b11" @@ -3350,90 +3332,6 @@ te = [ { name = "transformer-engine", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, ] -[package.metadata] -requires-dist = [ - { name = "accelerate" }, - { name = "causal-conv1d", marker = "extra == 'ssm'" }, - { name = "comet-ml", specifier = ">=3.50.0" }, - { name = "datasets", specifier = ">=2.20.0" }, - { name = "diffusers", specifier = ">=0.36.0" }, - { name = "einops" }, - { name = "flash-linear-attention" }, - { name = "flashinfer-cubin", specifier = "==0.6.8.post1" }, - { name = "flashinfer-python", specifier = "==0.6.8.post1" }, - { name = "hydra-core", specifier = ">1.3,<=1.3.2" }, - { name = "librosa", marker = "extra == 'audio'" }, - { name = "mamba-ssm", marker = "extra == 'ssm'" }, - { name = "megatron-core", extras = ["dev", "mlm"], editable = "3rdparty/Megatron-Bridge-workspace/Megatron-Bridge/3rdparty/Megatron-LM" }, - { name = "mistral-common", specifier = ">=1.10.0" }, - { name = "mlflow", specifier = ">=3.15.1" }, - { name = "nemo-run", marker = "extra == 'recipes'" }, - { name = "nvdlfw-inspect", marker = "extra == 'tensor-inspect'", specifier = "==0.2.1" }, - { name = "nvidia-resiliency-ext" }, - { name = "omegaconf", specifier = ">=2.3.0" }, - { name = "open-clip-torch", specifier = ">=3.2.0" }, - { name = "peft", specifier = ">=0.18.0" }, - { name = "peft", specifier = ">=0.18.1" }, - { name = "pyarrow", marker = "extra == 'parquet'", specifier = ">=14.0.0" }, - { name = "pyyaml", specifier = ">=6.0.2" }, - { name = "qwen-vl-utils" }, - { name = "regex", specifier = ">=2024.11.6" }, - { name = "rich" }, - { name = "six", specifier = ">=1.17.0" }, - { name = "tensorboard", specifier = ">=2.19.0" }, - { name = "timm" }, - { name = "torch", specifier = ">=2.6.0" }, - { name = "tqdm", specifier = ">=4.67.1" }, - { name = "transformer-engine", extras = ["core-cu13", "pytorch"], marker = "extra == 'te'" }, - { name = "transformers", specifier = ">=5.8,<=5.12.1" }, - { name = "typing-extensions" }, - { name = "wandb", specifier = ">=0.25.0" }, -] -provides-extras = ["recipes", "parquet", "tensor-inspect", "te", "ssm", "audio"] - -[package.metadata.requires-dev] -build = [ - { name = "cython", specifier = ">=3.0.0" }, - { name = "ninja" }, - { name = "numpy" }, - { name = "nvidia-cudnn-frontend", specifier = ">=1.25.0" }, - { name = "nvidia-mathdx" }, - { name = "pybind11" }, - { name = "setuptools" }, - { name = "torch" }, -] -dev = [ - { name = "mypy", specifier = ">=1.8.0" }, - { name = "pre-commit", specifier = ">=3.6.0" }, - { name = "ruff", specifier = ">=0.9.9" }, -] -diffusion = [ - { name = "av" }, - { name = "imageio" }, - { name = "imageio-ffmpeg" }, -] -docs = [ - { name = "myst-parser", specifier = ">=4.0.1" }, - { name = "nvidia-sphinx-theme", specifier = ">=0.0.8" }, - { name = "sphinx", specifier = ">=8.1.3" }, - { name = "sphinx-autobuild", specifier = ">=2024.10.3" }, - { name = "sphinx-autodoc2", specifier = ">=0.5.0" }, - { name = "sphinx-copybutton", specifier = ">=0.5.2" }, - { name = "sphinxcontrib-mermaid" }, -] -no-pypi-wheels = [{ name = "flash-mla", git = "https://github.com/deepseek-ai/FlashMLA?rev=nv_dev" }] -test = [ - { name = "click" }, - { name = "coverage", specifier = ">=7.8.1" }, - { name = "flake8", specifier = ">=7.2.0" }, - { name = "pygithub" }, - { name = "pylint", specifier = ">=3.3.7" }, - { name = "pytest", specifier = ">=8.3.5" }, - { name = "pytest-mock", specifier = ">=3.14.0" }, - { name = "pytest-runner", specifier = ">=6.0.1" }, - { name = "pytest-timeout", specifier = ">=2.4.0" }, -] - [[package]] name = "megatron-core" source = { editable = "3rdparty/Megatron-Bridge-workspace/Megatron-Bridge/3rdparty/Megatron-LM" } @@ -3478,108 +3376,6 @@ mlm = [ { name = "wandb" }, ] -[package.metadata] -requires-dist = [ - { name = "accelerate", marker = "extra == 'mlm'" }, - { name = "accelerate", marker = "extra == 'training'" }, - { name = "av", marker = "extra == 'dev'" }, - { name = "causal-conv1d", marker = "extra == 'ssm'", specifier = "~=1.5" }, - { name = "datasets", marker = "extra == 'dev'" }, - { name = "einops", marker = "extra == 'dev'", specifier = "~=0.8" }, - { name = "emerging-optimizers", marker = "extra == 'dev'", git = "https://github.com/NVIDIA-NeMo/Emerging-Optimizers.git?rev=v0.2.0" }, - { name = "fast-hadamard-transform", marker = "extra == 'dev'", git = "https://github.com/Dao-AILab/fast-hadamard-transform.git?rev=f134af63deb2df17e1171a9ec1ea4a7d8604d5ca" }, - { name = "fastapi", marker = "extra == 'dev'", specifier = "~=0.50" }, - { name = "flash-linear-attention", marker = "extra == 'dev'", specifier = "~=0.4.0" }, - { name = "flashinfer-python", marker = "extra == 'dev'", specifier = ">=0.5.0,<0.7.0" }, - { name = "flask-restful", marker = "extra == 'mlm'" }, - { name = "flask-restful", marker = "extra == 'training'" }, - { name = "hypercorn", marker = "extra == 'dev'" }, - { name = "mamba-ssm", marker = "extra == 'ssm'", git = "https://github.com/state-spaces/mamba.git?rev=0048fbf2e7b2f214dcbe703ea3dec2b9647595e1" }, - { name = "megatron-energon", extras = ["av-decode"], marker = "extra == 'dev'", specifier = "~=7.0" }, - { name = "multi-storage-client", marker = "extra == 'dev'", specifier = "~=0.50" }, - { name = "numpy" }, - { name = "nvidia-cudnn-frontend", extras = ["cutedsl"], marker = "extra == 'dev'", specifier = "==1.26.0" }, - { name = "nvidia-modelopt", extras = ["torch"], marker = "sys_platform != 'darwin' and extra == 'dev'", specifier = ">=0.44" }, - { name = "nvidia-resiliency-ext", marker = "extra == 'dev'", specifier = "==0.6.0" }, - { name = "omegaconf", marker = "extra == 'mlm'" }, - { name = "omegaconf", marker = "extra == 'training'" }, - { name = "onnxscript", marker = "extra == 'dev'" }, - { name = "openai", extras = ["aiohttp"], marker = "extra == 'dev'" }, - { name = "opentelemetry-api", marker = "extra == 'dev'", specifier = "~=1.33.1" }, - { name = "orjson", marker = "extra == 'dev'" }, - { name = "packaging", specifier = ">=24.2" }, - { name = "quart", marker = "extra == 'dev'" }, - { name = "sentencepiece", marker = "extra == 'mlm'" }, - { name = "sentencepiece", marker = "extra == 'training'" }, - { name = "tensorstore", marker = "extra == 'dev'", specifier = "~=0.1,!=0.1.46,!=0.1.72" }, - { name = "tiktoken", marker = "extra == 'mlm'" }, - { name = "tiktoken", marker = "extra == 'training'" }, - { name = "torch", specifier = ">=2.6.0" }, - { name = "tqdm", marker = "extra == 'dev'" }, - { name = "transformer-engine", extras = ["core-cu13", "pytorch"], marker = "extra == 'te'", git = "https://github.com/NVIDIA/TransformerEngine.git?rev=e7c550c5f80636cf841a8204b1d6f85a5f3f28b7" }, - { name = "transformers", marker = "extra == 'mlm'" }, - { name = "transformers", marker = "extra == 'training'" }, - { name = "wandb", marker = "extra == 'mlm'" }, - { name = "wandb", marker = "extra == 'training'" }, - { name = "wget", marker = "extra == 'dev'" }, - { name = "zstandard", marker = "extra == 'dev'" }, -] -provides-extras = ["training", "mlm", "dev", "lts", "te", "ssm"] - -[package.metadata.requires-dev] -batch-invariant = [{ name = "deep-gemm", git = "https://github.com/deepseek-ai/DeepGEMM.git?rev=714dd1a4a980f7937a74343d19a8eba4fe321480" }] -build = [ - { name = "cython", specifier = ">=3.0.0" }, - { name = "hatchling" }, - { name = "nvidia-cudnn-frontend", specifier = ">=1.25.0" }, - { name = "nvidia-mathdx" }, - { name = "packaging", specifier = ">=24.2" }, - { name = "pybind11" }, - { name = "setuptools", specifier = ">=80" }, - { name = "torch" }, -] -ci = [ - { name = "pandas" }, - { name = "python-gitlab" }, - { name = "slack-sdk" }, -] -docs = [ - { name = "myst-parser" }, - { name = "nvidia-sphinx-theme" }, - { name = "sphinx" }, - { name = "sphinx-autobuild" }, - { name = "sphinx-autodoc2" }, - { name = "sphinx-copybutton" }, -] -linting = [ - { name = "black", specifier = "==26.3.0" }, - { name = "flake8", specifier = "==7.1.0" }, - { name = "isort", specifier = "==5.13.2" }, - { name = "pylint", specifier = "==3.2.6" }, - { name = "ruff", specifier = "~=0.9.0" }, -] -no-pypi-wheels = [ - { name = "deep-gemm", git = "https://github.com/deepseek-ai/DeepGEMM.git?rev=714dd1a4a980f7937a74343d19a8eba4fe321480" }, - { name = "emerging-optimizers", git = "https://github.com/NVIDIA-NeMo/Emerging-Optimizers.git?rev=v0.2.0" }, - { name = "flash-mla", git = "https://github.com/deepseek-ai/FlashMLA?rev=nv_dev" }, -] -test = [ - { name = "coverage" }, - { name = "mock" }, - { name = "nemo-run", git = "https://github.com/NVIDIA-NeMo/Run.git?rev=e3935393a290aed1822af52139b4b8ee270fed1f" }, - { name = "nltk" }, - { name = "pydantic" }, - { name = "pygithub" }, - { name = "pytest", specifier = "==9.1.1" }, - { name = "pytest-asyncio" }, - { name = "pytest-cov" }, - { name = "pytest-mock" }, - { name = "pytest-random-order" }, - { name = "pyyaml" }, - { name = "tensorboard" }, - { name = "wrapt" }, -] - [[package]] name = "megatron-energon" version = "7.4.0" @@ -4022,106 +3818,6 @@ moe = [ { name = "transformer-engine", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, ] -[package.metadata] -requires-dist = [ - { name = "albumentations", marker = "extra == 'vlm'" }, - { name = "backoff", marker = "extra == 'vlm'" }, - { name = "bitsandbytes", marker = "extra == 'cuda-source'" }, - { name = "causal-conv1d", marker = "extra == 'cuda'" }, - { name = "databricks-sql-connector", marker = "extra == 'delta-databricks'", specifier = ">=3.0.0" }, - { name = "datasets", specifier = ">=4.0.0" }, - { name = "deep-ep", marker = "extra == 'moe'", git = "https://github.com/deepseek-ai/DeepEP.git?rev=7febc6e25660af0f54d95dd781ecdcd62265ecca" }, - { name = "deltalake", marker = "extra == 'delta-databricks'", specifier = ">=1.0.0" }, - { name = "diffusers", marker = "extra == 'diffusion'", specifier = ">=0.37.0" }, - { name = "flash-attn", marker = "extra == 'fa'", specifier = "<=2.8.3" }, - { name = "flash-linear-attention", marker = "extra == 'fla'", specifier = ">=0.4.2" }, - { name = "flashoptim", specifier = ">=0.1.3" }, - { name = "ftfy", marker = "extra == 'diffusion'" }, - { name = "imageio", marker = "extra == 'diffusion'" }, - { name = "imageio-ffmpeg", marker = "extra == 'diffusion'" }, - { name = "kernels", marker = "extra == 'diffusion-kernels'" }, - { name = "mamba-ssm", marker = "extra == 'cuda'" }, - { name = "megatron-fsdp", specifier = ">=0.2.3" }, - { name = "mistral-common", extras = ["audio", "hf-hub", "image", "sentencepiece"] }, - { name = "mistral-common", extras = ["opencv"], marker = "extra == 'vlm'", specifier = ">=1.11.0" }, - { name = "mlflow" }, - { name = "nemo-automodel", extras = ["cuda"], marker = "extra == 'all'" }, - { name = "nemo-automodel", extras = ["cuda"], marker = "extra == 'moe'" }, - { name = "nemo-automodel", extras = ["delta-databricks"], marker = "extra == 'all'" }, - { name = "nemo-automodel", extras = ["diffusion"], marker = "extra == 'all'" }, - { name = "nemo-automodel", extras = ["diffusion"], marker = "extra == 'diffusion-kernels'" }, - { name = "nemo-automodel", extras = ["extra"], marker = "extra == 'all'" }, - { name = "nemo-automodel", extras = ["fla"], marker = "extra == 'all'" }, - { name = "nemo-automodel", extras = ["fla"], marker = "extra == 'moe'" }, - { name = "nemo-automodel", extras = ["vlm"], marker = "extra == 'all'" }, - { name = "numba", marker = "extra == 'vlm'" }, - { name = "numpy", marker = "extra == 'vlm'" }, - { name = "nv-grouped-gemm", marker = "extra == 'cuda'" }, - { name = "onnxscript", marker = "extra == 'cuda'", specifier = ">=0.5.6" }, - { name = "open-clip-torch", marker = "extra == 'vlm'" }, - { name = "opencv-python-headless", specifier = "==4.10.0.84" }, - { name = "opencv-python-headless", marker = "extra == 'diffusion'" }, - { name = "perceptron", marker = "extra == 'extra'" }, - { name = "pillow", marker = "extra == 'vlm'" }, - { name = "pybind11" }, - { name = "pyyaml" }, - { name = "pyyaml", marker = "extra == 'cli'" }, - { name = "qwen-omni-utils", marker = "extra == 'vlm'" }, - { name = "qwen-vl-utils", extras = ["decord"], marker = "platform_machine == 'x86_64' and sys_platform != 'darwin' and extra == 'vlm'" }, - { name = "sentencepiece", marker = "extra == 'extra'" }, - { name = "tiktoken" }, - { name = "timm", marker = "extra == 'vlm'", specifier = "<=1.0.22" }, - { name = "torch", marker = "sys_platform != 'darwin' and sys_platform != 'linux'", specifier = ">=2.6.0", index = "https://download.pytorch.org/whl/cpu" }, - { name = "torch", marker = "sys_platform == 'darwin'", specifier = ">=2.6.0", index = "https://pypi.org/simple" }, - { name = "torch", marker = "sys_platform == 'linux'", specifier = ">=2.6.0", index = "https://download.pytorch.org/whl/cu129" }, - { name = "torchao" }, - { name = "torchcodec", marker = "platform_machine == 'x86_64' and sys_platform != 'darwin' and extra == 'vlm'" }, - { name = "torchdata" }, - { name = "torchvision", marker = "sys_platform == 'darwin' and extra == 'diffusion'", index = "https://pypi.org/simple" }, - { name = "torchvision", marker = "sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'diffusion'", index = "https://download.pytorch.org/whl/cpu" }, - { name = "torchvision", marker = "sys_platform == 'linux' and extra == 'diffusion'", index = "https://download.pytorch.org/whl/cu129" }, - { name = "transformer-engine", extras = ["pytorch"], marker = "extra == 'cuda'", specifier = "<=2.11.0" }, - { name = "transformers", specifier = "==5.5.0" }, - { name = "wandb" }, -] -provides-extras = ["diffusion", "diffusion-kernels", "cuda", "cuda-source", "extra", "fa", "fla", "delta-databricks", "moe", "vlm", "cli", "all"] - -[package.metadata.requires-dev] -build = [ - { name = "packaging" }, - { name = "psutil" }, - { name = "setuptools" }, - { name = "torch", marker = "sys_platform != 'darwin' and sys_platform != 'linux'", specifier = "<=2.10.0", index = "https://download.pytorch.org/whl/cpu" }, - { name = "torch", marker = "sys_platform == 'darwin'", specifier = "<=2.10.0", index = "https://pypi.org/simple" }, - { name = "torch", marker = "sys_platform == 'linux'", specifier = "<=2.10.0", index = "https://download.pytorch.org/whl/cu129" }, -] -dev = [ - { name = "cut-cross-entropy", git = "https://github.com/apple/ml-cross-entropy.git?rev=87a86aba72cfd2f0d8abecaf81c13c4528ea07d8" }, - { name = "dion", marker = "platform_machine == 'x86_64' and sys_platform != 'darwin'", git = "https://github.com/microsoft/dion.git" }, - { name = "liger-kernel", marker = "platform_machine == 'x86_64' and sys_platform != 'darwin'", git = "https://github.com/linkedin/Liger-Kernel.git?rev=1f1a4b8d6a6c3c1ddb3573a46480c41cef25bde6" }, -] -docs = [ - { name = "myst-parser" }, - { name = "nvidia-sphinx-theme" }, - { name = "sphinx" }, - { name = "sphinx-autobuild" }, - { name = "sphinx-autodoc2" }, - { name = "sphinx-copybutton" }, - { name = "sphinx-design" }, -] -linting = [ - { name = "import-linter", specifier = "~=2.4" }, - { name = "pre-commit", specifier = ">=4.2.0" }, - { name = "ruff", specifier = "~=0.9.0" }, - { name = "ty", specifier = ">=0.0.20" }, -] -test = [ - { name = "coverage" }, - { name = "onnxruntime-gpu", marker = "platform_machine == 'x86_64' and sys_platform != 'darwin'", index = "https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-13/pypi/simple/" }, - { name = "peft", specifier = ">=0.18.1" }, - { name = "pytest" }, -] - [[package]] name = "nemo-gym" source = { editable = "3rdparty/Gym-workspace/Gym" } @@ -4134,7 +3830,6 @@ dependencies = [ { name = "fastapi", version = "0.139.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-mcore' or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, { name = "fonttools" }, { name = "gitpython" }, - { name = "gprof2dot" }, { name = "hydra-core", version = "1.3.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, { name = "hydra-core", version = "1.3.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or extra != 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, { name = "itsdangerous" }, @@ -4144,6 +3839,7 @@ dependencies = [ { name = "omegaconf" }, { name = "openai" }, { name = "orjson" }, + { name = "packaging" }, { name = "psutil" }, { name = "pyarrow" }, { name = "pyasn1" }, @@ -4166,9 +3862,11 @@ all = [ { name = "boto3" }, { name = "coverage" }, { name = "daytona" }, + { name = "defusedxml" }, + { name = "gprof2dot" }, + { name = "httpx-aiohttp" }, { name = "mypy" }, { name = "opensandbox" }, - { name = "openshell" }, { name = "pre-commit" }, { name = "pytest" }, { name = "pytest-asyncio" }, @@ -4180,6 +3878,8 @@ all = [ ] dev = [ { name = "coverage" }, + { name = "defusedxml" }, + { name = "gprof2dot" }, { name = "mypy" }, { name = "pre-commit" }, { name = "pytest" }, @@ -4189,11 +3889,14 @@ dev = [ { name = "requests-mock" }, { name = "ruff" }, ] +openshell = [ + { name = "openshell" }, +] sandbox = [ { name = "boto3" }, { name = "daytona" }, + { name = "httpx-aiohttp" }, { name = "opensandbox" }, - { name = "openshell" }, { name = "tenacity" }, ] vllm = [ @@ -4201,58 +3904,11 @@ vllm = [ { name = "vllm", version = "0.24.0", source = { registry = "https://pypi.org/simple" } }, ] -[package.metadata] -requires-dist = [ - { name = "aiohttp", specifier = ">=3.14.1" }, - { name = "anthropic", specifier = "<=0.109.2" }, - { name = "boto3", marker = "extra == 'sandbox'", specifier = ">=1.34" }, - { name = "coverage", extras = ["toml"], marker = "extra == 'dev'" }, - { name = "datasets" }, - { name = "daytona", marker = "extra == 'sandbox'", specifier = ">=0.179.0" }, - { name = "devtools" }, - { name = "fastapi" }, - { name = "flashinfer-python", marker = "extra == 'vllm'", specifier = "==0.6.12" }, - { name = "fonttools", specifier = ">=4.60.2" }, - { name = "gitpython", specifier = ">=3.1.57" }, - { name = "gprof2dot" }, - { name = "hydra-core" }, - { name = "itsdangerous" }, - { name = "mcp", specifier = ">=1.28.1,<2" }, - { name = "mlflow", specifier = ">=3.15.1" }, - { name = "mlflow-skinny", specifier = ">=3.15.1" }, - { name = "mypy", marker = "extra == 'dev'", specifier = ">=1.8.0" }, - { name = "nemo-gym", extras = ["dev", "sandbox"], marker = "extra == 'all'", editable = "3rdparty/Gym-workspace/Gym" }, - { name = "omegaconf" }, - { name = "openai", specifier = "<=2.7.2" }, - { name = "opensandbox", marker = "extra == 'sandbox'", specifier = ">=0.1.15" }, - { name = "openshell", marker = "extra == 'sandbox'", specifier = ">=0.0.92,<0.1" }, - { name = "orjson" }, - { name = "pre-commit", marker = "extra == 'dev'", specifier = ">=3.6.0" }, - { name = "psutil" }, - { name = "pyarrow", specifier = ">=23.0.1" }, - { name = "pyasn1", specifier = ">=0.6.4" }, - { name = "pydantic" }, - { name = "pydantic-core" }, - { name = "pydot" }, - { name = "pytest", marker = "extra == 'dev'" }, - { name = "pytest-asyncio", marker = "extra == 'dev'" }, - { name = "pytest-cov", marker = "extra == 'dev'" }, - { name = "pytest-xdist", marker = "extra == 'dev'" }, - { name = "python-multipart", specifier = ">=0.0.22" }, - { name = "ray", extras = ["default"], specifier = ">=2.56.1" }, - { name = "requests-mock", marker = "extra == 'dev'" }, - { name = "rich" }, - { name = "ruff", marker = "extra == 'dev'" }, - { name = "tenacity", marker = "extra == 'sandbox'", specifier = ">=9.1.4" }, - { name = "tqdm" }, - { name = "urllib3", specifier = ">=2.7.0" }, - { name = "uvicorn" }, - { name = "uvloop" }, - { name = "vllm", marker = "extra == 'vllm'", specifier = "==0.24.0" }, - { name = "wandb" }, - { name = "yappi" }, +[package.dev-dependencies] +codec = [ + { name = "torchaudio" }, + { name = "torchvision" }, ] -provides-extras = ["all", "vllm", "sandbox", "dev"] [[package]] name = "nemo-rl" @@ -4423,156 +4079,6 @@ test = [ { name = "pytest-timeout" }, ] -[package.metadata] -requires-dist = [ - { name = "accelerate", specifier = ">=0.26" }, - { name = "awscrt", specifier = ">=0.35.0" }, - { name = "blobfile" }, - { name = "causal-conv1d", marker = "extra == 'automodel'", git = "https://github.com/Dao-AILab/causal-conv1d?rev=4f6ae4e26ae5fe8af9372f8d312ab25cc4595223" }, - { name = "causal-conv1d", marker = "extra == 'fsdp'", git = "https://github.com/Dao-AILab/causal-conv1d?rev=4f6ae4e26ae5fe8af9372f8d312ab25cc4595223" }, - { name = "causal-conv1d", marker = "extra == 'mcore'", git = "https://github.com/Dao-AILab/causal-conv1d?rev=4f6ae4e26ae5fe8af9372f8d312ab25cc4595223" }, - { name = "colored", specifier = "==2.2.3" }, - { name = "cuda-bindings", marker = "sys_platform != 'darwin'" }, - { name = "cuda-python", marker = "extra == 'vllm'" }, - { name = "cupy-cuda13x", marker = "extra == 'mcore'" }, - { name = "datasets", specifier = ">=4.0.0" }, - { name = "debugpy" }, - { name = "deep-ep", marker = "platform_machine == 'aarch64' and extra == 'automodel'", git = "https://github.com/deepseek-ai/DeepEP.git?rev=a48493600c4886c1b297aaa78db0e1ebc2d8dd6c" }, - { name = "deep-ep", marker = "platform_machine == 'aarch64' and extra == 'mcore'", git = "https://github.com/deepseek-ai/DeepEP.git?rev=a48493600c4886c1b297aaa78db0e1ebc2d8dd6c" }, - { name = "deep-ep", marker = "platform_machine == 'aarch64' and extra == 'vllm'", git = "https://github.com/deepseek-ai/DeepEP.git?rev=a48493600c4886c1b297aaa78db0e1ebc2d8dd6c" }, - { name = "deep-ep", marker = "platform_machine == 'x86_64' and extra == 'automodel'", git = "https://github.com/deepseek-ai/DeepEP.git?rev=29d31c095796f3c8ece47ee9cdcc167051bbeed9" }, - { name = "deep-ep", marker = "platform_machine == 'x86_64' and extra == 'mcore'", git = "https://github.com/deepseek-ai/DeepEP.git?rev=29d31c095796f3c8ece47ee9cdcc167051bbeed9" }, - { name = "deep-ep", marker = "platform_machine == 'x86_64' and extra == 'vllm'", git = "https://github.com/deepseek-ai/DeepEP.git?rev=29d31c095796f3c8ece47ee9cdcc167051bbeed9" }, - { name = "deep-gemm", marker = "extra == 'vllm'", git = "https://github.com/deepseek-ai/DeepGEMM.git?rev=67fc64863d43521080bf2005e6528d0fceee9510" }, - { name = "fastokens-b10", specifier = ">=0.1.1" }, - { name = "flash-attn", marker = "platform_machine == 'aarch64' and sys_platform == 'linux' and extra == 'automodel'", url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_aarch64.whl" }, - { name = "flash-attn", marker = "platform_machine == 'aarch64' and sys_platform == 'linux' and extra == 'fsdp'", url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_aarch64.whl" }, - { name = "flash-attn", marker = "platform_machine == 'aarch64' and sys_platform == 'linux' and extra == 'mcore'", url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_aarch64.whl" }, - { name = "flash-attn", marker = "platform_machine == 'x86_64' and sys_platform == 'linux' and extra == 'automodel'", url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_x86_64.whl" }, - { name = "flash-attn", marker = "platform_machine == 'x86_64' and sys_platform == 'linux' and extra == 'fsdp'", url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_x86_64.whl" }, - { name = "flash-attn", marker = "platform_machine == 'x86_64' and sys_platform == 'linux' and extra == 'mcore'", url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_x86_64.whl" }, - { name = "flash-attn", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'automodel') or (sys_platform != 'linux' and extra == 'automodel')", specifier = "==2.8.1" }, - { name = "flash-attn", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'fsdp') or (sys_platform != 'linux' and extra == 'fsdp')", specifier = "==2.8.1" }, - { name = "flash-attn", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'mcore') or (sys_platform != 'linux' and extra == 'mcore')", specifier = "==2.8.1" }, - { name = "flashinfer-cubin", marker = "extra == 'mcore'", specifier = "==0.6.8.post1" }, - { name = "flashinfer-cubin", marker = "extra == 'sglang'", specifier = "==0.6.11.post1" }, - { name = "flashinfer-cubin", marker = "extra == 'vllm'", specifier = "==0.6.13" }, - { name = "flashinfer-jit-cache", marker = "extra == 'mcore'", specifier = "==0.6.8.post1", index = "https://flashinfer.ai/whl/cu130" }, - { name = "flashinfer-jit-cache", marker = "extra == 'sglang'", specifier = "==0.6.11.post1", index = "https://flashinfer.ai/whl/cu130" }, - { name = "flashinfer-jit-cache", marker = "extra == 'vllm'", specifier = "==0.6.13", index = "https://flashinfer.ai/whl/cu130" }, - { name = "flashinfer-python", marker = "extra == 'mcore'", specifier = "==0.6.8.post1" }, - { name = "flashinfer-python", marker = "extra == 'sglang'", specifier = "==0.6.11.post1" }, - { name = "flashinfer-python", marker = "extra == 'vllm'", specifier = "==0.6.13" }, - { name = "hydra-core" }, - { name = "kernels", marker = "extra == 'sglang'", specifier = ">=0.12.0,<0.13" }, - { name = "mamba-ssm", marker = "extra == 'automodel'", git = "https://github.com/state-spaces/mamba.git?rev=a14b1dff0454a3bc27d9eb31355dc01e4b2490ec" }, - { name = "mamba-ssm", marker = "extra == 'fsdp'", git = "https://github.com/state-spaces/mamba.git?rev=a14b1dff0454a3bc27d9eb31355dc01e4b2490ec" }, - { name = "mamba-ssm", marker = "extra == 'mcore'", git = "https://github.com/state-spaces/mamba.git?rev=a14b1dff0454a3bc27d9eb31355dc01e4b2490ec" }, - { name = "math-verify" }, - { name = "matplotlib" }, - { name = "megatron-bridge", extras = ["ssm", "te"], marker = "extra == 'mcore'", editable = "3rdparty/Megatron-Bridge-workspace/Megatron-Bridge" }, - { name = "mistral-common", marker = "extra == 'automodel'", specifier = ">=1.11.0" }, - { name = "mlflow", specifier = ">=3.13.0" }, - { name = "mooncake-transfer-engine-cuda13", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')", specifier = "==0.3.11.post1" }, - { name = "nccl4py", marker = "sys_platform != 'darwin'" }, - { name = "nemo-automodel", extras = ["moe"], marker = "extra == 'automodel'", editable = "3rdparty/Automodel-workspace/Automodel" }, - { name = "nemo-gym", marker = "extra == 'nemo-gym'", editable = "3rdparty/Gym-workspace/Gym" }, - { name = "ninja" }, - { name = "nixl", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')", specifier = "==1.3.0" }, - { name = "num2words", specifier = ">=0.5.14" }, - { name = "num2words", marker = "extra == 'vllm'", specifier = ">=0.5.14" }, - { name = "numpy" }, - { name = "nv-grouped-gemm", marker = "extra == 'automodel'", git = "https://github.com/fanshiqing/grouped_gemm?tag=v1.1.4.post7" }, - { name = "nvidia-cudnn-cu13", marker = "sys_platform != 'darwin'", specifier = "==9.20.0.48" }, - { name = "nvidia-cutlass-dsl", extras = ["cu13"], marker = "extra == 'vllm'", specifier = "==4.5.2" }, - { name = "nvidia-ml-py" }, - { name = "nvidia-modelopt", marker = "extra == 'modelopt'", git = "https://github.com/NVIDIA/Model-Optimizer?rev=c3b913b9cc1d82d5a0af9fa77b4db87829e6f158" }, - { name = "nvidia-nvshmem-cu13", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')", specifier = ">=3.6.5" }, - { name = "nvidia-resiliency-ext", marker = "extra == 'nvrx'", git = "https://github.com/NVIDIA/nvidia-resiliency-ext.git?tag=v0.6.0" }, - { name = "nvshmem4py-cu13", marker = "extra == 'mcore'", specifier = ">=0.2.1" }, - { name = "nvtx" }, - { name = "omegaconf" }, - { name = "open-clip-torch", specifier = ">=3.2.0" }, - { name = "pillow", specifier = ">=12.3.0" }, - { name = "pip" }, - { name = "plotly" }, - { name = "pybase64" }, - { name = "pyzmq" }, - { name = "ray", extras = ["default"], specifier = ">=2.55.1" }, - { name = "rich" }, - { name = "sentencepiece" }, - { name = "setuptools" }, - { name = "sglang", marker = "extra == 'sglang'", specifier = "==0.5.12.post1" }, - { name = "sglang-kernel", marker = "extra == 'sglang'", specifier = "==0.4.2.post2" }, - { name = "sglang-router", marker = "extra == 'sglang'" }, - { name = "soundfile", specifier = ">=0.13.1" }, - { name = "swanlab" }, - { name = "sympy", specifier = ">=1.14.0" }, - { name = "tensorboard" }, - { name = "tensordict" }, - { name = "tensorrt-llm", marker = "extra == 'trtllm'", directory = "3rdparty/TensorRT-LLM-workspace" }, - { name = "tiktoken" }, - { name = "tilelang", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, - { name = "timm" }, - { name = "torch", marker = "sys_platform != 'darwin'", specifier = "==2.11.0", index = "https://download.pytorch.org/whl/cu130" }, - { name = "torch", marker = "sys_platform == 'darwin'", specifier = "==2.11.0", index = "https://pypi.org/simple" }, - { name = "torchdata" }, - { name = "torchvision", marker = "sys_platform != 'darwin'", specifier = "==0.26.0", index = "https://download.pytorch.org/whl/cu130" }, - { name = "torchvision", marker = "sys_platform == 'darwin'", specifier = "==0.26.0", index = "https://pypi.org/simple" }, - { name = "transferqueue", git = "https://github.com/Ascend/TransferQueue.git?rev=c51614308b68c8d7a87c9b3ef62d59e14c69bde2" }, - { name = "transformer-engine", extras = ["core-cu13", "pytorch"], marker = "extra == 'automodel'", git = "https://github.com/NVIDIA/TransformerEngine.git?rev=v2.14.1" }, - { name = "transformers", specifier = ">=5.5.0,<5.9.0" }, - { name = "transformers", marker = "extra == 'automodel'", specifier = ">=5.5.0,<5.6.0" }, - { name = "transformers", marker = "extra == 'sglang'", specifier = "==5.6.0" }, - { name = "triton", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')", index = "https://download.pytorch.org/whl/cu130" }, - { name = "vllm", marker = "platform_machine == 'aarch64' and sys_platform == 'linux' and extra == 'vllm'", url = "https://github.com/vllm-project/vllm/releases/download/v0.25.1/vllm-0.25.1-cp38-abi3-manylinux_2_28_aarch64.whl" }, - { name = "vllm", marker = "platform_machine == 'x86_64' and sys_platform == 'linux' and extra == 'vllm'", url = "https://github.com/vllm-project/vllm/releases/download/v0.25.1/vllm-0.25.1-cp38-abi3-manylinux_2_28_x86_64.whl" }, - { name = "vllm", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'vllm') or (sys_platform != 'linux' and extra == 'vllm')", specifier = "==0.25.1" }, - { name = "wandb", specifier = ">=0.28.0" }, - { name = "zstandard" }, -] -provides-extras = ["fsdp", "automodel", "vllm", "sglang", "mcore", "trtllm", "modelopt", "nvrx", "nemo-gym"] - -[package.metadata.requires-dev] -build = [ - { name = "einops" }, - { name = "hatchling" }, - { name = "packaging" }, - { name = "psutil" }, - { name = "pybind11" }, - { name = "setuptools" }, - { name = "torch", marker = "sys_platform != 'darwin'", specifier = "==2.11.0", index = "https://download.pytorch.org/whl/cu130" }, - { name = "torch", marker = "sys_platform == 'darwin'", specifier = "==2.11.0", index = "https://pypi.org/simple" }, -] -dev = [ - { name = "pre-commit", specifier = ">=4.2.0" }, - { name = "pyrefly", specifier = "==0.24.2" }, - { name = "ruff", specifier = "==0.9.9" }, - { name = "types-pyyaml" }, - { name = "types-requests" }, -] -docs = [ - { name = "gitpython", specifier = ">=3.1.45" }, - { name = "myst-parser" }, - { name = "nvidia-sphinx-theme" }, - { name = "python-dotenv" }, - { name = "sphinx" }, - { name = "sphinx-autobuild" }, - { name = "sphinx-autodoc2" }, - { name = "sphinx-copybutton" }, - { name = "sphinx-design" }, - { name = "sphinxcontrib-mermaid" }, - { name = "swagger-plugin-for-sphinx" }, -] -test = [ - { name = "pytest", specifier = ">=8.4.2" }, - { name = "pytest-asyncio" }, - { name = "pytest-cov" }, - { name = "pytest-shard" }, - { name = "pytest-testmon" }, - { name = "pytest-timeout" }, -] - [[package]] name = "networkx" version = "3.6.1" @@ -7431,16 +6937,6 @@ test = [ { name = "pytest-timeout" }, ] -[package.metadata] -requires-dist = [{ name = "nemo-rl", editable = "." }] - -[package.metadata.requires-dev] -test = [ - { name = "pytest", specifier = ">=7.0.0" }, - { name = "pytest-cov" }, - { name = "pytest-timeout" }, -] - [[package]] name = "tenacity" version = "9.1.4" @@ -7613,85 +7109,6 @@ dependencies = [ { name = "xgrammar" }, ] -[package.metadata] -requires-dist = [ - { name = "accelerate", specifier = ">=1.7.0" }, - { name = "aenum" }, - { name = "apache-tvm-ffi", specifier = "==0.1.6" }, - { name = "backoff" }, - { name = "blake3" }, - { name = "blobfile" }, - { name = "cache-dit", specifier = ">=1.3.5" }, - { name = "click", specifier = ">=8.3.1" }, - { name = "click-option-group" }, - { name = "colored" }, - { name = "cuda-python", specifier = ">=13" }, - { name = "cuda-tile", specifier = ">=1.0.1" }, - { name = "diffusers", specifier = ">=0.37.1" }, - { name = "einops" }, - { name = "etcd-sdk-python", specifier = "==0.0.7" }, - { name = "fastapi", specifier = ">=0.120.1" }, - { name = "flash-attn-4", specifier = "==4.0.0b11" }, - { name = "flashinfer-python", specifier = "==0.6.12" }, - { name = "ftfy" }, - { name = "h5py", specifier = "==3.12.1" }, - { name = "jsonschema" }, - { name = "lark" }, - { name = "librosa" }, - { name = "llguidance", specifier = ">=1.3.0,<1.4.0" }, - { name = "llist" }, - { name = "matplotlib" }, - { name = "mistral-common", specifier = ">=1.10.0" }, - { name = "mpi4py" }, - { name = "mpmath", specifier = ">=1.3.0" }, - { name = "msgpack" }, - { name = "numexpr" }, - { name = "numpy", specifier = ">=2.0.0,<2.4" }, - { name = "nvidia-cuda-nvrtc" }, - { name = "nvidia-cuda-tileiras", specifier = ">=13.1,<13.2" }, - { name = "nvidia-cutlass-dsl", extras = ["cu13"], specifier = "==4.5.0" }, - { name = "nvidia-ml-py", specifier = ">=13" }, - { name = "nvidia-modelopt", specifier = ">=0.37.0" }, - { name = "nvidia-nccl-cu13", specifier = ">=2.28.9,<=2.30.7" }, - { name = "nvtx" }, - { name = "omegaconf" }, - { name = "onnx", specifier = ">=1.21.0" }, - { name = "onnx-graphsurgeon", specifier = ">=0.5.2" }, - { name = "openai" }, - { name = "openai-harmony", specifier = "==0.0.4" }, - { name = "opencv-python-headless" }, - { name = "optimum" }, - { name = "ordered-set" }, - { name = "pandas" }, - { name = "partial-json-parser" }, - { name = "peft", specifier = ">=0.18.1" }, - { name = "pillow" }, - { name = "plotly" }, - { name = "prometheus-client" }, - { name = "prometheus-fastapi-instrumentator", specifier = ">=8.0.2" }, - { name = "psutil" }, - { name = "pulp" }, - { name = "pydantic", specifier = ">=2.9.1" }, - { name = "pydantic-settings", extras = ["yaml"] }, - { name = "python-multipart" }, - { name = "pyzmq" }, - { name = "sentencepiece", specifier = ">=0.1.99" }, - { name = "smg-grpc-proto", specifier = ">=0.4.2" }, - { name = "soundfile" }, - { name = "starlette", specifier = ">=0.49.1" }, - { name = "strenum" }, - { name = "tensorrt", specifier = "~=10.16.1" }, - { name = "tiktoken" }, - { name = "torch", specifier = ">=2.11.0" }, - { name = "torch-c-dlpack-ext", specifier = "==0.1.3" }, - { name = "torchao", specifier = ">=0.14.1,<0.16.0" }, - { name = "torchvision" }, - { name = "transformers", specifier = "==5.5.4" }, - { name = "uvicorn" }, - { name = "xdsl", specifier = ">=0.59.0" }, - { name = "xgrammar", specifier = ">=0.1.32" }, -] - [[package]] name = "tensorstore" version = "0.1.84" @@ -8516,109 +7933,6 @@ wheels = [ { url = "https://github.com/vllm-project/vllm/releases/download/v0.25.1/vllm-0.25.1-cp38-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:902be760af4c5ebfad8af5b8ea07a53ae14e5a6c839c8ab56da30581abb75ad2" }, ] -[package.metadata] -requires-dist = [ - { name = "aiohttp", specifier = ">=3.13.3" }, - { name = "anthropic", specifier = ">=0.71.0" }, - { name = "apache-tvm-ffi", specifier = "==0.1.9" }, - { name = "av", marker = "extra == 'audio'" }, - { name = "blake3" }, - { name = "cachetools" }, - { name = "cbor2" }, - { name = "cloudpickle" }, - { name = "compressed-tensors", specifier = "==0.17.0" }, - { name = "datasets", marker = "extra == 'bench'" }, - { name = "depyf", specifier = "==0.20.0" }, - { name = "diskcache", specifier = "==5.6.3" }, - { name = "einops" }, - { name = "fastapi", extras = ["standard"], specifier = ">=0.133.0,<0.137.0" }, - { name = "fastsafetensors", specifier = ">=0.3.2" }, - { name = "fastsafetensors", marker = "extra == 'fastsafetensors'", specifier = ">=0.3.2" }, - { name = "filelock", specifier = ">=3.16.1" }, - { name = "flashinfer-cubin", specifier = "==0.6.13" }, - { name = "flashinfer-python", specifier = "==0.6.13" }, - { name = "helion", marker = "extra == 'helion'", specifier = "==1.1.0" }, - { name = "humming-kernels", extras = ["cu13"], specifier = "==0.1.10" }, - { name = "ijson" }, - { name = "instanttensor", marker = "extra == 'instanttensor'", specifier = ">=0.1.5" }, - { name = "jsonschema", specifier = ">=4.23.0" }, - { name = "lark", specifier = "==1.2.2" }, - { name = "llguidance", marker = "platform_machine == 'aarch64' or platform_machine == 'arm64' or platform_machine == 'ppc64le' or platform_machine == 'x86_64'", specifier = ">=1.7.0,<1.8.0" }, - { name = "lm-format-enforcer", specifier = "==0.11.3" }, - { name = "matplotlib", marker = "extra == 'bench'" }, - { name = "mcp" }, - { name = "mistral-common", extras = ["audio"], marker = "extra == 'audio'" }, - { name = "mistral-common", extras = ["image"], specifier = ">=1.11.5" }, - { name = "model-hosting-container-standards", specifier = ">=0.1.14,<1.0.0" }, - { name = "msgspec" }, - { name = "ninja" }, - { name = "numba", specifier = "==0.65.0" }, - { name = "numpy" }, - { name = "nvidia-cudnn-frontend", specifier = ">=1.19.1" }, - { name = "nvidia-cutlass-dsl", extras = ["cu13"], specifier = "==4.5.2" }, - { name = "nvtx", specifier = "==0.2.15" }, - { name = "openai", specifier = ">=2.0.0" }, - { name = "openai-harmony", specifier = ">=0.0.3" }, - { name = "opencv-python-headless", specifier = ">=4.13.0" }, - { name = "opentelemetry-api", specifier = ">=1.27.0" }, - { name = "opentelemetry-api", marker = "extra == 'otel'", specifier = ">=1.26.0" }, - { name = "opentelemetry-exporter-otlp", specifier = ">=1.27.0" }, - { name = "opentelemetry-exporter-otlp", marker = "extra == 'otel'", specifier = ">=1.26.0" }, - { name = "opentelemetry-sdk", specifier = ">=1.27.0" }, - { name = "opentelemetry-sdk", marker = "extra == 'otel'", specifier = ">=1.26.0" }, - { name = "opentelemetry-semantic-conventions-ai", specifier = ">=0.4.1" }, - { name = "opentelemetry-semantic-conventions-ai", marker = "extra == 'otel'", specifier = ">=0.4.1" }, - { name = "outlines-core", specifier = "==0.2.14" }, - { name = "pandas", marker = "extra == 'bench'" }, - { name = "partial-json-parser" }, - { name = "pillow" }, - { name = "plotly", marker = "extra == 'bench'" }, - { name = "prometheus-client", specifier = ">=0.18.0" }, - { name = "prometheus-fastapi-instrumentator", specifier = ">=8.0.0" }, - { name = "protobuf", specifier = ">=5.29.6,!=6.30.*,!=6.31.*,!=6.32.*,!=6.33.0.*,!=6.33.1.*,!=6.33.2.*,!=6.33.3.*,!=6.33.4.*" }, - { name = "psutil" }, - { name = "py-cpuinfo" }, - { name = "pybase64" }, - { name = "pydantic", specifier = ">=2.12.0" }, - { name = "pynvvideocodec", specifier = "==2.0.4" }, - { name = "python-json-logger" }, - { name = "pyyaml" }, - { name = "pyzmq", specifier = ">=25.0.0" }, - { name = "quack-kernels", specifier = ">=0.3.3" }, - { name = "regex" }, - { name = "requests", specifier = ">=2.26.0" }, - { name = "runai-model-streamer", extras = ["azure", "gcs", "s3"], marker = "extra == 'runai'", specifier = ">=0.15.7" }, - { name = "safetensors", specifier = ">=0.6.2" }, - { name = "scipy", marker = "extra == 'audio'" }, - { name = "scipy", marker = "extra == 'bench'" }, - { name = "seaborn", marker = "extra == 'bench'" }, - { name = "sentencepiece" }, - { name = "setproctitle" }, - { name = "setuptools", marker = "python_full_version >= '3.12'", specifier = ">=77.0.3,<81.0.0" }, - { name = "six", marker = "python_full_version >= '3.12'", specifier = ">=1.16.0" }, - { name = "smg-grpc-servicer", extras = ["vllm"], marker = "extra == 'grpc'", specifier = ">=0.5.2" }, - { name = "soundfile", marker = "extra == 'audio'" }, - { name = "soxr", marker = "extra == 'audio'" }, - { name = "starlette", specifier = ">=1.0.1" }, - { name = "tensorizer", marker = "extra == 'tensorizer'", specifier = "==2.10.1" }, - { name = "tiktoken", specifier = ">=0.6.0" }, - { name = "tilelang", specifier = "==0.1.9" }, - { name = "tokenizers", specifier = ">=0.21.1" }, - { name = "tokenspeed-mla", marker = "sys_platform == 'linux'", specifier = "==0.1.2" }, - { name = "torch", specifier = "==2.11.0" }, - { name = "torchaudio", specifier = "==2.11.0" }, - { name = "torchcodec", specifier = ">=0.14" }, - { name = "torchvision", specifier = "==0.26.0" }, - { name = "tqdm" }, - { name = "transformers", specifier = ">=5.5.3" }, - { name = "typing-extensions", specifier = ">=4.10" }, - { name = "vllm-gguf-plugin", marker = "extra == 'extra-quant'", specifier = ">=0.0.2" }, - { name = "watchfiles" }, - { name = "xgrammar", marker = "platform_machine == 'aarch64' or platform_machine == 'arm64' or platform_machine == 'ppc64le' or platform_machine == 's390x' or platform_machine == 'x86_64'", specifier = ">=0.2.1,<1.0.0" }, - { name = "zentorch", marker = "extra == 'zen'", specifier = "==2.11.0.0" }, -] -provides-extras = ["zen", "bench", "tensorizer", "fastsafetensors", "instanttensor", "runai", "audio", "video", "flashinfer", "helion", "grpc", "otel", "extra-quant"] - [[package]] name = "vllm" version = "0.25.1" @@ -8706,109 +8020,6 @@ wheels = [ { url = "https://github.com/vllm-project/vllm/releases/download/v0.25.1/vllm-0.25.1-cp38-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:16fc7a28df1576eb6f7ca0455026551b8f9adb674c19c66059359ef3e964bd1e" }, ] -[package.metadata] -requires-dist = [ - { name = "aiohttp", specifier = ">=3.13.3" }, - { name = "anthropic", specifier = ">=0.71.0" }, - { name = "apache-tvm-ffi", specifier = "==0.1.9" }, - { name = "av", marker = "extra == 'audio'" }, - { name = "blake3" }, - { name = "cachetools" }, - { name = "cbor2" }, - { name = "cloudpickle" }, - { name = "compressed-tensors", specifier = "==0.17.0" }, - { name = "datasets", marker = "extra == 'bench'" }, - { name = "depyf", specifier = "==0.20.0" }, - { name = "diskcache", specifier = "==5.6.3" }, - { name = "einops" }, - { name = "fastapi", extras = ["standard"], specifier = ">=0.133.0,<0.137.0" }, - { name = "fastsafetensors", specifier = ">=0.3.2" }, - { name = "fastsafetensors", marker = "extra == 'fastsafetensors'", specifier = ">=0.3.2" }, - { name = "filelock", specifier = ">=3.16.1" }, - { name = "flashinfer-cubin", specifier = "==0.6.13" }, - { name = "flashinfer-python", specifier = "==0.6.13" }, - { name = "helion", marker = "extra == 'helion'", specifier = "==1.1.0" }, - { name = "humming-kernels", extras = ["cu13"], specifier = "==0.1.10" }, - { name = "ijson" }, - { name = "instanttensor", marker = "extra == 'instanttensor'", specifier = ">=0.1.5" }, - { name = "jsonschema", specifier = ">=4.23.0" }, - { name = "lark", specifier = "==1.2.2" }, - { name = "llguidance", marker = "platform_machine == 'aarch64' or platform_machine == 'arm64' or platform_machine == 'ppc64le' or platform_machine == 'x86_64'", specifier = ">=1.7.0,<1.8.0" }, - { name = "lm-format-enforcer", specifier = "==0.11.3" }, - { name = "matplotlib", marker = "extra == 'bench'" }, - { name = "mcp" }, - { name = "mistral-common", extras = ["audio"], marker = "extra == 'audio'" }, - { name = "mistral-common", extras = ["image"], specifier = ">=1.11.5" }, - { name = "model-hosting-container-standards", specifier = ">=0.1.14,<1.0.0" }, - { name = "msgspec" }, - { name = "ninja" }, - { name = "numba", specifier = "==0.65.0" }, - { name = "numpy" }, - { name = "nvidia-cudnn-frontend", specifier = ">=1.19.1" }, - { name = "nvidia-cutlass-dsl", extras = ["cu13"], specifier = "==4.5.2" }, - { name = "nvtx", specifier = "==0.2.15" }, - { name = "openai", specifier = ">=2.0.0" }, - { name = "openai-harmony", specifier = ">=0.0.3" }, - { name = "opencv-python-headless", specifier = ">=4.13.0" }, - { name = "opentelemetry-api", specifier = ">=1.27.0" }, - { name = "opentelemetry-api", marker = "extra == 'otel'", specifier = ">=1.26.0" }, - { name = "opentelemetry-exporter-otlp", specifier = ">=1.27.0" }, - { name = "opentelemetry-exporter-otlp", marker = "extra == 'otel'", specifier = ">=1.26.0" }, - { name = "opentelemetry-sdk", specifier = ">=1.27.0" }, - { name = "opentelemetry-sdk", marker = "extra == 'otel'", specifier = ">=1.26.0" }, - { name = "opentelemetry-semantic-conventions-ai", specifier = ">=0.4.1" }, - { name = "opentelemetry-semantic-conventions-ai", marker = "extra == 'otel'", specifier = ">=0.4.1" }, - { name = "outlines-core", specifier = "==0.2.14" }, - { name = "pandas", marker = "extra == 'bench'" }, - { name = "partial-json-parser" }, - { name = "pillow" }, - { name = "plotly", marker = "extra == 'bench'" }, - { name = "prometheus-client", specifier = ">=0.18.0" }, - { name = "prometheus-fastapi-instrumentator", specifier = ">=8.0.0" }, - { name = "protobuf", specifier = ">=5.29.6,!=6.30.*,!=6.31.*,!=6.32.*,!=6.33.0.*,!=6.33.1.*,!=6.33.2.*,!=6.33.3.*,!=6.33.4.*" }, - { name = "psutil" }, - { name = "py-cpuinfo" }, - { name = "pybase64" }, - { name = "pydantic", specifier = ">=2.12.0" }, - { name = "pynvvideocodec", specifier = "==2.0.4" }, - { name = "python-json-logger" }, - { name = "pyyaml" }, - { name = "pyzmq", specifier = ">=25.0.0" }, - { name = "quack-kernels", specifier = ">=0.3.3" }, - { name = "regex" }, - { name = "requests", specifier = ">=2.26.0" }, - { name = "runai-model-streamer", extras = ["azure", "gcs", "s3"], marker = "extra == 'runai'", specifier = ">=0.15.7" }, - { name = "safetensors", specifier = ">=0.6.2" }, - { name = "scipy", marker = "extra == 'audio'" }, - { name = "scipy", marker = "extra == 'bench'" }, - { name = "seaborn", marker = "extra == 'bench'" }, - { name = "sentencepiece" }, - { name = "setproctitle" }, - { name = "setuptools", marker = "python_full_version >= '3.12'", specifier = ">=77.0.3,<81.0.0" }, - { name = "six", marker = "python_full_version >= '3.12'", specifier = ">=1.16.0" }, - { name = "smg-grpc-servicer", extras = ["vllm"], marker = "extra == 'grpc'", specifier = ">=0.5.2" }, - { name = "soundfile", marker = "extra == 'audio'" }, - { name = "soxr", marker = "extra == 'audio'" }, - { name = "starlette", specifier = ">=1.0.1" }, - { name = "tensorizer", marker = "extra == 'tensorizer'", specifier = "==2.10.1" }, - { name = "tiktoken", specifier = ">=0.6.0" }, - { name = "tilelang", specifier = "==0.1.9" }, - { name = "tokenizers", specifier = ">=0.21.1" }, - { name = "tokenspeed-mla", marker = "sys_platform == 'linux'", specifier = "==0.1.2" }, - { name = "torch", specifier = "==2.11.0" }, - { name = "torchaudio", specifier = "==2.11.0" }, - { name = "torchcodec", specifier = ">=0.14" }, - { name = "torchvision", specifier = "==0.26.0" }, - { name = "tqdm" }, - { name = "transformers", specifier = ">=5.5.3" }, - { name = "typing-extensions", specifier = ">=4.10" }, - { name = "vllm-gguf-plugin", marker = "extra == 'extra-quant'", specifier = ">=0.0.2" }, - { name = "watchfiles" }, - { name = "xgrammar", marker = "platform_machine == 'aarch64' or platform_machine == 'arm64' or platform_machine == 'ppc64le' or platform_machine == 's390x' or platform_machine == 'x86_64'", specifier = ">=0.2.1,<1.0.0" }, - { name = "zentorch", marker = "extra == 'zen'", specifier = "==2.11.0.0" }, -] -provides-extras = ["zen", "bench", "tensorizer", "fastsafetensors", "instanttensor", "runai", "audio", "video", "flashinfer", "helion", "grpc", "otel", "extra-quant"] - [[package]] name = "wandb" version = "0.28.1" From 4a42db78c1ce98e8cef22376101fcb1fea267cb6 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 19 Aug 2026 06:52:29 -0700 Subject: [PATCH 29/52] fix(launcher): mount locked local uv sources Signed-off-by: Pranav Thombre --- ultra_launch.sh | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/ultra_launch.sh b/ultra_launch.sh index fae8bcaf145..a313cdbfbde 100755 --- a/ultra_launch.sh +++ b/ultra_launch.sh @@ -562,9 +562,10 @@ fi # ============================================================================= # Container mounts # ============================================================================= -# By default, the project metadata/lock, nemo_rl (Python package), and -# examples/configs (YAML configs) from the code snapshot are overlaid into the -# container. Everything else uses the container's built-in code at /opt/nemo-rl. +# By default, the project metadata/lock, nemo_rl (Python package), +# examples/configs (YAML configs), and local uv sources/workspace members from +# the code snapshot are overlaid into the container. Keeping the project files +# and their local sources together is required for `uv sync --locked`. # # To overlay additional components (e.g. a local Megatron-LM checkout), pass # EXTRA_MOUNTS as a comma-separated list of host:container pairs: @@ -601,10 +602,19 @@ if [[ -d "${OVERLAY_SOURCE}/examples/configs" ]]; then _append_mount "${OVERLAY_SOURCE}/examples/configs:/opt/nemo-rl/examples/configs" echo " Mount: configs → /opt/nemo-rl/examples/configs" fi -if [[ -d "${OVERLAY_SOURCE}/3rdparty/Gym-workspace/Gym" ]]; then - _append_mount "${OVERLAY_SOURCE}/3rdparty/Gym-workspace/Gym:/opt/nemo-rl/3rdparty/Gym-workspace/Gym" - echo " Mount: Gym → /opt/nemo-rl/3rdparty/Gym-workspace/Gym" -fi +_local_project_paths=( + "3rdparty/TensorRT-LLM-workspace" + "3rdparty/Automodel-workspace/Automodel" + "3rdparty/Megatron-Bridge-workspace/Megatron-Bridge" + "3rdparty/Gym-workspace/Gym" + "research/template_project" +) +for _local_project_path in "${_local_project_paths[@]}"; do + if [[ -d "${OVERLAY_SOURCE}/${_local_project_path}" ]]; then + _append_mount "${OVERLAY_SOURCE}/${_local_project_path}:/opt/nemo-rl/${_local_project_path}" + echo " Mount: ${_local_project_path} → /opt/nemo-rl/${_local_project_path}" + fi +done if [[ "${USE_SNAPSHOT}" == "1" ]]; then _append_mount "${SNAPSHOT_DIR}:${SNAPSHOT_DIR}" From c9602d44750a76f792888077ded996a54831f63e Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Wed, 19 Aug 2026 14:39:49 -0700 Subject: [PATCH 30/52] fix(build): isolate Gym server extras from RL workspace --- 3rdparty/Gym-workspace/Gym | 2 +- pyproject.toml | 27 +- uv.lock | 984 +++++++------------------------------ 3 files changed, 181 insertions(+), 832 deletions(-) diff --git a/3rdparty/Gym-workspace/Gym b/3rdparty/Gym-workspace/Gym index 1b34208461e..10b3490861d 160000 --- a/3rdparty/Gym-workspace/Gym +++ b/3rdparty/Gym-workspace/Gym @@ -1 +1 @@ -Subproject commit 1b34208461eca1451477005cd94d9190bfd87275 +Subproject commit 10b3490861d66c71bd1b549222b24fbdc58bcb9e diff --git a/pyproject.toml b/pyproject.toml index f290d42d8e5..8a08cb91190 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -265,7 +265,11 @@ test = [ tensorrt-llm = { path = "3rdparty/TensorRT-LLM-workspace" } nemo-automodel = { path = "3rdparty/Automodel-workspace/Automodel", editable = true } megatron-bridge = { path = "3rdparty/Megatron-Bridge-workspace/Megatron-Bridge", editable = true } -nemo_gym = { workspace = true } +# Keep Gym as an editable path dependency rather than a workspace member. Both +# projects define a ``vllm`` extra with intentionally different server stacks; +# uv applies a requested extra name to every selected workspace member, which +# makes the RL vLLM + base Gym worker environment impossible to select. +nemo_gym = { path = "3rdparty/Gym-workspace/Gym", editable = true } nemo_run = { git = "https://github.com/NVIDIA-NeMo/Run", rev = "414f0077c648fde2c71bb1186e97ccbf96d6844c" } # torch/torchvision/triton all come from the torch index in order to pick up aarch64 wheels torch = [ @@ -289,7 +293,6 @@ flashinfer-jit-cache = { index = "flashinfer-cu130" } [tool.uv.workspace] members = [ - "3rdparty/Gym-workspace/Gym", # Research projects are also added here in order for them to share the global root level uv.lock. # If we don't do this, the research projects do not see the global uv.lock, and may mistakenly # install numpy>=2.0 because nemo-rl's core [dependencies] do not pin numpy, but when you inspect @@ -442,26 +445,6 @@ constraint-dependencies = [ exclude-dependencies = ["nvidia-cutlass-dsl-libs-base"] conflicts = [ - [ - { package = "nemo-rl", extra = "automodel" }, - { package = "nemo-gym", extra = "vllm" }, - ], - [ - { package = "nemo-rl", extra = "vllm" }, - { package = "nemo-gym", extra = "vllm" }, - ], - [ - { package = "nemo-rl", extra = "sglang" }, - { package = "nemo-gym", extra = "vllm" }, - ], - [ - { package = "nemo-rl", extra = "mcore" }, - { package = "nemo-gym", extra = "vllm" }, - ], - [ - { package = "nemo-rl", extra = "trtllm" }, - { package = "nemo-gym", extra = "vllm" }, - ], [ { extra = "fsdp" }, { extra = "sglang" }, diff --git a/uv.lock b/uv.lock index 8c7331ac648..bf86d3de96c 100644 --- a/uv.lock +++ b/uv.lock @@ -10,21 +10,6 @@ supported-markers = [ "platform_machine == 'aarch64' and sys_platform == 'linux'", ] conflicts = [[ - { package = "nemo-gym", extra = "vllm" }, - { package = "nemo-rl", extra = "automodel" }, -], [ - { package = "nemo-gym", extra = "vllm" }, - { package = "nemo-rl", extra = "vllm" }, -], [ - { package = "nemo-gym", extra = "vllm" }, - { package = "nemo-rl", extra = "sglang" }, -], [ - { package = "nemo-gym", extra = "vllm" }, - { package = "nemo-rl", extra = "mcore" }, -], [ - { package = "nemo-gym", extra = "vllm" }, - { package = "nemo-rl", extra = "trtllm" }, -], [ { package = "nemo-rl", extra = "fsdp" }, { package = "nemo-rl", extra = "sglang" }, ], [ @@ -64,7 +49,6 @@ conflicts = [[ [manifest] members = [ - "nemo-gym", "nemo-rl", "template-project", ] @@ -315,18 +299,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/98/3b/40a68de458904bcc143622015fff2352b6461cd92fd66d3527bf1c6f5716/aiohttp_cors-0.8.1-py3-none-any.whl", hash = "sha256:3180cf304c5c712d626b9162b195b1db7ddf976a2a25172b35bb2448b890a80d", size = 25231, upload-time = "2025-03-31T14:16:18.478Z" }, ] -[[package]] -name = "aiohttp-retry" -version = "2.9.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "aiohttp" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/9d/61/ebda4d8e3d8cfa1fd3db0fb428db2dd7461d5742cea35178277ad180b033/aiohttp_retry-2.9.1.tar.gz", hash = "sha256:8eb75e904ed4ee5c2ec242fefe85bf04240f685391c4879d8f541d6028ff01f1", size = 13608, upload-time = "2024-11-06T10:44:54.574Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/1a/99/84ba7273339d0f3dfa57901b846489d2e5c2cd731470167757f1935fffbd/aiohttp_retry-2.9.1-py3-none-any.whl", hash = "sha256:66d2759d1921838256a05a3f80ad7e724936f083e35be5abb5e16eed6be6dc54", size = 9981, upload-time = "2024-11-06T10:44:52.917Z" }, -] - [[package]] name = "aioitertools" version = "0.13.0" @@ -440,7 +412,7 @@ name = "apache-tvm-ffi" version = "0.1.9" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "typing-extensions", marker = "platform_machine == 'x86_64' or extra != 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "typing-extensions", marker = "platform_machine == 'x86_64' or extra != 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/6f/60/1e787a0b5ebf318483235be2a689ee367173983067e441b8379564f667c0/apache_tvm_ffi-0.1.9.tar.gz", hash = "sha256:d2d402587e8906de0a07f4746aa78f3d452c7efe3625d4bb39ac2ad693bce530", size = 2513731, upload-time = "2026-02-27T19:28:06.602Z" } wheels = [ @@ -450,19 +422,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/70/ef/5402da5d37f5270fd88ea0348acca78dba9be8bdbf6c2bcae0935eb03ef1/apache_tvm_ffi-0.1.9-cp312-abi3-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f45eb43499acac45ff6c93564f0ff2d3ca27b69656d540fd56ce59d51c0b4c65", size = 2278991, upload-time = "2026-02-27T19:27:30.729Z" }, ] -[[package]] -name = "ast-serialize" -version = "0.6.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/58/ad/0d70a3a2d6e01968d985415259e8ec7ad3f777903f9b1c1f3c8c44642c60/ast_serialize-0.6.0.tar.gz", hash = "sha256:aadd3ffcf4858c9726bf3515f7b199c7eadbe504f96028e4a87172c0da65a8fe", size = 61489, upload-time = "2026-06-30T20:02:55.555Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/6f/02/613a7534a41d0122f37d1e0c64aa8ac78bfb831f8c92f6db057a311abb3c/ast_serialize-0.6.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:305802f2ce2a7c4e87835078ea85c58b586ddda8095b92fe2ead9364ae19c80a", size = 1238620, upload-time = "2026-06-30T20:02:33.664Z" }, - { url = "https://files.pythonhosted.org/packages/ca/87/b4d6c38e0ccd5e85dc54cecdf933a152c60b28fe5d993a6d8a72fa6d5896/ast_serialize-0.6.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:dcbed41e9386059fc0261d602445ede0976c2ecec2939688bcbcb9ed0b6f28b7", size = 1261693, upload-time = "2026-06-30T20:02:39.123Z" }, - { url = "https://files.pythonhosted.org/packages/b1/f2/13736d920ab3d49bbee80ef1a277dd7b7aaf3b3545efd9d2a8114fe05525/ast_serialize-0.6.0-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:c4af9a1386166e40ed01464991806f89038a2d89782576c7774876fa77034e32", size = 1413698, upload-time = "2026-06-30T20:02:44.179Z" }, - { url = "https://files.pythonhosted.org/packages/95/93/5524a3dc6c3f593de3228ed9cbef73afa047625b7000ec21b7f58e6eb4d4/ast_serialize-0.6.0-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:4ed29121da8b3fdc291002801a1de0f76248fa07dce89157a5f277842cf6126e", size = 1457164, upload-time = "2026-06-30T20:02:48.294Z" }, - { url = "https://files.pythonhosted.org/packages/4f/c0/36a6ffb4d653cf621427b4c4928671f53ad800c453474de2b82564a44ad9/ast_serialize-0.6.0-cp39-abi3-pyemscripten_2026_0_wasm32.whl", hash = "sha256:b1dac4e09d341c1300ba69cdcbe62867b32a8c75d90db9bf4d083bec3b039f0b", size = 863014, upload-time = "2026-06-30T20:02:49.742Z" }, -] - [[package]] name = "astor" version = "0.8.1" @@ -580,15 +539,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/88/c6/92fcd42f1ba33e1184263f25bfabf3d27c383410470f169e4b8163bf9c17/beautifulsoup4-4.15.0-py3-none-any.whl", hash = "sha256:d6f88de62e1d4e38ecb1077eb9724cd0eff29d2a08ca16a401e9b9e93f117cf9", size = 109924, upload-time = "2026-06-07T16:44:21.566Z" }, ] -[[package]] -name = "bidict" -version = "0.23.1" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/9a/6e/026678aa5a830e07cd9498a05d3e7e650a4f56a42f267a53d22bcda1bdc9/bidict-0.23.1.tar.gz", hash = "sha256:03069d763bc387bbd20e7d49914e75fc4132a41937fa3405417e1a5a2d006d71", size = 29093, upload-time = "2024-02-18T19:09:05.748Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/99/37/e8730c3587a65eb5645d4aba2d27aae48e8003614d6aaf15dda67f702f1f/bidict-0.23.1-py3-none-any.whl", hash = "sha256:5dae8d4d79b552a71cbabc7deb25dfe8ce710b17ff41711e13010ead2abfc3e5", size = 32764, upload-time = "2024-02-18T19:09:04.156Z" }, -] - [[package]] name = "blake3" version = "1.0.9" @@ -713,9 +663,9 @@ name = "causal-conv1d" version = "1.5.4" source = { git = "https://github.com/Dao-AILab/causal-conv1d?rev=4f6ae4e26ae5fe8af9372f8d312ab25cc4595223#4f6ae4e26ae5fe8af9372f8d312ab25cc4595223" } dependencies = [ - { name = "ninja", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-fsdp' or extra == 'extra-7-nemo-rl-mcore' or extra != 'extra-7-nemo-rl-sglang' or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "packaging", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-fsdp' or extra == 'extra-7-nemo-rl-mcore' or extra != 'extra-7-nemo-rl-sglang' or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "torch", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-fsdp' or extra == 'extra-7-nemo-rl-mcore' or extra != 'extra-7-nemo-rl-sglang' or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "ninja", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-fsdp' or extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "packaging", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-fsdp' or extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "torch", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-fsdp' or extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] [[package]] @@ -744,7 +694,7 @@ name = "cffi" version = "2.1.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "pycparser", marker = "implementation_name != 'PyPy' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "pycparser", marker = "implementation_name != 'PyPy' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/57/5f/ff100cae70ebe9d8df1c01a00e510e45d9adb5c1fdda84791b199141de97/cffi-2.1.0.tar.gz", hash = "sha256:efc1cdd798b1aaf39b4610bba7aad28c9bea9b910f25c784ccf9ec1fa719d1f9", size = 531036, upload-time = "2026-07-06T21:34:30.382Z" } wheels = [ @@ -841,7 +791,7 @@ version = "3.58.4" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "dulwich" }, - { name = "everett", extra = ["ini"], marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "everett", extra = ["ini"], marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "jsonschema" }, { name = "psutil" }, { name = "python-box" }, @@ -945,7 +895,7 @@ name = "cryptography" version = "48.0.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "cffi", marker = "platform_python_implementation != 'PyPy' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "cffi", marker = "platform_python_implementation != 'PyPy' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/12/45/870e7f4bef50e5f53b9f51d4428aee5290eedf58ba443f16b1ebb7ab8e66/cryptography-48.0.1.tar.gz", hash = "sha256:266f4ee051abb2f725b74ef8072b521ce1feacf685a3364fa6a6b45548db791a", size = 832989, upload-time = "2026-06-09T22:32:31.8Z" } wheels = [ @@ -1161,134 +1111,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/05/66/73034ad30b59f13439b75e620989dacba4c047256e358ba7c2e9ec98ea22/datasets-5.0.0-py3-none-any.whl", hash = "sha256:7dd34927a0fd7046e98aad5cb9430e699c373238a15befa7b9bf22b991a7fee6", size = 555084, upload-time = "2026-06-05T13:18:24.435Z" }, ] -[[package]] -name = "daytona" -version = "0.200.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "aiofiles" }, - { name = "aiohttp" }, - { name = "daytona-analytics-api-client" }, - { name = "daytona-analytics-api-client-async" }, - { name = "daytona-api-client" }, - { name = "daytona-api-client-async" }, - { name = "daytona-toolbox-api-client" }, - { name = "daytona-toolbox-api-client-async" }, - { name = "deprecated" }, - { name = "httpx" }, - { name = "httpx-ws" }, - { name = "obstore" }, - { name = "opentelemetry-api" }, - { name = "opentelemetry-exporter-otlp-proto-http" }, - { name = "opentelemetry-instrumentation-aiohttp-client" }, - { name = "opentelemetry-sdk" }, - { name = "pydantic" }, - { name = "python-dotenv" }, - { name = "python-multipart" }, - { name = "python-socketio", extra = ["asyncio-client", "client"] }, - { name = "toml" }, - { name = "typing-extensions" }, - { name = "urllib3" }, - { name = "wsproto" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/47/eb/2d2db2068f23697964b2aab383bb8391b618462df1ae6dca1df464f27259/daytona-0.200.1.tar.gz", hash = "sha256:02fc040cdbb54417ba7c1dc6bc1aa7489543bd15fc3fe9aea6968b0516f7ea4b", size = 177604, upload-time = "2026-07-22T12:23:22.305Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/c1/32/87982084d380e6a34aff1b2fac8eb6f8649f831c4f8f39c8d0eb90a0749f/daytona-0.200.1-py3-none-any.whl", hash = "sha256:68029ff267499a8dcc9d946fc71e79b9ea456170c77fb836bf4b0cd3c1e59959", size = 213381, upload-time = "2026-07-22T12:23:23.43Z" }, -] - -[[package]] -name = "daytona-analytics-api-client" -version = "0.200.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "pydantic" }, - { name = "python-dateutil" }, - { name = "typing-extensions" }, - { name = "urllib3" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/f3/de/9e95c70c792f20ef0c43e157a5a8f56d71d57b716335da2dc7d29ddb082a/daytona_analytics_api_client-0.200.1.tar.gz", hash = "sha256:1624d437cbc50c91f921c4b46567f55f8ddf89894fa80840acf01788daf85a4b", size = 30058, upload-time = "2026-07-22T12:22:47.817Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/a7/34/e17297664a6e8a0a921ad971683e220bce69e57a3c97f37c2815a3f70991/daytona_analytics_api_client-0.200.1-py3-none-any.whl", hash = "sha256:9f8ad5648f0da73b34ce9b13ce3c60173562d48404ddc7f6331eeec4beb2c772", size = 45010, upload-time = "2026-07-22T12:22:48.643Z" }, -] - -[[package]] -name = "daytona-analytics-api-client-async" -version = "0.200.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "aiohttp" }, - { name = "aiohttp-retry" }, - { name = "pydantic" }, - { name = "python-dateutil" }, - { name = "typing-extensions" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/77/ec/e9bd405a74382f97b1a16376686bf90ed9c3b183718897d519eb8ced73d9/daytona_analytics_api_client_async-0.200.1.tar.gz", hash = "sha256:f5edc0b1a6590ae802490d97789f21f94d3bf73c53a4d984c980d89cc0eca132", size = 30071, upload-time = "2026-07-22T12:22:39.053Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/d5/c5/2979f438def31f26dc98b3e23a9e8787570d2bb95019f85596fa7eafb8a6/daytona_analytics_api_client_async-0.200.1-py3-none-any.whl", hash = "sha256:637106455f9498d6785efb51be9d589a2ecd3f257e2a3d9f1c7b6bd8c461721a", size = 45283, upload-time = "2026-07-22T12:22:39.882Z" }, -] - -[[package]] -name = "daytona-api-client" -version = "0.200.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "pydantic" }, - { name = "python-dateutil" }, - { name = "typing-extensions" }, - { name = "urllib3" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/ef/be/9e604a59ea3b3dc5a120b9dd216b1db12f4da2e6b8601e7d4f49b1874c96/daytona_api_client-0.200.1.tar.gz", hash = "sha256:fcf17e051eb7e0a9b8a38d02ef74f7ed67f84c3b4d150c8dd76c8c942f521fa9", size = 124815, upload-time = "2026-07-22T12:22:56.725Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/43/1a/094725c4f495eba4411666a12307f885cead3aa29917eccf72f3630f36ff/daytona_api_client-0.200.1-py3-none-any.whl", hash = "sha256:7d7c41d7c5b601d9df908bd0ab0f0f567ade2913edf2eb087f65cfc6fcbe73a0", size = 316076, upload-time = "2026-07-22T12:22:57.798Z" }, -] - -[[package]] -name = "daytona-api-client-async" -version = "0.200.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "aiohttp" }, - { name = "aiohttp-retry" }, - { name = "pydantic" }, - { name = "python-dateutil" }, - { name = "typing-extensions" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/5e/dc/132b3ac1f44434ef4115abe99761542da9ac025a20f011fc081638148245/daytona_api_client_async-0.200.1.tar.gz", hash = "sha256:269874af7f97368531a08762639590a59f0ece95e194fa6ebe998fbf8258d4dd", size = 125346, upload-time = "2026-07-22T12:22:39.687Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/2a/47/ea80967ccdcc4fbabbf628a1b162f0f500edd75eebffc55fea88ada41ea8/daytona_api_client_async-0.200.1-py3-none-any.whl", hash = "sha256:df80d0aeb52c82a4660d746464842bdb6c8a2e0203b0cdc6374037b164c47824", size = 318668, upload-time = "2026-07-22T12:22:41.49Z" }, -] - -[[package]] -name = "daytona-toolbox-api-client" -version = "0.200.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "pydantic" }, - { name = "python-dateutil" }, - { name = "typing-extensions" }, - { name = "urllib3" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/0e/d8/650c9b70734931ac369dc5369d4168bc81e65fca244d19afb6d77b496bd9/daytona_toolbox_api_client-0.200.1.tar.gz", hash = "sha256:c5c74b390c20396494a3a70e843a5da70187ba5d197b26fe98be366e2a1a81da", size = 86312, upload-time = "2026-07-22T12:22:49.038Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/7d/a2/329d03f4898e3b768bf285b3353f3db347e6946f8f7a3b7587325f7e6468/daytona_toolbox_api_client-0.200.1-py3-none-any.whl", hash = "sha256:7baaebf323d663e669db6dbbba816fccd7facca101d6cffa86959dee06593ce3", size = 247064, upload-time = "2026-07-22T12:22:50.356Z" }, -] - -[[package]] -name = "daytona-toolbox-api-client-async" -version = "0.200.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "aiohttp" }, - { name = "aiohttp-retry" }, - { name = "pydantic" }, - { name = "python-dateutil" }, - { name = "typing-extensions" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/8f/ad/92aa88099f210e2ff73cebfa0ca08538f05068e6c5c25a3e565eb9438d75/daytona_toolbox_api_client_async-0.200.1.tar.gz", hash = "sha256:ffe6c6bd9289b56a448a7e0f1f36a295fda431e0d73fc74c53cfb9492f590174", size = 80178, upload-time = "2026-07-22T12:22:39.642Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/b6/00/b3f5b3e40a37af8b883590d78a7d6935a464e74364129732daa322c4bd37/daytona_toolbox_api_client_async-0.200.1-py3-none-any.whl", hash = "sha256:fc953a1ee1d9967900b21c5eb6188c54617b1d66d5ad6b737e457a9e76a4fb4a", size = 245534, upload-time = "2026-07-22T12:22:40.771Z" }, -] - [[package]] name = "debugpy" version = "1.8.21" @@ -1353,18 +1175,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/07/6c/aa3f2f849e01cb6a001cd8554a88d4c77c5c1a31c95bdf1cf9301e6d9ef4/defusedxml-0.7.1-py2.py3-none-any.whl", hash = "sha256:a352e7e428770286cc899e2542b6cdaedb2b4953ff269a210103ec58f6198a61", size = 25604, upload-time = "2021-03-08T10:59:24.45Z" }, ] -[[package]] -name = "deprecated" -version = "1.3.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "wrapt" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/49/85/12f0a49a7c4ffb70572b6c2ef13c90c88fd190debda93b23f026b25f9634/deprecated-1.3.1.tar.gz", hash = "sha256:b1b50e0ff0c1fddaa5708a2c6b0a6588bb09b892825ab2b214ac9ea9d92a5223", size = 2932523, upload-time = "2025-10-30T08:19:02.757Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/84/d0/205d54408c08b13550c733c4b85429e7ead111c7f0014309637425520a9a/deprecated-1.3.1-py2.py3-none-any.whl", hash = "sha256:597bfef186b6f60181535a29fbe44865ce137a5079f295b479886c82729d5f3f", size = 11298, upload-time = "2025-10-30T08:19:00.758Z" }, -] - [[package]] name = "deprecation" version = "2.1.0" @@ -1595,15 +1405,6 @@ ini = [ { name = "configobj" }, ] -[[package]] -name = "execnet" -version = "2.1.2" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/bf/89/780e11f9588d9e7128a3f87788354c7946a9cbb1401ad38a48c4db9a4f07/execnet-2.1.2.tar.gz", hash = "sha256:63d83bfdd9a23e35b9c6a3261412324f964c2ec8dcd8d3c6916ee9373e0befcd", size = 166622, upload-time = "2025-11-12T09:56:37.75Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/ab/84/02fc1827e8cdded4aa65baef11296a9bbe595c474f0d6d758af082d849fd/execnet-2.1.2-py3-none-any.whl", hash = "sha256:67fba928dd5a544b783f6056f449e5e3931a5c378b128bc18501f7ea79e296ec", size = 40708, upload-time = "2025-11-12T09:56:36.333Z" }, -] - [[package]] name = "executing" version = "2.2.1" @@ -1632,11 +1433,11 @@ resolution-markers = [ "platform_machine == 'aarch64' and sys_platform == 'linux'", ] dependencies = [ - { name = "annotated-doc", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "pydantic", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "starlette", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "typing-extensions", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "typing-inspection", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "annotated-doc", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "pydantic", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "starlette", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "typing-extensions", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "typing-inspection", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/81/2d/ff8d91d7b564d464629a0fd50a4489c97fcb836ac230bf3a7269232a9b1f/fastapi-0.136.3.tar.gz", hash = "sha256:e487fae93ad408e6f47641ee4dfe389864fd7bec92e547ea8498fc13f43e83ab", size = 396410, upload-time = "2026-05-23T18:53:15.192Z" } wheels = [ @@ -1646,14 +1447,14 @@ wheels = [ [package.optional-dependencies] standard = [ { name = "email-validator" }, - { name = "fastapi-cli", extra = ["standard"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "fastapi-cli", extra = ["standard"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "fastar" }, { name = "httpx" }, { name = "jinja2" }, { name = "pydantic-extra-types" }, { name = "pydantic-settings" }, { name = "python-multipart" }, - { name = "uvicorn", extra = ["standard"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "uvicorn", extra = ["standard"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] [[package]] @@ -1665,11 +1466,11 @@ resolution-markers = [ "platform_machine == 'aarch64' and sys_platform == 'linux'", ] dependencies = [ - { name = "annotated-doc", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-mcore' or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "pydantic", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-mcore' or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "starlette", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-mcore' or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "typing-extensions", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-mcore' or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "typing-inspection", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-mcore' or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, + { name = "annotated-doc", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-mcore' or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-trtllm' or extra != 'extra-7-nemo-rl-vllm'" }, + { name = "pydantic", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-mcore' or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-trtllm' or extra != 'extra-7-nemo-rl-vllm'" }, + { name = "starlette", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-mcore' or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-trtllm' or extra != 'extra-7-nemo-rl-vllm'" }, + { name = "typing-extensions", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-mcore' or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-trtllm' or extra != 'extra-7-nemo-rl-vllm'" }, + { name = "typing-inspection", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-mcore' or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-trtllm' or extra != 'extra-7-nemo-rl-vllm'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/cd/95/d3f0ae10836324a2eab98a52b61210ac609f08200bf4bb0dc8132d32f78a/fastapi-0.139.2.tar.gz", hash = "sha256:333145a6891e9b5b3cfceb69baf817e8240cde4d4588ae5a10bf56ffacb6255e", size = 423428, upload-time = "2026-07-16T15:06:17.912Z" } wheels = [ @@ -1683,7 +1484,7 @@ source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "rich-toolkit" }, { name = "typer" }, - { name = "uvicorn", extra = ["standard"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "uvicorn", extra = ["standard"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/33/eb/3b534c6f8e157f9ddbf2a153512307c886cad0b258739c200dd8ff8c4452/fastapi_cli-0.0.32.tar.gz", hash = "sha256:38024d2345275e1b37ce8848727a580d84901b570e96b3256d9d36a9a5039424", size = 26636, upload-time = "2026-07-16T12:16:58.678Z" } wheels = [ @@ -1693,7 +1494,7 @@ wheels = [ [package.optional-dependencies] standard = [ { name = "fastapi-cloud-cli" }, - { name = "uvicorn", extra = ["standard"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "uvicorn", extra = ["standard"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] [[package]] @@ -1704,12 +1505,12 @@ dependencies = [ { name = "detect-installer" }, { name = "fastar" }, { name = "httpx" }, - { name = "pydantic", extra = ["email"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "pydantic", extra = ["email"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "rich-toolkit" }, { name = "rignore" }, { name = "sentry-sdk" }, { name = "typer" }, - { name = "uvicorn", extra = ["standard"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "uvicorn", extra = ["standard"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/25/f2/36bfe990baa656de89a2b98a77a15dcd018474f7245c8e4a10cada0553c5/fastapi_cloud_cli-0.22.2.tar.gz", hash = "sha256:7ec78c1fed58f578af5eb1fb54ec4b456eba4dc1eaca1c3a93cf499a7cbc7ab3", size = 94480, upload-time = "2026-07-14T09:27:01.349Z" } wheels = [ @@ -1815,11 +1616,11 @@ resolution-markers = [ "platform_machine == 'aarch64' and sys_platform == 'linux'", ] dependencies = [ - { name = "einops", marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "ninja", marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "psutil", marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "setuptools", marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "torch", marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "einops", marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "ninja", marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "psutil", marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "setuptools", marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "torch", marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] wheels = [ { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_aarch64.whl", hash = "sha256:9fdfbdf2d4ca984ffa34518a4054c62c295fa06e184d7a233848dc06be8f1ec2" }, @@ -1833,11 +1634,11 @@ resolution-markers = [ "platform_machine == 'x86_64' and sys_platform == 'linux'", ] dependencies = [ - { name = "einops", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "ninja", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "psutil", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "setuptools", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "torch", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "einops", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "ninja", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "psutil", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "setuptools", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "torch", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] wheels = [ { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_x86_64.whl", hash = "sha256:a14fea157eb61dfee56f605a24a7a6150704517759125b3942dd04bffa05d6cf" }, @@ -1945,18 +1746,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/56/0c/80dad211d424c3f25199ccd9bb1913c3e2d7378b5cd3dbcd2f75a635b6dd/flashinfer_cubin-0.6.11.post1-py3-none-any.whl", hash = "sha256:1eb801fc80b5576760d356e31eb452d05ab240f44185f539331bd5f2236e504a", size = 360908523, upload-time = "2026-05-13T01:31:43.386Z" }, ] -[[package]] -name = "flashinfer-cubin" -version = "0.6.12" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "platform_machine == 'x86_64' and sys_platform == 'linux'", - "platform_machine == 'aarch64' and sys_platform == 'linux'", -] -wheels = [ - { url = "https://files.pythonhosted.org/packages/7d/c6/63b1bb7b1a7ae612ecf53c0e568312c3d004f9f7558b0ab5edcf7900c360/flashinfer_cubin-0.6.12-py3-none-any.whl", hash = "sha256:01de132c493bb21d5df42ebe6890966cf83b40aa970dae06b2a3c0bed85f13ec", size = 447533460, upload-time = "2026-05-29T23:45:27.579Z" }, -] - [[package]] name = "flashinfer-cubin" version = "0.6.13" @@ -2077,13 +1866,12 @@ resolution-markers = [ dependencies = [ { name = "apache-tvm-ffi" }, { name = "click" }, - { name = "cuda-tile", version = "1.1.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "cuda-tile", version = "1.1.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, { name = "einops" }, { name = "ninja" }, { name = "numpy" }, { name = "nvidia-cudnn-frontend" }, - { name = "nvidia-cutlass-dsl", version = "4.5.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "nvidia-cutlass-dsl", version = "4.5.2", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "nvidia-cutlass-dsl", version = "4.5.0", source = { registry = "https://pypi.org/simple" } }, { name = "nvidia-ml-py" }, { name = "packaging" }, { name = "requests" }, @@ -2107,7 +1895,7 @@ resolution-markers = [ dependencies = [ { name = "apache-tvm-ffi" }, { name = "click" }, - { name = "cuda-tile", version = "1.1.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "cuda-tile", version = "1.1.0", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "einops" }, { name = "ninja" }, { name = "numpy" }, @@ -2328,15 +2116,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e7/c8/e2645aa8ed02fd4c7a2f59d68783b65b1f3cbdfe39a6308e156509d1fee8/googleapis_common_protos-1.75.0-py3-none-any.whl", hash = "sha256:961ed60399c457ceb0ee8f285a84c870aabc9c6a832b9d37bb281b5bebde43ed", size = 300631, upload-time = "2026-05-07T08:03:30.345Z" }, ] -[[package]] -name = "gprof2dot" -version = "2025.4.14" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/bb/fd/cad13fa1f7a463a607176432c4affa33ea162f02f58cc36de1d40d3e6b48/gprof2dot-2025.4.14.tar.gz", hash = "sha256:35743e2d2ca027bf48fa7cba37021aaf4a27beeae1ae8e05a50b55f1f921a6ce", size = 39536, upload-time = "2025-04-14T07:21:45.76Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/71/ed/89d760cb25279109b89eb52975a7b5479700d3114a2421ce735bfb2e7513/gprof2dot-2025.4.14-py3-none-any.whl", hash = "sha256:0742e4c0b4409a5e8777e739388a11e1ed3750be86895655312ea7c20bd0090e", size = 37555, upload-time = "2025-04-14T07:21:43.319Z" }, -] - [[package]] name = "graphene" version = "3.4.3" @@ -2588,21 +2367,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d2/fd/6668e5aec43ab844de6fc74927e155a3b37bf40d7c3790e49fc0406b6578/httpx_sse-0.4.3-py3-none-any.whl", hash = "sha256:0ac1c9fe3c0afad2e0ebb25a934a59f4c7823b60792691f779fad2c5568830fc", size = 8960, upload-time = "2025-10-10T21:48:21.158Z" }, ] -[[package]] -name = "httpx-ws" -version = "0.9.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "anyio" }, - { name = "httpcore" }, - { name = "httpx" }, - { name = "wsproto" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/cd/cd/ca91a07ae446451f7476bf3fcc909e98cb942ff032ebfda0e3fe449aca7b/httpx_ws-0.9.0.tar.gz", hash = "sha256:797373326f70eec1ae96f6e43ae9f12002fd7d73aee139a4985eaab964338a08", size = 107105, upload-time = "2026-03-28T14:11:10.781Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/98/f8/a6bc80313a9e93c888fa10534dfce2ad76ff86911b6f485777ce6de6a073/httpx_ws-0.9.0-py3-none-any.whl", hash = "sha256:71640d2fb1bf9a225775015b33cd755cfd4c5f7e21c885192fe3adc4c387b248", size = 15759, upload-time = "2026-03-28T14:11:11.887Z" }, -] - [[package]] name = "huey" version = "3.2.1" @@ -2632,47 +2396,10 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5f/c3/aeaaf3911d2529614be18d1c8b5496afc185560e76568063d517283318af/huggingface_hub-1.24.0-py3-none-any.whl", hash = "sha256:6ed4120a84a6beec900640aa7e346bd766a6b7341e41526fef5dc8bd81fb7d59", size = 771904, upload-time = "2026-07-17T09:53:59.106Z" }, ] -[[package]] -name = "humming-kernels" -version = "0.1.6" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "platform_machine == 'x86_64' and sys_platform == 'linux'", - "platform_machine == 'aarch64' and sys_platform == 'linux'", -] -dependencies = [ - { name = "cuda-bindings" }, - { name = "jinja2" }, - { name = "numpy" }, - { name = "nvidia-ml-py" }, - { name = "pyelftools" }, - { name = "safetensors" }, - { name = "tabulate" }, - { name = "torch" }, - { name = "tqdm" }, - { name = "triton" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/29/5a/fbf574dcd83e9fea6aa3fa96b37bbdec40b8672407b1ed9679efa31fff1d/humming_kernels-0.1.6.tar.gz", hash = "sha256:882b9f382a010165a7cf8eecbad943bfe8d6b17566328fb57611c9a34bdccc9a", size = 214408, upload-time = "2026-06-20T04:04:43.221Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/25/85/490681b9ba24531da91d0bae801d2b26850e5a80bbd02c2efc500756e36b/humming_kernels-0.1.6-py3-none-any.whl", hash = "sha256:e64c0883fca930074bf920f4ba47cbf3acd244d7352f6c74c8d2182439770d8f", size = 178759, upload-time = "2026-06-20T04:04:41.66Z" }, -] - -[package.optional-dependencies] -cu13 = [ - { name = "nvidia-cuda-cccl" }, - { name = "nvidia-cuda-nvcc" }, - { name = "nvidia-cuda-nvrtc" }, - { name = "nvidia-cuda-runtime" }, -] - [[package]] name = "humming-kernels" version = "0.1.10" source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "platform_machine == 'x86_64' and sys_platform == 'linux'", - "platform_machine == 'aarch64' and sys_platform == 'linux'", -] dependencies = [ { name = "cuda-bindings" }, { name = "jinja2" }, @@ -2707,9 +2434,9 @@ resolution-markers = [ "platform_machine == 'aarch64' and sys_platform == 'linux'", ] dependencies = [ - { name = "antlr4-python3-runtime", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "omegaconf", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "packaging", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "antlr4-python3-runtime", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "omegaconf", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "packaging", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/6d/8e/07e42bc434a847154083b315779b0a81d567154504624e181caf2c71cd98/hydra-core-1.3.2.tar.gz", hash = "sha256:8a878ed67216997c3e9d88a8e72e7b4767e81af37afb4ea3334b269a4390a824", size = 3263494, upload-time = "2023-02-23T18:33:43.03Z" } wheels = [ @@ -2725,9 +2452,9 @@ resolution-markers = [ "platform_machine == 'aarch64' and sys_platform == 'linux'", ] dependencies = [ - { name = "antlr4-python3-runtime", marker = "extra == 'extra-7-nemo-rl-automodel' or extra != 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "omegaconf", marker = "extra == 'extra-7-nemo-rl-automodel' or extra != 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "packaging", marker = "extra == 'extra-7-nemo-rl-automodel' or extra != 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "antlr4-python3-runtime", marker = "extra == 'extra-7-nemo-rl-automodel' or extra != 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "omegaconf", marker = "extra == 'extra-7-nemo-rl-automodel' or extra != 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "packaging", marker = "extra == 'extra-7-nemo-rl-automodel' or extra != 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/10/dd/220f0e91743136725352497e98540772a01fc7c3ab96ff16c3c74424e984/hydra_core-1.3.4.tar.gz", hash = "sha256:ad0f7b05a0242255a8984d5a4ed2f6847f7b783ed727368a2c0155ec52d6c34c", size = 3263348, upload-time = "2026-07-04T16:25:38.891Z" } wheels = [ @@ -3058,19 +2785,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b5/ba/c63c5786dfee4c3417094c4b00966e61e4a63efecee22cb7b4c0387dda83/librosa-0.11.0-py3-none-any.whl", hash = "sha256:0b6415c4fd68bff4c29288abe67c6d80b587e0e1e2cfb0aad23e4559504a7fa1", size = 260749, upload-time = "2025-03-11T15:09:52.982Z" }, ] -[[package]] -name = "librt" -version = "0.13.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/dc/2f/3908645ddddab7120b46295e541ead308109fa48dbec7d67d7a778870d60/librt-0.13.0.tar.gz", hash = "sha256:1d2a610c14ac0d0750ee0a3ab8548e83155258387891caaca04def4bf7289781", size = 211402, upload-time = "2026-07-08T12:26:29.834Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/0e/16/52b1c99bf19057a062aac39c900cbb81499f6f75d6c537c14463d247ba78/librt-0.13.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:32c26893cd085c1efe83219e78d866da23fb20a066101b8f68210004361d224c", size = 502499, upload-time = "2026-07-08T12:25:22.055Z" }, - { url = "https://files.pythonhosted.org/packages/8f/f8/094d6b2bd93f3fdaa54db54cc788c4a365333bddad65ab02e04da0b1d004/librt-0.13.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:94b85d664d777bab6c0d709416cb42938251fda9e221b79e3a2215d85df5f4f9", size = 531576, upload-time = "2026-07-08T12:25:24.648Z" }, - { url = "https://files.pythonhosted.org/packages/c6/b5/255673cfdbf5ba663339d36cd863c897289ab4337577e19f9405ce059f36/librt-0.13.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:109b84a9edf69ad89dc1f66358659e14a031baca95e3e5b0060bd903ede8efd6", size = 543053, upload-time = "2026-07-08T12:25:27.436Z" }, - { url = "https://files.pythonhosted.org/packages/5a/76/927e267a6daa290174ac281b23c9804c8829b042ade9c6f24a065f540958/librt-0.13.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9f836c37478f167a81200d8c8b2c920a22224564bed2c23d7aeec760965c367a", size = 573582, upload-time = "2026-07-08T12:25:31.507Z" }, - { url = "https://files.pythonhosted.org/packages/10/24/b6c5213efe39c19f9e13605644d0cf063b4ddaa33ac2e45b088e23a70e2e/librt-0.13.0-cp313-cp313-pyemscripten_2025_0_wasm32.whl", hash = "sha256:4000d961ff9598ac6ea603c6c836a5ed49bc205ade5fc378b998dfe1e2c36628", size = 82189, upload-time = "2026-07-08T12:25:32.675Z" }, -] - [[package]] name = "llguidance" version = "1.3.0" @@ -3152,10 +2866,10 @@ name = "mamba-ssm" version = "2.2.6.post3" source = { git = "https://github.com/state-spaces/mamba.git?rev=a14b1dff0454a3bc27d9eb31355dc01e4b2490ec#a14b1dff0454a3bc27d9eb31355dc01e4b2490ec" } dependencies = [ - { name = "causal-conv1d", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-fsdp' or extra == 'extra-7-nemo-rl-mcore' or extra != 'extra-7-nemo-rl-sglang' or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "ninja", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-fsdp' or extra == 'extra-7-nemo-rl-mcore' or extra != 'extra-7-nemo-rl-sglang' or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "packaging", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-fsdp' or extra == 'extra-7-nemo-rl-mcore' or extra != 'extra-7-nemo-rl-sglang' or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "torch", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-fsdp' or extra == 'extra-7-nemo-rl-mcore' or extra != 'extra-7-nemo-rl-sglang' or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "causal-conv1d", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-fsdp' or extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "ninja", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-fsdp' or extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "packaging", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-fsdp' or extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "torch", marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-fsdp' or extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] [[package]] @@ -3302,7 +3016,7 @@ dependencies = [ { name = "flashinfer-cubin", version = "0.6.8.post1", source = { registry = "https://pypi.org/simple" } }, { name = "flashinfer-python", version = "0.6.8.post1", source = { registry = "https://pypi.org/simple" } }, { name = "hydra-core", version = "1.3.2", source = { registry = "https://pypi.org/simple" } }, - { name = "megatron-core", extra = ["dev", "mlm"], marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "megatron-core", extra = ["dev", "mlm"], marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "mistral-common" }, { name = "mlflow" }, { name = "nvidia-resiliency-ext" }, @@ -3329,7 +3043,7 @@ ssm = [ { name = "mamba-ssm" }, ] te = [ - { name = "transformer-engine", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "transformer-engine", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, ] [[package]] @@ -3351,13 +3065,13 @@ dev = [ { name = "flash-linear-attention", version = "0.4.2", source = { registry = "https://pypi.org/simple" } }, { name = "flashinfer-python", version = "0.6.8.post1", source = { registry = "https://pypi.org/simple" } }, { name = "hypercorn" }, - { name = "megatron-energon", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "megatron-energon", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "multi-storage-client" }, { name = "nvidia-cudnn-frontend" }, - { name = "nvidia-modelopt", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "nvidia-modelopt", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "nvidia-resiliency-ext" }, { name = "onnxscript" }, - { name = "openai", extra = ["aiohttp"], marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "openai", extra = ["aiohttp"], marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "opentelemetry-api" }, { name = "orjson" }, { name = "quart" }, @@ -3508,8 +3222,8 @@ dependencies = [ { name = "click" }, { name = "cloudpickle" }, { name = "databricks-sdk" }, - { name = "fastapi", version = "0.136.3", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "fastapi", version = "0.139.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-mcore' or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, + { name = "fastapi", version = "0.136.3", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "fastapi", version = "0.139.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-mcore' or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-trtllm' or extra != 'extra-7-nemo-rl-vllm'" }, { name = "gitpython" }, { name = "importlib-metadata" }, { name = "opentelemetry-api" }, @@ -3718,34 +3432,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/7e/82/69e539c4c2027f1e1697e09aaa2449243085a0edf81ae2c6341e84d769b6/multiprocess-0.70.19-py39-none-any.whl", hash = "sha256:0d4b4397ed669d371c81dcd1ef33fd384a44d6c3de1bd0ca7ac06d837720d3c5", size = 133477, upload-time = "2026-01-19T06:47:38.619Z" }, ] -[[package]] -name = "mypy" -version = "2.3.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "ast-serialize" }, - { name = "librt", marker = "platform_python_implementation != 'PyPy' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "mypy-extensions" }, - { name = "pathspec" }, - { name = "typing-extensions" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/12/af/4e516a05d3ca2eb9283e9ec45b2c02225c1514dd6da49fd3c9eaa6639370/mypy-2.3.0.tar.gz", hash = "sha256:465965d41cd9a2726694e983e8ce7113259327bec798115d1e1dfa2a52fb666e", size = 3988104, upload-time = "2026-07-13T11:34:53.387Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/d9/fb/43c031f0190513d1ec248ed037eceb742ddd2a4d74bbf406658a28173837/mypy-2.3.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:de6d2c484742a4d7b0ed6d07b143375624d3b899c5749c7b3c947f56261f48a6", size = 14151586, upload-time = "2026-07-13T11:29:18.615Z" }, - { url = "https://files.pythonhosted.org/packages/ec/c3/f8b2ffc60883084da91be51af58e88a7ffd4ff9795acb7d902ff88d31eb1/mypy-2.3.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7da939dd335cfd2ad788bdfd081c9f4e47634ab995e5a45eb15fd1e5bc052f8b", size = 15227411, upload-time = "2026-07-13T11:30:29.904Z" }, - { url = "https://files.pythonhosted.org/packages/83/2e/16b917fc7adcf03f1aadddfc93aab804ffb234b1ab09c0ffd6d92a5d34a2/mypy-2.3.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:7247eb2824f996722a949530183394921ca71deb9680052a338cf53cff7925c2", size = 15478790, upload-time = "2026-07-13T11:33:14.686Z" }, - { url = "https://files.pythonhosted.org/packages/2c/fa/fdc54fe583ba3cafbcedfb70eeeaf03849f75b1827a07096c7bd996f582d/mypy-2.3.0-py3-none-any.whl", hash = "sha256:6b1cdb579446b60432432b2b2403a6201b4b475a004d7f488511c9ba177c9e88", size = 2753292, upload-time = "2026-07-13T11:33:18.48Z" }, -] - -[[package]] -name = "mypy-extensions" -version = "1.1.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/a2/6e/371856a3fb9d31ca8dac321cda606860fa4548858c0cc45d9d1d4ca2628b/mypy_extensions-1.1.0.tar.gz", hash = "sha256:52e68efc3284861e772bbcd66823fde5ae21fd2fdb51c62a211403730b916558", size = 6343, upload-time = "2025-04-22T14:54:24.164Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl", hash = "sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505", size = 4963, upload-time = "2025-04-22T14:54:22.983Z" }, -] - [[package]] name = "myst-parser" version = "5.1.0" @@ -3794,7 +3480,7 @@ dependencies = [ { name = "datasets" }, { name = "flashoptim" }, { name = "megatron-fsdp" }, - { name = "mistral-common", extra = ["audio", "hf-hub", "sentencepiece"], marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "mistral-common", extra = ["audio", "hf-hub", "sentencepiece"], marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "mlflow" }, { name = "pybind11" }, { name = "pyyaml" }, @@ -3809,13 +3495,13 @@ dependencies = [ [package.optional-dependencies] moe = [ { name = "causal-conv1d" }, - { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=29d31c095796f3c8ece47ee9cdcc167051bbeed9#29d31c095796f3c8ece47ee9cdcc167051bbeed9" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=a48493600c4886c1b297aaa78db0e1ebc2d8dd6c#a48493600c4886c1b297aaa78db0e1ebc2d8dd6c" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=29d31c095796f3c8ece47ee9cdcc167051bbeed9#29d31c095796f3c8ece47ee9cdcc167051bbeed9" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=a48493600c4886c1b297aaa78db0e1ebc2d8dd6c#a48493600c4886c1b297aaa78db0e1ebc2d8dd6c" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "flash-linear-attention", version = "0.5.1", source = { registry = "https://pypi.org/simple" } }, { name = "mamba-ssm" }, { name = "nv-grouped-gemm" }, { name = "onnxscript" }, - { name = "transformer-engine", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "transformer-engine", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] [[package]] @@ -3826,12 +3512,12 @@ dependencies = [ { name = "anthropic" }, { name = "datasets" }, { name = "devtools" }, - { name = "fastapi", version = "0.136.3", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "fastapi", version = "0.139.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-mcore' or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, + { name = "fastapi", version = "0.136.3", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "fastapi", version = "0.139.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or extra == 'extra-7-nemo-rl-mcore' or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-trtllm' or extra != 'extra-7-nemo-rl-vllm'" }, { name = "fonttools" }, { name = "gitpython" }, - { name = "hydra-core", version = "1.3.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "hydra-core", version = "1.3.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or extra != 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "hydra-core", version = "1.3.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "hydra-core", version = "1.3.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or extra != 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "itsdangerous" }, { name = "mcp" }, { name = "mlflow" }, @@ -3857,59 +3543,6 @@ dependencies = [ { name = "yappi" }, ] -[package.optional-dependencies] -all = [ - { name = "boto3" }, - { name = "coverage" }, - { name = "daytona" }, - { name = "defusedxml" }, - { name = "gprof2dot" }, - { name = "httpx-aiohttp" }, - { name = "mypy" }, - { name = "opensandbox" }, - { name = "pre-commit" }, - { name = "pytest" }, - { name = "pytest-asyncio" }, - { name = "pytest-cov" }, - { name = "pytest-xdist" }, - { name = "requests-mock" }, - { name = "ruff" }, - { name = "tenacity" }, -] -dev = [ - { name = "coverage" }, - { name = "defusedxml" }, - { name = "gprof2dot" }, - { name = "mypy" }, - { name = "pre-commit" }, - { name = "pytest" }, - { name = "pytest-asyncio" }, - { name = "pytest-cov" }, - { name = "pytest-xdist" }, - { name = "requests-mock" }, - { name = "ruff" }, -] -openshell = [ - { name = "openshell" }, -] -sandbox = [ - { name = "boto3" }, - { name = "daytona" }, - { name = "httpx-aiohttp" }, - { name = "opensandbox" }, - { name = "tenacity" }, -] -vllm = [ - { name = "flashinfer-python", version = "0.6.12", source = { registry = "https://pypi.org/simple" } }, - { name = "vllm", version = "0.24.0", source = { registry = "https://pypi.org/simple" } }, -] - -[package.dev-dependencies] -codec = [ - { name = "torchaudio" }, - { name = "torchvision" }, -] - [[package]] name = "nemo-rl" source = { editable = "." } @@ -3922,8 +3555,8 @@ dependencies = [ { name = "datasets" }, { name = "debugpy" }, { name = "fastokens-b10" }, - { name = "hydra-core", version = "1.3.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "hydra-core", version = "1.3.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or extra != 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "hydra-core", version = "1.3.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "hydra-core", version = "1.3.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or extra != 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "math-verify" }, { name = "matplotlib" }, { name = "mlflow" }, @@ -3954,17 +3587,17 @@ dependencies = [ { name = "tensorboard" }, { name = "tensordict" }, { name = "tiktoken" }, - { name = "tilelang", version = "0.1.8", source = { registry = "https://pypi.org/simple" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "tilelang", version = "0.1.9", source = { registry = "https://pypi.org/simple" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "tilelang", version = "0.1.8", source = { registry = "https://pypi.org/simple" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "tilelang", version = "0.1.9", source = { registry = "https://pypi.org/simple" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "timm" }, { name = "torch" }, { name = "torchdata" }, { name = "torchvision" }, { name = "transferqueue" }, - { name = "transformers", version = "5.5.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "transformers", version = "5.5.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "transformers", version = "5.6.0", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "transformers", version = "5.8.1", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "transformers", version = "5.5.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "transformers", version = "5.5.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, + { name = "transformers", version = "5.6.0", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "transformers", version = "5.8.1", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "triton" }, { name = "wandb" }, { name = "zstandard" }, @@ -3973,35 +3606,35 @@ dependencies = [ [package.optional-dependencies] automodel = [ { name = "causal-conv1d" }, - { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=29d31c095796f3c8ece47ee9cdcc167051bbeed9#29d31c095796f3c8ece47ee9cdcc167051bbeed9" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=a48493600c4886c1b297aaa78db0e1ebc2d8dd6c#a48493600c4886c1b297aaa78db0e1ebc2d8dd6c" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "flash-attn", version = "2.8.1+cu13torch2.10cxx11abitrue", source = { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_aarch64.whl" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "flash-attn", version = "2.8.1+cu13torch2.10cxx11abitrue", source = { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_x86_64.whl" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=29d31c095796f3c8ece47ee9cdcc167051bbeed9#29d31c095796f3c8ece47ee9cdcc167051bbeed9" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=a48493600c4886c1b297aaa78db0e1ebc2d8dd6c#a48493600c4886c1b297aaa78db0e1ebc2d8dd6c" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "flash-attn", version = "2.8.1+cu13torch2.10cxx11abitrue", source = { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_aarch64.whl" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "flash-attn", version = "2.8.1+cu13torch2.10cxx11abitrue", source = { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_x86_64.whl" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "mamba-ssm" }, { name = "mistral-common" }, - { name = "nemo-automodel", extra = ["moe"], marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "nemo-automodel", extra = ["moe"], marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "nv-grouped-gemm" }, - { name = "transformer-engine", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "transformer-engine", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "transformers", version = "5.5.0", source = { registry = "https://pypi.org/simple" } }, ] fsdp = [ { name = "causal-conv1d" }, - { name = "flash-attn", version = "2.8.1+cu13torch2.10cxx11abitrue", source = { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_aarch64.whl" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'x86_64' and extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "flash-attn", version = "2.8.1+cu13torch2.10cxx11abitrue", source = { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_x86_64.whl" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "flash-attn", version = "2.8.1+cu13torch2.10cxx11abitrue", source = { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_aarch64.whl" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "flash-attn", version = "2.8.1+cu13torch2.10cxx11abitrue", source = { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_x86_64.whl" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "mamba-ssm" }, ] mcore = [ { name = "causal-conv1d" }, { name = "cupy-cuda13x" }, - { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=29d31c095796f3c8ece47ee9cdcc167051bbeed9#29d31c095796f3c8ece47ee9cdcc167051bbeed9" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=a48493600c4886c1b297aaa78db0e1ebc2d8dd6c#a48493600c4886c1b297aaa78db0e1ebc2d8dd6c" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "flash-attn", version = "2.8.1+cu13torch2.10cxx11abitrue", source = { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_aarch64.whl" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "flash-attn", version = "2.8.1+cu13torch2.10cxx11abitrue", source = { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_x86_64.whl" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=29d31c095796f3c8ece47ee9cdcc167051bbeed9#29d31c095796f3c8ece47ee9cdcc167051bbeed9" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=a48493600c4886c1b297aaa78db0e1ebc2d8dd6c#a48493600c4886c1b297aaa78db0e1ebc2d8dd6c" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "flash-attn", version = "2.8.1+cu13torch2.10cxx11abitrue", source = { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_aarch64.whl" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "flash-attn", version = "2.8.1+cu13torch2.10cxx11abitrue", source = { url = "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu13torch2.10cxx11abiTRUE-cp313-cp313-linux_x86_64.whl" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "flashinfer-cubin", version = "0.6.8.post1", source = { registry = "https://pypi.org/simple" } }, { name = "flashinfer-jit-cache", version = "0.6.8.post1+cu130", source = { registry = "https://flashinfer.ai/whl/cu130" } }, { name = "flashinfer-python", version = "0.6.8.post1", source = { registry = "https://pypi.org/simple" } }, { name = "mamba-ssm" }, - { name = "megatron-bridge", extra = ["ssm", "te"], marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "megatron-bridge", extra = ["ssm", "te"], marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "nvshmem4py-cu13" }, ] modelopt = [ @@ -4028,16 +3661,16 @@ trtllm = [ ] vllm = [ { name = "cuda-python" }, - { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=29d31c095796f3c8ece47ee9cdcc167051bbeed9#29d31c095796f3c8ece47ee9cdcc167051bbeed9" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=a48493600c4886c1b297aaa78db0e1ebc2d8dd6c#a48493600c4886c1b297aaa78db0e1ebc2d8dd6c" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=29d31c095796f3c8ece47ee9cdcc167051bbeed9#29d31c095796f3c8ece47ee9cdcc167051bbeed9" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "deep-ep", version = "1.2.1+29d31c0", source = { git = "https://github.com/deepseek-ai/DeepEP.git?rev=a48493600c4886c1b297aaa78db0e1ebc2d8dd6c#a48493600c4886c1b297aaa78db0e1ebc2d8dd6c" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "deep-gemm" }, { name = "flashinfer-cubin", version = "0.6.13", source = { registry = "https://pypi.org/simple" } }, { name = "flashinfer-jit-cache", version = "0.6.13+cu130", source = { registry = "https://flashinfer.ai/whl/cu130" } }, { name = "flashinfer-python", version = "0.6.13", source = { registry = "https://pypi.org/simple" } }, { name = "num2words" }, - { name = "nvidia-cutlass-dsl", version = "4.5.2", source = { registry = "https://pypi.org/simple" }, extra = ["cu13"], marker = "extra == 'extra-7-nemo-rl-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "vllm", version = "0.25.1", source = { url = "https://github.com/vllm-project/vllm/releases/download/v0.25.1/vllm-0.25.1-cp38-abi3-manylinux_2_28_aarch64.whl" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "vllm", version = "0.25.1", source = { url = "https://github.com/vllm-project/vllm/releases/download/v0.25.1/vllm-0.25.1-cp38-abi3-manylinux_2_28_x86_64.whl" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "nvidia-cutlass-dsl", version = "4.5.2", source = { registry = "https://pypi.org/simple" }, extra = ["cu13"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "vllm", version = "0.25.1", source = { url = "https://github.com/vllm-project/vllm/releases/download/v0.25.1/vllm-0.25.1-cp38-abi3-manylinux_2_28_aarch64.whl" }, marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "vllm", version = "0.25.1", source = { url = "https://github.com/vllm-project/vllm/releases/download/v0.25.1/vllm-0.25.1-cp38-abi3-manylinux_2_28_x86_64.whl" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, ] [package.dev-dependencies] @@ -4620,18 +4253,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/38/56/c7e8645061cc2fc23f3a54f33e1e340df59216f07dcfb97d46b8ae7dd26c/nvtx-0.2.15-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3741edac4678b92f03d22a3f0a2dfd469f422f85e63db71b038e02525b2404ad", size = 788639, upload-time = "2026-03-18T10:12:01.69Z" }, ] -[[package]] -name = "obstore" -version = "0.11.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/2e/2f/f83afaab7945509d72245b2b00af0b4834ce78fdd2d9ae9f0ad1a3036a91/obstore-0.11.0.tar.gz", hash = "sha256:a2f55163bcd348b4a60d12e6893eac50eddc742bad8032a1705d49140b992204", size = 130565, upload-time = "2026-06-25T18:29:49.405Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/5d/59/3f47822683ee2b6db8685faa25829946d6343a561251ec2704548455d946/obstore-0.11.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a2ff6d3ed553298828fb760b4aef6347fbcc7b5c5e3ce3f8381ce805c370021a", size = 5638724, upload-time = "2026-06-25T18:28:43.897Z" }, - { url = "https://files.pythonhosted.org/packages/23/50/1df335fdf9b527b3933f1e94ab6fc720ad314260fab8591cb0b6668ff192/obstore-0.11.0-cp311-abi3-manylinux_2_24_aarch64.whl", hash = "sha256:39d04b324fcf984e7050734ebda77b81764025b0c011750201a0d8954087f7aa", size = 5413508, upload-time = "2026-06-25T18:28:45.624Z" }, - { url = "https://files.pythonhosted.org/packages/de/dc/a259aba149b841ca7c91fea177df9972a60a636b54077beed1a35b254994/obstore-0.11.0-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:37c0d15d775b1370ef5204ee3919a5ddf7e2592d11815213105f8db031f2ab8d", size = 5619995, upload-time = "2026-06-25T18:28:47.599Z" }, - { url = "https://files.pythonhosted.org/packages/57/b7/577a965f440e9ea64243518663f9d16be7df8eafc7123818e8e841fa21ce/obstore-0.11.0-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:9c8fd2a544e2e0b926669c47fcfb8d2314e234abc240ea165dae04ee42e1d7ac", size = 5869187, upload-time = "2026-06-25T18:28:53.166Z" }, -] - [[package]] name = "omegaconf" version = "2.3.1" @@ -4790,36 +4411,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/10/68/162c97ea78c957d68ecf78a5c5041d2e25bd5562bdf5d89a6cbf7f8429bf/opencensus_context-0.1.3-py2.py3-none-any.whl", hash = "sha256:073bb0590007af276853009fac7e4bab1d523c3f03baf4cb4511ca38967c6039", size = 5060, upload-time = "2022-08-03T22:20:20.352Z" }, ] -[[package]] -name = "opensandbox" -version = "0.1.15" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "attrs" }, - { name = "httpx" }, - { name = "pydantic" }, - { name = "python-dateutil" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/43/21/654a3d69815b09690e926d553f3f4a178640d1206000a1b49f5e22c8eb68/opensandbox-0.1.15.tar.gz", hash = "sha256:017abc9b399b88da51bf077d6fb94ee89b1783e601495e85ba85fed478fed1b0", size = 228729, upload-time = "2026-07-24T09:36:29.988Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/d5/5c/ab87ea696531210790feb8f575471036fccba29dedaff201736f15bbb3a7/opensandbox-0.1.15-py3-none-any.whl", hash = "sha256:992b01490551f4d8e3f99caa25e34cb9d1690f0c5027eeebab912738291957d1", size = 538522, upload-time = "2026-07-24T09:36:28.277Z" }, -] - -[[package]] -name = "openshell" -version = "0.0.102" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "cloudpickle" }, - { name = "grpcio" }, - { name = "httpx" }, - { name = "protobuf" }, -] -wheels = [ - { url = "https://files.pythonhosted.org/packages/5d/4e/8fc4660a1cd57e92f4bc160234c29ef95ffef12b7b442f04a033b7cf7fe2/openshell-0.0.102-py3-none-manylinux_2_39_aarch64.whl", hash = "sha256:f9905a5e8fdab13300fac1a911619e3af3bab12e75059dd3b5bdba5067fbdc69", size = 8564345, upload-time = "2026-08-10T15:06:32.369Z" }, - { url = "https://files.pythonhosted.org/packages/f2/b7/6444a2320d75d244a872a00db479592e1c587cc44b8acd4461491eba80a7/openshell-0.0.102-py3-none-manylinux_2_39_x86_64.whl", hash = "sha256:0a4e9160e32ba8eabd28ac1eba1dff679a80b59e285b0a7b567fbcb41bde6a63", size = 9047752, upload-time = "2026-08-10T15:06:55.301Z" }, -] - [[package]] name = "opentelemetry-api" version = "1.44.0" @@ -4907,37 +4498,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/82/0d/d7cdc030edeed0fea03448446b88f1a1f8c4a565bad30dac0f5477ebe290/opentelemetry_exporter_prometheus-0.65b0-py3-none-any.whl", hash = "sha256:3b3d24b586d0ad9712c7b52b7d19c8a9dfbb318b9b284121b5f95e90ed019367", size = 13031, upload-time = "2026-07-16T15:25:20.906Z" }, ] -[[package]] -name = "opentelemetry-instrumentation" -version = "0.65b0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "opentelemetry-api" }, - { name = "opentelemetry-semantic-conventions" }, - { name = "packaging" }, - { name = "wrapt" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/13/91/3c58961cb0360cd60509064734f0be4275383c8681d73c580a40ca83ddce/opentelemetry_instrumentation-0.65b0.tar.gz", hash = "sha256:071d9d9eced9bd6460444ec3b0c77229870ed05a881c22c84fdede58e4eed09b", size = 42689, upload-time = "2026-07-16T15:25:50.275Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/40/7b/85eab1215f72adf0e68d3dc4a679b9bff993fa679ff34cd8dd378e2659fd/opentelemetry_instrumentation-0.65b0-py3-none-any.whl", hash = "sha256:ea967a72b9939b5fcfdad572753b4306c59dcb99e3f382d95dae04286805e137", size = 36717, upload-time = "2026-07-16T15:24:51.424Z" }, -] - -[[package]] -name = "opentelemetry-instrumentation-aiohttp-client" -version = "0.65b0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "opentelemetry-api" }, - { name = "opentelemetry-instrumentation" }, - { name = "opentelemetry-semantic-conventions" }, - { name = "opentelemetry-util-http" }, - { name = "wrapt" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/a9/33/3ff7230b035e8b696db6be54f5c52dfa409829d634d91431c076ad789820/opentelemetry_instrumentation_aiohttp_client-0.65b0.tar.gz", hash = "sha256:85906a2806ee5641756b5c33274e9aa75c3cc2441e3b830aa5804cf0e1fa9dd1", size = 19042, upload-time = "2026-07-16T15:25:51.632Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/71/f9/5c8459224f175829601cabbcffaeab0f9041903c086e4a0368f9971093a5/opentelemetry_instrumentation_aiohttp_client-0.65b0-py3-none-any.whl", hash = "sha256:3a060efa53fa44d02ba7372a7ed2b42cdfa6be6df81b089845067ad840e25729", size = 13677, upload-time = "2026-07-16T15:24:53.361Z" }, -] - [[package]] name = "opentelemetry-proto" version = "1.44.0" @@ -4990,15 +4550,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/55/22/41fb05f1dc5fda2c468e05a41814c20859016c85117b66c8a257cae814f6/opentelemetry_semantic_conventions_ai-0.5.1-py3-none-any.whl", hash = "sha256:25aeb22bd261543b4898a73824026d96770e5351209c7d07a0b1314762b1f6e4", size = 11250, upload-time = "2026-03-26T14:20:37.108Z" }, ] -[[package]] -name = "opentelemetry-util-http" -version = "0.65b0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/32/a9/d7525a59fdd240e69b5af4a6338e78fafa1b4203394122cbd6701fb5f84a/opentelemetry_util_http-0.65b0.tar.gz", hash = "sha256:84f82d826978bba416ab453460ff6a7391cdc3534c93a786595e4068680016b7", size = 11243, upload-time = "2026-07-16T15:26:27.898Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/23/3f/ab8d29df207ce5f470a07fa96ebb48af4e95b7fab7e7635311b9a32f2fab/opentelemetry_util_http-0.65b0-py3-none-any.whl", hash = "sha256:7553b606f963097cb190536dc30556cce85090692e471a422fff30ca29b04348", size = 8245, upload-time = "2026-07-16T15:25:46.482Z" }, -] - [[package]] name = "optimum" version = "2.2.0" @@ -5084,8 +4635,8 @@ dependencies = [ { name = "numpy" }, { name = "python-dateutil" }, { name = "pytz" }, - { name = "tzdata", version = "2025.3", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "tzdata", version = "2026.3", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or extra != 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "tzdata", version = "2025.3", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "tzdata", version = "2026.3", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or extra != 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/33/01/d40b85317f86cf08d853a4f495195c73815fdf205eef3993821720274518/pandas-2.3.3.tar.gz", hash = "sha256:e05e1af93b977f7eafa636d043f9f94c7ee3ac81af99c13508215942e64c993b", size = 4495223, upload-time = "2025-09-29T23:34:51.853Z" } wheels = [ @@ -5140,8 +4691,8 @@ dependencies = [ { name = "safetensors" }, { name = "torch" }, { name = "tqdm" }, - { name = "transformers", version = "5.5.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "transformers", version = "5.8.1", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "transformers", version = "5.5.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, + { name = "transformers", version = "5.8.1", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/86/cf/037f1e3d5186496c05513a6754639e2dab3038a05f384284d49a9bd06a2d/peft-0.19.1.tar.gz", hash = "sha256:0d97542fe96dcdaa20d3b81c06f26f988618f416a73544ab23c3618ccb674a40", size = 763738, upload-time = "2026-04-16T15:46:45.105Z" } wheels = [ @@ -5775,19 +5326,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fa/b6/3127540ecdf1464a00e5a01ee60a1b09175f6913f0644ac748494d9c4b21/pytest_timeout-2.4.0-py3-none-any.whl", hash = "sha256:c42667e5cdadb151aeb5b26d114aff6bdf5a907f176a007a30b940d3d865b5c2", size = 14382, upload-time = "2025-05-05T19:44:33.502Z" }, ] -[[package]] -name = "pytest-xdist" -version = "3.8.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "execnet" }, - { name = "pytest" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/78/b4/439b179d1ff526791eb921115fca8e44e596a13efeda518b9d845a619450/pytest_xdist-3.8.0.tar.gz", hash = "sha256:7e578125ec9bc6050861aa93f2d59f1d8d085595d6551c2c90b6f4fad8d3a9f1", size = 88069, upload-time = "2025-07-01T13:30:59.346Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/ca/31/d4e37e9e550c2b92a9cbc2e4d0b7420a27224968580b5a447f420847c975/pytest_xdist-3.8.0-py3-none-any.whl", hash = "sha256:202ca578cfeb7370784a8c33d6d05bc6e13b4f25b5053c30a152269fd10f0b88", size = 46396, upload-time = "2025-07-01T13:30:56.632Z" }, -] - [[package]] name = "python-box" version = "6.1.0" @@ -5831,18 +5369,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0b/d7/1959b9648791274998a9c3526f6d0ec8fd2233e4d4acce81bbae76b44b2a/python_dotenv-1.2.2-py3-none-any.whl", hash = "sha256:1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a", size = 22101, upload-time = "2026-03-01T16:00:25.09Z" }, ] -[[package]] -name = "python-engineio" -version = "4.13.3" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "simple-websocket" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/fb/a0/f75491f942184d9960b15e763270f765fe9f239745ca5f9e16289011aed4/python_engineio-4.13.3.tar.gz", hash = "sha256:572b7783e341fed21edbc7cea297ccd378dad79265fdde96aa4664420a7c06c9", size = 79734, upload-time = "2026-06-20T22:53:52.197Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/5b/96/82f6328e410515fab21d5602ba35b9377a47b5a141a0c1f9efa00ce21eb4/python_engineio-4.13.3-py3-none-any.whl", hash = "sha256:1f60ecaf1358190f0e26c48c578a60428dc02a8f1295bc3dbf53d1b31116821f", size = 59993, upload-time = "2026-06-20T22:53:50.775Z" }, -] - [[package]] name = "python-json-logger" version = "4.1.0" @@ -5861,28 +5387,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e1/04/e8135ebd1ad02c56ec633277529b2602ff99ff634be76cdba5744cf554fd/python_multipart-0.0.32-py3-none-any.whl", hash = "sha256:ff6d3f776f16878c894e52e107296ffc890e913c611b1a4ec6c44e2821fe2e23", size = 30042, upload-time = "2026-06-04T16:18:57.319Z" }, ] -[[package]] -name = "python-socketio" -version = "5.16.3" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "bidict" }, - { name = "python-engineio" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/32/2d/ffce71017c106b75099fea569df6518c63fee5d6202ce0cfe7b01e6f22c3/python_socketio-5.16.3.tar.gz", hash = "sha256:89b136f677ae65607a84cecda9b4d6c5377b40a97582c504c25df89af16d520e", size = 128095, upload-time = "2026-06-15T22:07:04.003Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/0a/38/8c5e72d53ff8eb27497c4f268a7f6d9121e727a50b65248288ad79a93053/python_socketio-5.16.3-py3-none-any.whl", hash = "sha256:e7ad14202a5e6448824c7c2f86161d04e13dec05992257df5c709e6a2798c041", size = 82087, upload-time = "2026-06-15T22:07:02.498Z" }, -] - -[package.optional-dependencies] -asyncio-client = [ - { name = "aiohttp" }, -] -client = [ - { name = "requests" }, - { name = "websocket-client" }, -] - [[package]] name = "pytz" version = "2026.2" @@ -5918,7 +5422,7 @@ name = "pyzmq" version = "27.1.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "cffi", marker = "implementation_name == 'pypy' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "cffi", marker = "implementation_name == 'pypy' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/04/0b/3c9baedbdf613ecaa7aa07027780b8867f57b6293b6ee50de316c9f3222b/pyzmq-27.1.0.tar.gz", hash = "sha256:ac0765e3d44455adb6ddbf4417dcce460fc40a05978c08efdf2948072f6db540", size = 281750, upload-time = "2025-09-08T23:10:18.157Z" } wheels = [ @@ -5939,12 +5443,12 @@ source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "apache-tvm-ffi" }, { name = "einops" }, - { name = "nvidia-cutlass-dsl", version = "4.5.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "nvidia-cutlass-dsl", version = "4.5.1", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "nvidia-cutlass-dsl", version = "4.5.2", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "nvidia-cutlass-dsl", version = "4.5.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, + { name = "nvidia-cutlass-dsl", version = "4.5.1", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "nvidia-cutlass-dsl", version = "4.5.2", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "torch" }, - { name = "torch-c-dlpack-ext", version = "0.1.3", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "torch-c-dlpack-ext", version = "0.1.5", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "torch-c-dlpack-ext", version = "0.1.3", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, + { name = "torch-c-dlpack-ext", version = "0.1.5", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/2a/58/58b82e91b236539f424ff5681e7095b1f2860ddfb7778fe0be14d8fb58de/quack_kernels-0.4.1.tar.gz", hash = "sha256:9d7d6ba412bc0c8a9b1331c52a73db76280adb9dc2f2750df4851ddabef1466b", size = 274766, upload-time = "2026-04-30T14:37:55.65Z" } wheels = [ @@ -6079,18 +5583,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", size = 73075, upload-time = "2026-05-14T19:25:26.443Z" }, ] -[[package]] -name = "requests-mock" -version = "1.12.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "requests" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/92/32/587625f91f9a0a3d84688bf9cfc4b2480a7e8ec327cefd0ff2ac891fd2cf/requests-mock-1.12.1.tar.gz", hash = "sha256:e9e12e333b525156e82a3c852f22016b9158220d2f47454de9cae8a77d371401", size = 60901, upload-time = "2024-03-29T03:54:29.446Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/97/ec/889fbc557727da0c34a33850950310240f2040f3b1955175fdb2b36a8910/requests_mock-1.12.1-py2.py3-none-any.whl", hash = "sha256:b1e37054004cdd5e56c84454cc7df12b25f90f382159087f4b6915aaeef39563", size = 27695, upload-time = "2024-03-29T03:54:27.64Z" }, -] - [[package]] name = "requests-toolbelt" version = "1.0.0" @@ -6345,7 +5837,7 @@ dependencies = [ { name = "msgspec" }, { name = "ninja" }, { name = "numpy" }, - { name = "nvidia-cutlass-dsl", version = "4.5.1", source = { registry = "https://pypi.org/simple" }, extra = ["cu13"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "nvidia-cutlass-dsl", version = "4.5.1", source = { registry = "https://pypi.org/simple" }, extra = ["cu13"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "nvidia-ml-py" }, { name = "openai" }, { name = "openai-harmony" }, @@ -6429,18 +5921,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl", hash = "sha256:7ecfff8f2fd72616f7481040475a65b2bf8af90a56c89140852d1120324e8686", size = 9755, upload-time = "2023-10-24T04:13:38.866Z" }, ] -[[package]] -name = "simple-websocket" -version = "1.1.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "wsproto" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/b0/d4/bfa032f961103eba93de583b161f0e6a5b63cebb8f2c7d0c6e6efe1e3d2e/simple_websocket-1.1.0.tar.gz", hash = "sha256:7939234e7aa067c534abdab3a9ed933ec9ce4691b0713c78acb195560aa52ae4", size = 17300, upload-time = "2024-10-10T22:39:31.412Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl", hash = "sha256:4af6069630a38ed6c561010f0e11a5bc0d4ca569b36306eb257cd9a192497c8c", size = 13842, upload-time = "2024-10-10T22:39:29.645Z" }, -] - [[package]] name = "simplejson" version = "4.1.1" @@ -6937,15 +6417,6 @@ test = [ { name = "pytest-timeout" }, ] -[[package]] -name = "tenacity" -version = "9.1.4" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/47/c6/ee486fd809e357697ee8a44d3d69222b344920433d3b6666ccd9b374630c/tenacity-9.1.4.tar.gz", hash = "sha256:adb31d4c263f2bd041081ab33b498309a57c77f9acf2db65aadf0898179cf93a", size = 49413, upload-time = "2026-02-07T10:45:33.841Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/d7/c1/eb8f9debc45d3b7918a32ab756658a0904732f75e555402972246b0b8e71/tenacity-9.1.4-py3-none-any.whl", hash = "sha256:6095a360c919085f28c6527de529e76a06ad89b23659fa881ae0649b867a9d55", size = 28926, upload-time = "2026-02-07T10:45:32.24Z" }, -] - [[package]] name = "tensorboard" version = "2.21.0" @@ -7067,9 +6538,9 @@ dependencies = [ { name = "numpy" }, { name = "nvidia-cuda-nvrtc" }, { name = "nvidia-cuda-tileiras" }, - { name = "nvidia-cutlass-dsl", version = "4.5.0", source = { registry = "https://pypi.org/simple" }, extra = ["cu13"], marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, + { name = "nvidia-cutlass-dsl", version = "4.5.0", source = { registry = "https://pypi.org/simple" }, extra = ["cu13"], marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, { name = "nvidia-ml-py" }, - { name = "nvidia-modelopt", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, + { name = "nvidia-modelopt", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, { name = "nvidia-nccl-cu13" }, { name = "nvtx" }, { name = "omegaconf" }, @@ -7089,7 +6560,7 @@ dependencies = [ { name = "psutil" }, { name = "pulp" }, { name = "pydantic" }, - { name = "pydantic-settings", extra = ["yaml"], marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, + { name = "pydantic-settings", extra = ["yaml"], marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, { name = "python-multipart" }, { name = "pyzmq" }, { name = "sentencepiece" }, @@ -7161,17 +6632,17 @@ resolution-markers = [ "platform_machine == 'aarch64' and sys_platform == 'linux'", ] dependencies = [ - { name = "apache-tvm-ffi", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "cloudpickle", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "ml-dtypes", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "numpy", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "psutil", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "setuptools", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "torch", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "torch-c-dlpack-ext", version = "0.1.5", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "tqdm", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "typing-extensions", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "z3-solver", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "apache-tvm-ffi", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "cloudpickle", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "ml-dtypes", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "numpy", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "psutil", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "setuptools", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "torch", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "torch-c-dlpack-ext", version = "0.1.5", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "tqdm", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "typing-extensions", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "z3-solver", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/e6/27/6e363f48f878389078e2899756b8fecc326388b585122fd7f8a86590dfab/tilelang-0.1.8.tar.gz", hash = "sha256:da967821698eb7a79a76d27fbe25e314a3273f2b12ba4833e981658139d0e6d9", size = 93247335, upload-time = "2026-02-16T14:03:28.706Z" } wheels = [ @@ -7188,18 +6659,18 @@ resolution-markers = [ "platform_machine == 'aarch64' and sys_platform == 'linux'", ] dependencies = [ - { name = "apache-tvm-ffi", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or extra == 'extra-8-nemo-gym-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, - { name = "cloudpickle", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or extra == 'extra-8-nemo-gym-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, - { name = "ml-dtypes", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or extra == 'extra-8-nemo-gym-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, - { name = "numpy", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or extra == 'extra-8-nemo-gym-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, - { name = "psutil", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or extra == 'extra-8-nemo-gym-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, - { name = "setuptools", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or extra == 'extra-8-nemo-gym-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, - { name = "torch", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or extra == 'extra-8-nemo-gym-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, - { name = "torch-c-dlpack-ext", version = "0.1.3", source = { registry = "https://pypi.org/simple" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-trtllm') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "torch-c-dlpack-ext", version = "0.1.5", source = { registry = "https://pypi.org/simple" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or extra == 'extra-7-nemo-rl-vllm' or extra == 'extra-8-nemo-gym-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, - { name = "tqdm", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or extra == 'extra-8-nemo-gym-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, - { name = "typing-extensions", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or extra == 'extra-8-nemo-gym-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, - { name = "z3-solver", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or extra == 'extra-8-nemo-gym-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, + { name = "apache-tvm-ffi", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, + { name = "cloudpickle", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, + { name = "ml-dtypes", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, + { name = "numpy", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, + { name = "psutil", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, + { name = "setuptools", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, + { name = "torch", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, + { name = "torch-c-dlpack-ext", version = "0.1.3", source = { registry = "https://pypi.org/simple" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-trtllm') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-vllm') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "torch-c-dlpack-ext", version = "0.1.5", source = { registry = "https://pypi.org/simple" }, marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or extra == 'extra-7-nemo-rl-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, + { name = "tqdm", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, + { name = "typing-extensions", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, + { name = "z3-solver", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-sglang') or extra == 'extra-7-nemo-rl-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/56/70/5051f65821baa30a3d61fc48f8ba10c776490315e8c90f82559b92089756/tilelang-0.1.9.tar.gz", hash = "sha256:287f727c913bb648fcf6c1968809ba3390e55eeed257a5c6bb9a80bc05966af4", size = 93395292, upload-time = "2026-04-22T09:19:11.988Z" } wheels = [ @@ -7285,15 +6756,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/73/cf/6d0bb055e0fff9ef9284ec5455ea5f4d8d2e76f89cc4c853f7be69fdeef8/tokenspeed_triton-3.8.10.post20260709-cp312-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:15040e5514671ceb05a22dbead49030cf87ac3becb47361a67ed2f779433d04d", size = 87150923, upload-time = "2026-07-10T15:29:07.961Z" }, ] -[[package]] -name = "toml" -version = "0.10.2" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/be/ba/1f744cdc819428fc6b5084ec34d9b30660f6f9daaf70eead706e3203ec3c/toml-0.10.2.tar.gz", hash = "sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f", size = 22253, upload-time = "2020-11-01T01:40:22.204Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", hash = "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b", size = 16588, upload-time = "2020-11-01T01:40:20.672Z" }, -] - [[package]] name = "torch" version = "2.11.0+cu130" @@ -7330,7 +6792,7 @@ resolution-markers = [ "platform_machine == 'aarch64' and sys_platform == 'linux'", ] dependencies = [ - { name = "torch", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, + { name = "torch", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/34/1a/adafdc49546ffe0041f52471da51c084243094382b99332b5706e90ba26a/torch_c_dlpack_ext-0.1.3.tar.gz", hash = "sha256:4b5da66432af7224dcf02aad4f13cc416eeef5331cd153588b7e081a193f4972", size = 3334, upload-time = "2025-11-22T03:50:48.089Z" } wheels = [ @@ -7347,7 +6809,7 @@ resolution-markers = [ "platform_machine == 'aarch64' and sys_platform == 'linux'", ] dependencies = [ - { name = "torch", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-vllm' or extra == 'extra-8-nemo-gym-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm')" }, + { name = "torch", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-mcore') or extra == 'extra-7-nemo-rl-sglang' or extra == 'extra-7-nemo-rl-vllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/37/de/921b6491efce5c389a5ef9bbed3d2d6660005840dae488124173180859ab/torch_c_dlpack_ext-0.1.5.tar.gz", hash = "sha256:d06f0357d575d22a168cc77acb9020fc4bae30968ceb6718a055dcbe92bacabe", size = 12913, upload-time = "2026-01-12T11:25:08.484Z" } wheels = [ @@ -7479,8 +6941,8 @@ name = "transferqueue" version = "0.1.9" source = { git = "https://github.com/Ascend/TransferQueue.git?rev=c51614308b68c8d7a87c9b3ef62d59e14c69bde2#c51614308b68c8d7a87c9b3ef62d59e14c69bde2" } dependencies = [ - { name = "hydra-core", version = "1.3.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "hydra-core", version = "1.3.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or extra != 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "hydra-core", version = "1.3.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "hydra-core", version = "1.3.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-automodel' or extra != 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "msgspec" }, { name = "numpy" }, { name = "omegaconf" }, @@ -7511,15 +6973,15 @@ resolution-markers = [ "platform_machine == 'aarch64' and sys_platform == 'linux'", ] dependencies = [ - { name = "huggingface-hub", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "numpy", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "packaging", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "pyyaml", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "regex", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "safetensors", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "tokenizers", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "tqdm", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "typer", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "huggingface-hub", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "numpy", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "packaging", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "pyyaml", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "regex", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "safetensors", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "tokenizers", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "tqdm", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "typer", marker = "extra == 'extra-7-nemo-rl-automodel' or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/ff/9d/fb46e729b461985f41a5740167688b924a4019141e5c164bea77548d3d9e/transformers-5.5.0.tar.gz", hash = "sha256:c8db656cf51c600cd8c75f06b20ef85c72e8b8ff9abc880c5d3e8bc70e0ddcbd", size = 8237745, upload-time = "2026-04-02T16:13:08.113Z" } wheels = [ @@ -7535,15 +6997,15 @@ resolution-markers = [ "platform_machine == 'aarch64' and sys_platform == 'linux'", ] dependencies = [ - { name = "huggingface-hub", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "numpy", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "packaging", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "pyyaml", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "regex", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "safetensors", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "tokenizers", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "tqdm", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "typer", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, + { name = "huggingface-hub", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, + { name = "numpy", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, + { name = "packaging", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, + { name = "pyyaml", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, + { name = "regex", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, + { name = "safetensors", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, + { name = "tokenizers", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, + { name = "tqdm", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, + { name = "typer", marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/a5/1e/1e244ab2ab50a863e6b52cc55761910567fa532b69a6740f6e99c5fdbd98/transformers-5.5.4.tar.gz", hash = "sha256:2e67cadba81fc7608cc07c4dd54f524820bc3d95b1cabd0ef3db7733c4f8b82e", size = 8227649, upload-time = "2026-04-13T16:55:55.181Z" } wheels = [ @@ -7559,15 +7021,15 @@ resolution-markers = [ "platform_machine == 'aarch64' and sys_platform == 'linux'", ] dependencies = [ - { name = "huggingface-hub", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "numpy", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "packaging", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "pyyaml", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "regex", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "safetensors", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "tokenizers", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "tqdm", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "typer", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "huggingface-hub", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "numpy", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "packaging", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "pyyaml", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "regex", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "safetensors", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "tokenizers", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "tqdm", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "typer", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/88/14/e3eb58bd4c9df45af154f9de59a6e66a2ece30dd64434c710e40e5a4b1f1/transformers-5.6.0.tar.gz", hash = "sha256:291951976b79a6f93ec06d6ab14489a99aecdbc8f05aaabab538ea1d508c9a97", size = 8311711, upload-time = "2026-04-22T15:42:03.393Z" } wheels = [ @@ -7583,15 +7045,15 @@ resolution-markers = [ "platform_machine == 'aarch64' and sys_platform == 'linux'", ] dependencies = [ - { name = "huggingface-hub", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "numpy", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "packaging", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "pyyaml", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "regex", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "safetensors", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "tokenizers", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "tqdm", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "typer", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "huggingface-hub", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "numpy", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "packaging", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "pyyaml", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "regex", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "safetensors", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "tokenizers", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "tqdm", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "typer", marker = "extra == 'extra-7-nemo-rl-mcore' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/e7/e6/4134ea2fbea322cddc7ffc94a0d8ee47fe32ce8e876b320cd37d88edfc4d/transformers-5.8.1.tar.gz", hash = "sha256:4dd5b6de4105725104d84fd6abd74b305f4debfc251b38c648ee5dd087cf543b", size = 8532019, upload-time = "2026-05-13T03:21:57.234Z" } wheels = [ @@ -7759,93 +7221,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/c1/7c/4e7225d46d634a0d8d534dd8a6ce0c319d09b4d0cf0337eb314ca4789d8c/virtualenv-21.6.1-py3-none-any.whl", hash = "sha256:afe991df855715a2b2f60edfcc0107ef95a79fdfd8cb4cdaa71603d1c12e463b", size = 5506392, upload-time = "2026-07-10T19:33:51.629Z" }, ] -[[package]] -name = "vllm" -version = "0.24.0" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "platform_machine == 'x86_64' and sys_platform == 'linux'", - "platform_machine == 'aarch64' and sys_platform == 'linux'", -] -dependencies = [ - { name = "aiohttp" }, - { name = "anthropic" }, - { name = "apache-tvm-ffi" }, - { name = "blake3" }, - { name = "cachetools" }, - { name = "cbor2" }, - { name = "cloudpickle" }, - { name = "compressed-tensors", version = "0.17.0", source = { registry = "https://pypi.org/simple" } }, - { name = "depyf" }, - { name = "diskcache" }, - { name = "einops" }, - { name = "fastapi", version = "0.136.3", source = { registry = "https://pypi.org/simple" }, extra = ["standard"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "fastsafetensors" }, - { name = "filelock" }, - { name = "flashinfer-cubin", version = "0.6.12", source = { registry = "https://pypi.org/simple" } }, - { name = "flashinfer-python", version = "0.6.12", source = { registry = "https://pypi.org/simple" } }, - { name = "humming-kernels", version = "0.1.6", source = { registry = "https://pypi.org/simple" }, extra = ["cu13"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "ijson" }, - { name = "jsonschema" }, - { name = "lark" }, - { name = "llguidance" }, - { name = "lm-format-enforcer" }, - { name = "mcp" }, - { name = "mistral-common", marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "model-hosting-container-standards" }, - { name = "msgspec" }, - { name = "ninja" }, - { name = "numba" }, - { name = "numpy" }, - { name = "nvidia-cudnn-frontend" }, - { name = "nvidia-cutlass-dsl", version = "4.5.2", source = { registry = "https://pypi.org/simple" }, extra = ["cu13"], marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "openai" }, - { name = "openai-harmony" }, - { name = "opentelemetry-api" }, - { name = "opentelemetry-exporter-otlp" }, - { name = "opentelemetry-sdk" }, - { name = "opentelemetry-semantic-conventions-ai" }, - { name = "outlines-core" }, - { name = "partial-json-parser" }, - { name = "pillow" }, - { name = "prometheus-client" }, - { name = "prometheus-fastapi-instrumentator" }, - { name = "protobuf" }, - { name = "psutil" }, - { name = "py-cpuinfo" }, - { name = "pybase64" }, - { name = "pydantic" }, - { name = "python-json-logger" }, - { name = "pyyaml" }, - { name = "pyzmq" }, - { name = "quack-kernels" }, - { name = "regex" }, - { name = "requests" }, - { name = "safetensors" }, - { name = "sentencepiece" }, - { name = "setproctitle" }, - { name = "setuptools" }, - { name = "six" }, - { name = "starlette" }, - { name = "tiktoken" }, - { name = "tilelang", version = "0.1.9", source = { registry = "https://pypi.org/simple" } }, - { name = "tokenizers" }, - { name = "tokenspeed-mla", version = "0.1.2", source = { registry = "https://pypi.org/simple" } }, - { name = "torch" }, - { name = "torchaudio" }, - { name = "torchvision" }, - { name = "tqdm" }, - { name = "transformers", version = "5.8.1", source = { registry = "https://pypi.org/simple" } }, - { name = "typing-extensions" }, - { name = "watchfiles" }, - { name = "xgrammar" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/d4/17/ea541f9ffb31d438ed6a5a8e1f7b9805410664abf97e326f28416147a5da/vllm-0.24.0.tar.gz", hash = "sha256:0862453adc1f3339f1a0c9dca1179c34d6ed6e118f87b6e5bddd120af614ac66", size = 37236989, upload-time = "2026-06-30T01:18:35.996Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/9e/80/51a071305b4eed0f6f512dc1c1c6957cbb14ccce38db1be90ffcff2a2844/vllm-0.24.0-cp38-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:700db71c3cf14697d42583521f38b12fac38db1e7a8ad062e8e4d63a5dadebd5", size = 271361241, upload-time = "2026-06-30T01:17:52.566Z" }, - { url = "https://files.pythonhosted.org/packages/00/33/3f0abda52acff437a471cf3a2bf204213eb4102975b2677512cd76f2b45e/vllm-0.24.0-cp38-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:2d2831aeba311292250df0132dbc4d8e9f42c654536eaec48e6fe58acb1822cf", size = 279209310, upload-time = "2026-06-30T01:18:18.504Z" }, -] - [[package]] name = "vllm" version = "0.25.1" @@ -7865,26 +7240,26 @@ dependencies = [ { name = "depyf", marker = "platform_machine != 'x86_64'" }, { name = "diskcache", marker = "platform_machine != 'x86_64'" }, { name = "einops", marker = "platform_machine != 'x86_64'" }, - { name = "fastapi", version = "0.136.3", source = { registry = "https://pypi.org/simple" }, extra = ["standard"], marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "fastapi", version = "0.136.3", source = { registry = "https://pypi.org/simple" }, extra = ["standard"], marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "fastsafetensors", marker = "platform_machine != 'x86_64'" }, { name = "filelock", marker = "platform_machine != 'x86_64'" }, { name = "flashinfer-cubin", version = "0.6.13", source = { registry = "https://pypi.org/simple" }, marker = "platform_machine != 'x86_64'" }, { name = "flashinfer-python", version = "0.6.13", source = { registry = "https://pypi.org/simple" }, marker = "platform_machine != 'x86_64'" }, - { name = "humming-kernels", version = "0.1.10", source = { registry = "https://pypi.org/simple" }, extra = ["cu13"], marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "humming-kernels", extra = ["cu13"], marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "ijson", marker = "platform_machine != 'x86_64'" }, { name = "jsonschema", marker = "platform_machine != 'x86_64'" }, { name = "lark", marker = "platform_machine != 'x86_64'" }, { name = "llguidance", marker = "platform_machine != 'x86_64'" }, { name = "lm-format-enforcer", marker = "platform_machine != 'x86_64'" }, { name = "mcp", marker = "platform_machine != 'x86_64'" }, - { name = "mistral-common", marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "mistral-common", marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "model-hosting-container-standards", marker = "platform_machine != 'x86_64'" }, { name = "msgspec", marker = "platform_machine != 'x86_64'" }, { name = "ninja", marker = "platform_machine != 'x86_64'" }, { name = "numba", marker = "platform_machine != 'x86_64'" }, { name = "numpy", marker = "platform_machine != 'x86_64'" }, { name = "nvidia-cudnn-frontend", marker = "platform_machine != 'x86_64'" }, - { name = "nvidia-cutlass-dsl", version = "4.5.2", source = { registry = "https://pypi.org/simple" }, extra = ["cu13"], marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "nvidia-cutlass-dsl", version = "4.5.2", source = { registry = "https://pypi.org/simple" }, extra = ["cu13"], marker = "(platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "nvtx", marker = "platform_machine != 'x86_64'" }, { name = "openai", marker = "platform_machine != 'x86_64'" }, { name = "openai-harmony", marker = "platform_machine != 'x86_64'" }, @@ -7952,26 +7327,26 @@ dependencies = [ { name = "depyf", marker = "platform_machine == 'x86_64'" }, { name = "diskcache", marker = "platform_machine == 'x86_64'" }, { name = "einops", marker = "platform_machine == 'x86_64'" }, - { name = "fastapi", version = "0.136.3", source = { registry = "https://pypi.org/simple" }, extra = ["standard"], marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "fastapi", version = "0.136.3", source = { registry = "https://pypi.org/simple" }, extra = ["standard"], marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "fastsafetensors", marker = "platform_machine == 'x86_64'" }, { name = "filelock", marker = "platform_machine == 'x86_64'" }, { name = "flashinfer-cubin", version = "0.6.13", source = { registry = "https://pypi.org/simple" }, marker = "platform_machine == 'x86_64'" }, { name = "flashinfer-python", version = "0.6.13", source = { registry = "https://pypi.org/simple" }, marker = "platform_machine == 'x86_64'" }, - { name = "humming-kernels", version = "0.1.10", source = { registry = "https://pypi.org/simple" }, extra = ["cu13"], marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "humming-kernels", extra = ["cu13"], marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "ijson", marker = "platform_machine == 'x86_64'" }, { name = "jsonschema", marker = "platform_machine == 'x86_64'" }, { name = "lark", marker = "platform_machine == 'x86_64'" }, { name = "llguidance", marker = "platform_machine == 'x86_64'" }, { name = "lm-format-enforcer", marker = "platform_machine == 'x86_64'" }, { name = "mcp", marker = "platform_machine == 'x86_64'" }, - { name = "mistral-common", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "mistral-common", marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "model-hosting-container-standards", marker = "platform_machine == 'x86_64'" }, { name = "msgspec", marker = "platform_machine == 'x86_64'" }, { name = "ninja", marker = "platform_machine == 'x86_64'" }, { name = "numba", marker = "platform_machine == 'x86_64'" }, { name = "numpy", marker = "platform_machine == 'x86_64'" }, { name = "nvidia-cudnn-frontend", marker = "platform_machine == 'x86_64'" }, - { name = "nvidia-cutlass-dsl", version = "4.5.2", source = { registry = "https://pypi.org/simple" }, extra = ["cu13"], marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "nvidia-cutlass-dsl", version = "4.5.2", source = { registry = "https://pypi.org/simple" }, extra = ["cu13"], marker = "(platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "nvtx", marker = "platform_machine == 'x86_64'" }, { name = "openai", marker = "platform_machine == 'x86_64'" }, { name = "openai-harmony", marker = "platform_machine == 'x86_64'" }, @@ -8097,15 +7472,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d9/00/aca6beb3658dab4ed3dbb41a78e6e7f31342e0b41d28088f205525751601/webdataset-1.0.2-py3-none-any.whl", hash = "sha256:3dbfced32b25c0d199c6b9787937b6f85742bc3c84f652c846893075c1c082d9", size = 74956, upload-time = "2025-06-19T23:26:20.354Z" }, ] -[[package]] -name = "websocket-client" -version = "1.9.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/2c/41/aa4bf9664e4cda14c3b39865b12251e8e7d239f4cd0e3cc1b6c2ccde25c1/websocket_client-1.9.0.tar.gz", hash = "sha256:9e813624b6eb619999a97dc7958469217c3176312b3a16a4bd1bc7e08a46ec98", size = 70576, upload-time = "2025-10-07T21:16:36.495Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/34/db/b10e48aa8fff7407e67470363eac595018441cf32d5e1001567a7aeba5d2/websocket_client-1.9.0-py3-none-any.whl", hash = "sha256:af248a825037ef591efbf6ed20cc5faa03d3b47b9e5a2230a529eeee1c1fc3ef", size = 82616, upload-time = "2025-10-07T21:16:34.951Z" }, -] - [[package]] name = "websockets" version = "16.1.1" @@ -8225,9 +7591,9 @@ dependencies = [ { name = "numpy" }, { name = "pydantic" }, { name = "torch" }, - { name = "transformers", version = "5.5.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm' and extra != 'extra-8-nemo-gym-vllm')" }, - { name = "transformers", version = "5.6.0", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm' and extra == 'extra-8-nemo-gym-vllm')" }, - { name = "transformers", version = "5.8.1", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-8-nemo-gym-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-8-nemo-gym-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-8-nemo-gym-vllm')" }, + { name = "transformers", version = "5.5.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-nemo-rl-trtllm' or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-trtllm' and extra != 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-vllm')" }, + { name = "transformers", version = "5.6.0", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "transformers", version = "5.8.1", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-trtllm') or (extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra != 'extra-7-nemo-rl-sglang' and extra != 'extra-7-nemo-rl-trtllm' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "triton", marker = "platform_machine == 'x86_64'" }, { name = "typing-extensions" }, ] From 1f9af79875e66492cdeab867f06c58d48c0f0ecb Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Thu, 20 Aug 2026 04:49:07 -0700 Subject: [PATCH 31/52] feat(token-capture): assemble receipts from the Gym capture ledger MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit NeMo RL companion to Gym's gate->ledger replacement (lineage capture ledger; see token-capture-lineage-ledger-approach.md and docs/design-docs/token-capture-ledger.md): - nemo_gym env: configure token_id_capture.external_staging instead of the gate block; delete register/seal/fail control calls, capability stamping, and gate metrics; fetch GET .../rollouts/{id}/manifest at rollout end and assemble the RolloutReceipt locally (terminal row by logical_request_id, fail-closed poisoning on failure rows or a missing terminal row; retry dead-branch siblings stay enumerable but never join the terminal chain). - rollout_manager: abandonment path only aborts the reservation; abandoned staged rows are swept with the staging partition at run end. - single_controller: gate metrics logging removed; replaced by ledger-derived finalize/token_in_rate + finalize/capture_poisoned_rollouts counters computed from receipt manifests in the finalizer. - config: registration_ttl_s/staging_ttl_s deleted (no TTL machinery); exemplar YAML updated. - docs: new token-capture-ledger design doc; guide updated to the manifest/receipt-assembly flow and the new health metrics. - swe_nano.env: retarget the per-user launch block at this checkout (pthombre paths, nightly-gym 08-10 container, shared uv cache — never /tmp), porting the swe-r3-capture campaign hardening. - Gym submodule bump: token-capture-lineage-ledger @ 1e1a16cb. Signed-off-by: Prashant Thombre Co-Authored-By: Claude Fable 5 Signed-off-by: Prashant Thombre --- 3rdparty/Gym-workspace/Gym | 2 +- .../rollout-verification-boundary.md | 652 ++++++++++++++++++ docs/design-docs/token-capture-ledger.md | 123 ++++ docs/guides/nano-swe-token-capture.md | 102 +-- docs/index.md | 4 +- .../configs/ultra/nano_swe_teacher_sc.yaml | 12 +- .../algorithms/async_utils/replay_buffer.py | 4 +- nemo_rl/algorithms/single_controller.py | 23 - .../single_controller_utils/config.py | 20 +- .../single_controller_utils/setup.py | 6 +- nemo_rl/data_plane/tq_token_sink.py | 2 +- nemo_rl/environments/nemo_gym.py | 218 +++--- nemo_rl/experience/blackbox_finalizer.py | 31 +- nemo_rl/experience/rollout_manager.py | 26 +- .../models/generation/vllm/vllm_generation.py | 4 +- .../generation/vllm/vllm_worker_async.py | 12 +- swe_nano.env | 53 +- .../test_nemo_gym_token_capture.py | 138 ++-- tests/unit/experience/test_rollout_manager.py | 30 +- token-capture-external-sink-recommendation.md | 183 +++++ token-capture-lineage-ledger-approach.md | 443 ++++++++++++ 21 files changed, 1759 insertions(+), 329 deletions(-) create mode 100644 docs/design-docs/rollout-verification-boundary.md create mode 100644 docs/design-docs/token-capture-ledger.md create mode 100644 token-capture-external-sink-recommendation.md create mode 100644 token-capture-lineage-ledger-approach.md diff --git a/3rdparty/Gym-workspace/Gym b/3rdparty/Gym-workspace/Gym index 10b3490861d..1e1a16cb20c 160000 --- a/3rdparty/Gym-workspace/Gym +++ b/3rdparty/Gym-workspace/Gym @@ -1 +1 @@ -Subproject commit 10b3490861d66c71bd1b549222b24fbdc58bcb9e +Subproject commit 1e1a16cb20c749985464427048185ae215a5215f diff --git a/docs/design-docs/rollout-verification-boundary.md b/docs/design-docs/rollout-verification-boundary.md new file mode 100644 index 00000000000..c733ec69400 --- /dev/null +++ b/docs/design-docs/rollout-verification-boundary.md @@ -0,0 +1,652 @@ +# Freeze the Training Trajectory Before Verification + +Status: Proposal + +## Summary + +Token-capture rollouts need an explicit boundary between model generation and +reward verification. The boundary must identify the exact committed model call +whose ancestry becomes the training trajectory, and it must be durable before +the verifier is allowed to run. + +Add a `freeze_training_trajectory` transition to the Gym capture gate. The +trusted agent harness invokes it after the main agent stops and before the +verifier can consume the generated answer or patch. The gate atomically: + +1. verifies that every admitted model call has reached a terminal capture + state; +2. resolves the harness-provided terminal logical request to a committed model + call; +3. validates the terminal call's parent chain; +4. records an immutable `GenerationBoundary`; and +5. rejects any later training-capture admission for the rollout. + +Verification runs only after the gate acknowledges that boundary. After the +verifier returns a reward, the existing seal operation produces a token-free +`RolloutReceipt` using the already-frozen terminal model call. The finalizer +continues to fetch deltas from TransferQueue (TQ), verify them, and linearize +only the terminal call's ancestry. + +This makes the training trajectory deterministic without relying on TQ row +order, wall-clock timestamps, completion-file modification times, or whichever +model call happened to finish last globally. + +## Terminology + +- **Model call**: one request admitted by the capture gate and assigned a + `model_call_id`. +- **Logical request**: the harness-visible identity for a model request. It is + mapped one-to-one to a model call within a rollout. +- **Main trajectory**: the agent session whose result is submitted to the + verifier and whose ancestry is eligible for training. +- **Sibling call**: a committed call in the same rollout that is not an + ancestor of the selected terminal call, such as a subagent or abandoned + branch. +- **Generation boundary**: the durable gate record that freezes the terminal + model call before verification. +- **Seal**: the post-verification transition that binds the reward to the + frozen trajectory and returns a `RolloutReceipt`. + +"Terminal" does not mean the last call to finish across all concurrent agent +sessions. It means the last call in the selected main trajectory. Concurrent +subagent calls may finish later, but they must not replace the main terminal or +enter its parent chain. + +## Current behavior + +The current branch has strong per-call attribution: + +- NeMo RL assigns a unique `rollout_id` to each generation. +- Gym assigns a `model_call_id` to every correlated model request. +- The worker stages each delta under the deterministic TQ key + `/`. +- Staged rows and receipt records carry `rollout_id`, `model_call_id`, + `parent_call_id`, lengths, weight version, and integrity digests. +- The finalizer starts at `RolloutReceipt.terminal_model_call_id` and follows + only its parent ancestry to build the training row. + +The SWE harness currently derives `terminal_logical_request_id` from the +`response.id` in the newest copied completion artifact for the main session. +NeMo RL passes that value to the gate when it seals the rollout after Gym has +already run the verifier. + +This selects a terminal chain deterministically in the normal SWE path, but it +does not prove a generation/verification boundary: + +- TQ rows have identity but no authoritative execution order. +- The gate does not record when generation ended or verification began. +- The terminal is selected from a persisted artifact rather than handed off + directly at the agent-to-verifier transition. +- The gate remains open to new training-capture calls until the post-verifier + seal request arrives. +- `seal_rollout` can verify that no calls are currently in flight, but it + cannot prove that the selected call was frozen before verification started. + +Manifest order must not be used as a substitute. Calls can execute +concurrently, and insertion or commit order does not identify the main +trajectory. + +## Required invariants + +The implementation must enforce the following invariants. + +### Identity + +1. A logical request maps to at most one model call within a rollout. +2. The frozen terminal logical request resolves to exactly one committed model + call. +3. Every record in the selected ancestry belongs to the same rollout. +4. Every non-root record names a committed parent whose cumulative length + equals the child's `prev_len`. + +### Boundary ordering + +1. The main agent has stopped issuing calls before the freeze request. +2. The gate acknowledges the freeze before verification begins. +3. No training-capture call is admitted after the freeze. +4. A freeze fails while any admitted call is still committing or otherwise + lacks a known terminal capture outcome. + +### Training selection + +1. The terminal model call is immutable after the freeze. +2. The reward is attached only to that frozen terminal and its ancestry. +3. Sibling calls never enter the training row. +4. Missing or ambiguous terminal attribution fails closed; it never falls back + to manifest order or a timestamp. + +### Retry safety + +1. An identical freeze retry returns the same boundary. +2. Reusing the freeze operation ID with a different terminal is rejected. +3. An identical seal retry returns the same receipt. +4. A seal request cannot change the frozen terminal. + +## Proposed protocol + +### Lifecycle + +```mermaid +stateDiagram-v2 + [*] --> Open: register rollout + Open --> Open: admit and commit calls + Open --> Frozen: freeze_training_trajectory + Open --> Failed: capture or rollout failure + Frozen --> Frozen: idempotent freeze retry + Frozen --> Sealed: seal with verifier reward + Frozen --> Failed: explicit verifier/rollout abort + Sealed --> Sealed: idempotent seal retry + Failed --> [*] + Sealed --> [*] +``` + +The verifier is outside the gate state machine, but its ordering is strict: + +```text +agent calls complete + -> freeze_training_trajectory (durable ACK) + -> verifier starts + -> seal(reward, boundary_id) + -> finalizer verifies TQ rows and publishes the training row +``` + +### Structured terminal handoff + +The trusted harness must produce a structured generation result directly from +the completed main session: + +```python +GenerationOutcome( + rollout_id=..., + trajectory_id=..., + terminal_logical_request_id=..., + terminal_turn_index=..., +) +``` + +`terminal_logical_request_id` must come from the live model response or from a +logical request ID generated before dispatch and echoed by the response. It +must not be inferred by scanning files after verification. + +`trajectory_id` identifies the main session when the harness supports +subagents. `terminal_turn_index` is a monotonically increasing main-session +turn number used for validation and diagnostics. It is not a global ordering +across sibling sessions. + +Persisted completion files can remain observability artifacts. During +migration, an adapter may recover a terminal ID from them, but strict mode must +require the direct structured handoff. + +### Call admission metadata + +For the gate to validate which terminal is eligible, each admitted call should +carry optional trajectory metadata: + +```python +CallAttribution( + trajectory_id=..., + trajectory_role="main" | "subagent" | "other", + turn_index=..., +) +``` + +The gate enforces uniqueness of `(trajectory_id, turn_index)` within a rollout. +For the main trajectory, the frozen terminal must: + +- have `trajectory_role == "main"`; +- match the `trajectory_id` in `GenerationOutcome`; and +- have the highest committed `turn_index` in that main trajectory. + +The parent graph remains the token authority. Turn indices help reject an +incorrect terminal selection but are not used to concatenate tokens. + +Harnesses that cannot yet provide trajectory metadata may use a compatibility +mode that validates only the logical request and parent chain. Such rollouts +should expose a metric showing that the stronger main-session check was not +available. + +### Freeze request + +Add an authenticated control operation: + +```http +POST /training-token-capture/control/rollouts/{rollout_id}/freeze +``` + +```json +{ + "owner_id": "...", + "operation_id": "...", + "trajectory_id": "...", + "terminal_logical_request_id": "...", + "terminal_turn_index": 17 +} +``` + +The request is valid only while the rollout is `Open`. Under the rollout's +atomic state transaction, the gate: + +1. authenticates the boundary caller; +2. rejects any calls in `admitted` or `committing` state; +3. rejects a rollout with a failed or poisoned call; +4. resolves `terminal_logical_request_id` through the rollout's logical-request + index; +5. requires the resolved call to be committed and staged; +6. validates trajectory metadata when present; +7. validates the terminal call's complete, acyclic parent chain; +8. computes a digest over the selected call IDs and their immutable manifest + records; +9. persists a `GenerationBoundary`; and +10. changes the rollout state to `Frozen` before returning success. + +The response is token-free: + +```python +GenerationBoundary( + schema_version=1, + boundary_id=..., + rollout_id=..., + trajectory_id=..., + terminal_logical_request_id=..., + terminal_model_call_id=..., + terminal_turn_index=..., + terminal_commit_sequence=..., + selected_call_ids=(...), + selected_manifest_digest=..., +) +``` + +`terminal_commit_sequence` is a gate-assigned monotonic sequence useful for +audit and metrics. It does not define the trajectory; the explicit terminal +and parent links do. + +`boundary_id` should be a deterministic digest of the immutable request and +selected manifest, so an identical retry returns the same value. + +The digest input is canonical and domain-separated: + +```text +SHA-256( + "nemo-gym-generation-boundary-v1" + || rollout_id + || trajectory_id + || terminal_logical_request_id + || terminal_model_call_id + || terminal_turn_index + || each selected CallRecord in root-to-terminal order +) +``` + +Strings and records use the same length-delimited canonical encoding on every +producer and verifier. JSON serialization, dictionary insertion order, and +timestamps are not digest inputs. + +### Admission after freeze + +Once a rollout is `Frozen`, the training-capture route rejects new model calls +before inference dispatch. This is the enforcement that the current +post-verifier seal cannot provide. + +If verification itself needs an LLM, it must use one of the following: + +- an uncorrelated evaluation endpoint; +- a separately registered verification rollout; or +- a capability explicitly scoped to non-training capture. + +Verifier calls must never share the frozen rollout's training data capability. + +### Seal request + +After verification, NeMo RL seals using the boundary identity: + +```http +POST /training-token-capture/control/rollouts/{rollout_id}/seal +``` + +```json +{ + "owner_id": "...", + "operation_id": "...", + "boundary_id": "...", + "reward": 1.0 +} +``` + +The gate verifies that: + +- the rollout is `Frozen`; +- `boundary_id` matches the stored boundary; +- the selected manifest still matches its frozen digest; and +- no capture failure was recorded. + +The receipt's `terminal_model_call_id` is copied from the boundary. The seal +request cannot supply or replace it. + +For one compatibility release, `seal` may continue accepting +`terminal_logical_request_id`, but it must exactly match the stored boundary. +Calling `seal` on an unfrozen rollout should be allowed only behind an explicit +legacy compatibility flag and should increment a metric. + +### Receipt extension + +The sealed receipt embeds the complete token-free boundary because the +finalizer does not have access to live gate state: + +```python +RolloutReceipt( + rollout_id=..., + reward=..., + terminal_model_call_id=..., + manifest=[...], + generation_boundary=GenerationBoundary(...), + capture_poisoned=False, + failure_reason=None, +) +``` + +Receipt validation requires: + +- `generation_boundary.rollout_id == receipt.rollout_id`; +- `generation_boundary.terminal_model_call_id == + receipt.terminal_model_call_id`; +- every `selected_call_id` is present exactly once in `manifest`; and +- recomputing the boundary digest from the receipt manifest yields the embedded + `boundary_id`. + +This lets any finalizer validate terminal selection without trusting the +component that transported the receipt. + +## TransferQueue and finalization + +The minimal protocol does not require ordering fields in TQ. Each staged row +already has the identity and parent link required to verify a selected chain. +The boundary belongs in gate or ledger state because it describes rollout +lifecycle, not token payload storage. + +The finalizer should additionally validate that: + +1. the receipt contains a supported generation-boundary schema version; +2. `terminal_model_call_id` matches the boundary terminal; +3. the recomputed selected call-ID sequence and manifest digest match the + boundary; and +4. the linearized row contains exactly the frozen terminal ancestry. + +The initial implementation can keep the current receipt manifest containing +all committed calls. The finalizer will verify all rows but train only the +terminal chain. A later wire revision may split this into: + +- a selected-chain manifest used for training; and +- a cleanup manifest for unselected sibling staging keys. + +That split would prevent an unselected sibling's missing row from invalidating +an otherwise sound main trajectory, but it is not required to establish the +verification boundary. + +## Authorization and secret custody + +The model-calling agent sandbox currently receives a rollout-local data +capability. It must not receive authority to freeze or seal the rollout. +Otherwise untrusted agent code could end generation early or choose its own +training terminal. + +The freeze operation therefore needs a boundary capability held only by the +trusted Gym orchestration process. Two implementation options are acceptable: + +1. issue a rollout-scoped boundary capability at registration and pass it only + to the trusted agent wrapper; or +2. give the trusted Gym server a server-to-server credential scoped to freeze + operations. + +The preferred option is a rollout-scoped boundary capability because it limits +the effect of credential disclosure. Its plaintext value must not be written +to completion files, `params.json`, logs, the agent container, receipts, or TQ +tags. Gate state stores only its digest. + +The existing NeMo RL owner remains responsible for register, fail, and seal. + +## Integration with the capture ledger proposal + +This protocol works with both the current `GateStateStore` and the proposed +per-rollout [capture ledger](token-capture-ledger.md). + +With the current store, add the boundary and lifecycle state to +`GateRolloutState` and retain it in the seal tombstone. + +With `CaptureLedger`, add one event: + +```python +GenerationFrozen( + operation_id=..., + boundary=GenerationBoundary(...), +) +``` + +The event is appended and fsynced under the rollout lock. Later `CallStarted` +events are invalid. `RolloutSealed` references the boundary ID and reward. The +ledger fold can reproduce both the boundary and receipt for idempotent retries. + +The generation-boundary contract should be implemented independently of the +gate-storage migration so the correctness guarantee does not depend on which +state backend lands first. + +## Failure handling + +| Failure | Required outcome | +| --- | --- | +| Main agent exits without a terminal logical request | Freeze fails; rollout is non-trainable and staged rows are cleaned up | +| Terminal logical request is unknown | Freeze fails closed; never select another manifest row | +| Terminal call was admitted but did not commit | Freeze fails while the call is in flight; after timeout, mark the rollout failed | +| Any call is still committing | Return a retryable conflict; do not start verification | +| Terminal trajectory metadata does not match | Freeze fails closed | +| Parent is missing, cyclic, or length-inconsistent | Freeze fails closed | +| Freeze response is lost | Retry with the same operation ID and receive the identical boundary | +| A call arrives after freeze | Reject before inference dispatch | +| Verifier crashes | The boundary remains frozen; policy decides whether to seal with a failure reward or fail the rollout | +| Seal response is lost | Retry with the same operation ID and boundary ID | +| Gate process crashes after freeze | Recover the durable boundary and continue verification/seal retry | +| Boundary capability leaks to the agent | Treat as a security incident; scoped capability limits exposure to one rollout | + +Ambiguous capture outcomes remain non-trainable. The boundary must not turn a +missing acknowledgement into evidence that a call did not execute. + +## Metrics and observability + +Add the following gate metrics: + +- `generation_freeze_succeeded` +- `generation_freeze_retries` +- `generation_freeze_conflicts` +- `generation_freeze_missing_terminal` +- `generation_freeze_invalid_terminal` +- `generation_freeze_pending_calls` +- `post_freeze_calls_rejected` +- `legacy_unfrozen_seals` +- `trajectory_metadata_compatibility_mode` + +Logs should include rollout ID, boundary ID, terminal model-call ID, selected +call count, and rejection category. They must not include capability values or +token arrays. + +For end-to-end diagnosis, expose these non-authoritative timestamps: + +- main agent stopped; +- freeze requested; +- freeze committed; +- verifier started; +- verifier completed; and +- seal committed. + +Correctness uses state transitions, not timestamp comparisons. + +## Compatibility and configuration + +Use one centralized user-facing setting: + +```yaml +token_capture: + generation_boundary_mode: compat # off | compat | required +``` + +- `off`: retain the current post-verifier terminal selection. Intended only as + an emergency rollback mode. +- `compat`: use the boundary when the harness supplies one, otherwise use the + legacy seal path and increment `legacy_unfrozen_seals`. +- `required`: do not start verification without a durable boundary, and reject + receipts that do not embed one. + +The first release can default existing configurations to `compat` while the +SWE capture recipes explicitly set `required`. After all supported capture +harnesses implement the boundary, change the centralized schema default to +`required` and remove the legacy terminal argument in a later wire revision. + +No call site may silently substitute a mode. The default belongs in the +user-facing config schema and exemplar YAML, consistent with NeMo RL config +conventions. + +## Expected code changes + +| Repository path | Change | +| --- | --- | +| Gym `nemo_gym/token_id_capture/staging/records.py` | Add versioned `CallAttribution` and `GenerationBoundary` wire models; extend `RolloutReceipt` | +| Gym `nemo_gym/token_id_capture/gate_store.py` | Add rollout lifecycle, boundary-capability digest, frozen boundary, and retry tombstone state | +| Gym `nemo_gym/token_id_capture/gate.py` | Implement atomic freeze, graph validation, boundary digest, post-freeze admission rejection, and boundary-bound seal | +| Gym `nemo_gym/token_id_capture/control_routes.py` | Add the authenticated freeze route and typed request/response models | +| Gym capture middleware and model adapter | Carry trajectory attribution and reject frozen-rollout calls before engine dispatch | +| Gym `responses_api_agents/swe_agents/app.py` | Split agent completion from verification, retain the direct terminal logical ID, freeze, then unblock evaluation | +| NeMo RL `nemo_rl/environments/nemo_gym.py` | Configure boundary mode, distribute scoped credentials, and seal with `boundary_id` | +| NeMo RL `nemo_rl/experience/blackbox_finalizer.py` | Validate the embedded boundary and exact selected ancestry before publishing | +| NeMo RL token-capture config and exemplar YAMLs | Define the centralized boundary mode and strict SWE recipe posture | + +The SWE runner currently starts the evaluation container early and lets it wait +for the model patch. That optimization can remain, but the patch or other +verifier input must stay unavailable until the freeze acknowledgement is +durable. A prestarted process does not violate the boundary as long as it +cannot begin verification work. + +## Implementation plan + +### Phase 1: Gate boundary + +1. Add `Open`, `Frozen`, and terminal rollout state to the current Gym gate. +2. Add `GenerationBoundary` and the authenticated freeze route. +3. Reject training-capture admission after freeze. +4. Change seal to consume a boundary ID and copy the frozen terminal. +5. Preserve boundary information in sealed tombstones for retry recovery. + +### Phase 2: Trusted harness handoff + +1. Have the model-client adapter retain the main session's terminal logical + request ID directly. +2. Add main-trajectory identity and turn indices where agent frameworks expose + them. +3. Freeze after the agent process exits and before making its patch or answer + available to the verifier. +4. Keep completion-file extraction only as an explicitly measured compatibility + path. +5. Ensure subagent sessions cannot overwrite the main terminal. + +### Phase 3: NeMo RL integration + +1. Register or distribute the rollout-scoped boundary capability without + exposing it to the sandbox. +2. Carry `boundary_id` in the token-free Gym result. +3. Seal with `boundary_id` and reward. +4. Reject receipt-mode results that lack a boundary when strict mode is + enabled. +5. Validate boundary identity during black-box finalization. + +### Phase 4: Ledger backend + +1. Add `GenerationFrozen` to `CaptureLedger`. +2. Reproduce boundary and seal retry behavior by folding ledger events. +3. Remove duplicate boundary state when the gate-store migration completes. + +## Required tests + +### Gate unit tests + +- Freeze resolves a logical request to the expected model call. +- Freeze stores the terminal chain and rejects a sibling as terminal when it is + not part of the main trajectory. +- Concurrent main and subagent branches freeze the main terminal regardless of + completion order. +- Freeze fails with admitted or committing calls. +- Freeze fails for failed, poisoned, unknown, or uncommitted calls. +- Freeze validates an acyclic, length-consistent parent graph. +- Identical freeze retries return an identical boundary. +- Conflicting freeze retries are rejected. +- New call admission is rejected after freeze. +- Seal cannot change the terminal and requires the correct boundary ID. +- State-store restart preserves freeze and seal retry behavior. + +### Harness tests + +- The main session's direct response ID is selected without an mtime scan. +- The freeze acknowledgement occurs before the verifier is unblocked. +- Subagent completion files and later sibling completions cannot change the + terminal. +- The boundary capability is absent from sandbox mounts, persisted parameters, + completion files, logs, and returned metadata. +- Missing direct terminal attribution fails closed in strict mode. + +### NeMo RL tests + +- Receipt-mode postprocessing requires a boundary ID in strict mode. +- Seal forwards the boundary ID and reward without choosing a terminal. +- The finalizer rejects a receipt/boundary terminal mismatch. +- The finalizer linearizes exactly the frozen ancestry. +- A rejected or missing boundary produces the existing masked placeholder and + preserves GRPO group shape. + +### End-to-end tests + +- A SWE rollout stages multiple turns, freezes, verifies, seals, and trains the + exact main terminal chain. +- A concurrent subagent finishes after the main terminal but is excluded from + training. +- A verifier that attempts to reuse the training capture route is rejected. +- Gate-worker restart between freeze and seal preserves the selected + trajectory. +- Lost freeze and seal responses are recovered through idempotent retries. + +## Alternatives considered + +### Select the final manifest row + +Rejected. Manifest order reflects admission or storage behavior, not the main +trajectory, and concurrent calls make it ambiguous. + +### Select the greatest timestamp or sequence number + +Rejected as the trajectory definition. A later subagent or verifier call could +win. Gate sequence numbers remain useful only for audit after the main +trajectory is explicitly identified. + +### Keep selecting the newest completion file + +Rejected as the strict contract. File modification time is not an atomic +generation boundary and cannot stop later capture admissions. It remains a +temporary compatibility adapter. + +### Seal before verification + +Rejected because the current seal includes the reward and terminates rollout +state. Splitting terminal freeze from reward seal preserves the useful +post-verification receipt contract while establishing the boundary earlier. + +### Let the agent sandbox freeze its own trajectory + +Rejected. The sandbox is part of the evaluated workload and must not control +which of its calls becomes training data. + +## Recommendation + +Implement `freeze_training_trajectory` as a distinct, durable transition and +make verifier startup depend on its acknowledgement. Use an explicit main +trajectory terminal from the live model-call path, reject later +training-capture admissions, and allow the post-verification seal to attach +only the reward. + +This is the smallest protocol change that turns the current trusted terminal +hint into an enforceable guarantee that the exact pre-verification trajectory +is the one used for training. diff --git a/docs/design-docs/token-capture-ledger.md b/docs/design-docs/token-capture-ledger.md new file mode 100644 index 00000000000..0898ff14552 --- /dev/null +++ b/docs/design-docs/token-capture-ledger.md @@ -0,0 +1,123 @@ +# Token Capture Lineage Ledger + +Exact-token capture for blackbox agentic rollouts is coordinated by a single +per-rollout **capture ledger**: NeMo Gym's `LineageStore`, extended so that its +append-only JSONL rows are simultaneously the request-time lineage index and +the token-free record of rollout capture state. There is no separate gate +state machine; serving workers coordinate only through the ledger, and NeMo RL +(the rollout owner) assembles the `RolloutReceipt` itself at rollout end. + +The external staging contract (`StagingSink` / `StagingSource`), the vLLM +worker capture path, and the `verify_and_linearize()` trust boundary are +unchanged from the worker-custody design. The recipe exercising this path end +to end is [Nano SWE with Token Capture](../guides/nano-swe-token-capture.md); +the verification trust boundary is described in +[Rollout Verification Boundary](rollout-verification-boundary.md). + +## Why a ledger and not a gate + +An earlier iteration paired the lineage store with a `RolloutCaptureGate` and +a cross-process `GateStateStore`. The gate did not provide a second lineage +algorithm — parent resolution ran upstream through `LineageStore.resolve()`, +and the gate cross-checked that result against its own copy of the call state, +storing each call's cumulative token IDs **twice** (gate state + lineage +JSONL). Its file-backed state store also serialized the entire global gate +state — every live rollout's cumulative token arrays — under one exclusive +lock, three transactions per model call. + +Everything the gate legitimately provided — admission, rollout completeness, +terminal selection, cleanup — is either a pure function of the lineage result +or belongs to the framework that already owns the rollout. So each +responsibility moved to its natural owner and the redundant state machine was +deleted. + +## The ledger + +`FileLineageStore` writes one locked, fsynced JSONL row per committed call. +In external-staging mode (`token_id_capture.external_staging: true`) each row +additionally carries the token-free `CallRecord` custody columns — +`parent_call_id`, `staging_key`, `weight_version`, `prev_len` / `delta_len` / +`cum_len`, the staged record's `digest` and `extras_digest`, `mode`, and +`logical_request_id` (the client header when present, else the vLLM response +id). Three surfaces make it the single record of capture state (the +`CaptureLedger` protocol): + +- `record(...)` — the extended commit row, written by the model server's + commit hook after the worker's `CommitCoords` arrive. +- `record_failure(rollout_id, model_call_id, reason)` — a poison row for a + call whose capture did not commit. Failure rows carry no fingerprint, so + `resolve()` can never return them as parents. +- `manifest(rollout_id)` — the token-free read-back (committed rows + + failures), exposed over one bearer-protected control route: + `GET /training-token-capture/rollouts/{rollout_id}/manifest`. + +`InMemoryLineageStore` cannot serve the ledger role: its resolution index +evicts rollouts under memory bounds, which is fine for a cache but not for a +completeness record. External staging requires a non-evicting store and +rejects the in-memory store at startup. + +## Admission is a pure function + +When external staging is enabled, `resolve_parent()` builds the +`CaptureAdmission` directly from the lineage result — a strict tri-state: + +| Lineage outcome | Admission | +| --- | --- | +| `ROOT` — empty assistant fingerprint, or unmatched fingerprint on a rollout with no ledger rows (seeded assistant history) | `text` mode, no parent | +| `MATCH` — unique fingerprint match with verified context digest | `token_in` mode, `required_prefix_token_ids` = parent's cumulative tokens | +| `UNRESOLVED` — non-empty fingerprint with no match, ambiguity, or digest mismatch | no admission; `record_failure()` poisons the call | + +`UNRESOLVED` is never silently converted into a new root: doing so would turn +earlier policy-generated tokens into mask-zero prompt tokens and corrupt the +training row. The completion still serves the agent; only training capture is +poisoned. + +## Commit ordering + +The invariant the external sink requires — *a call must not become a lineage +parent until its staged record is durable* — holds structurally: the worker +stages through `StagingSink.stage()` before acknowledging, coordinates exist +only after the bytes are durable, and the ledger row (which is what makes a +call resolvable as a parent) is written only after the coordinates arrive. +On `disposition == "staged"` the commit hook reconstructs +`cumulative = parent_tokens + token_ids_delta` and appends the extended row; +on `capture_failed`, missing coordinates, or any acknowledgement error it +appends a failure row instead. A request that dies after admission is poisoned +from the capture middleware's `finally` hook. + +## Framework-owned receipt and cleanup + +NeMo RL fetches the manifest at rollout end and assembles the receipt locally: + +- `manifest` = the fetched `CallRecord` list, deduped by `model_call_id`; +- `terminal_model_call_id` = the row whose `logical_request_id` matches the + rollout's reported terminal logical request (a response id); +- `capture_poisoned` = any failure row present, or no row for the terminal + request. + +`verify_and_linearize(receipt, snapshots)` runs unchanged. Retry duplicates +appear as dead-branch sibling rows in the manifest: their staged rows are +fetched, verified, and cleaned like any other, but they never join the +terminal chain (`_validate_manifest_graph` tolerates rows unreferenced by the +terminal chain). Cleanup is manifest-enumerated in the finalizer; an abandoned +dispatch's staged rows are swept with the staging partition at run end (there +is no prefix-clear primitive in the data plane yet). + +## Failure semantics (all fail-closed) + +- **Capture fails mid-rollout:** the model call still succeeds for the agent; + a failure row is written. Later calls miss resolution → `UNRESOLVED` → + more failure rows. Finalization sees failure rows → poisoned → masked + placeholder row (the group still publishes exactly N rows). +- **Terminal response lost, harness retries:** the retry is a sibling row + (per-request `uuid4` identity). The harness reports the retry's response id, + so receipt assembly selects the retry's row; the lost attempt is a dead + branch. An ambiguous mid-rollout sibling (identical regenerated text) + poisons via `UNRESOLVED` instead of silently becoming a root. +- **Crash after staging, before the ledger append:** descendants resolve + `UNRESOLVED` and poison; a terminal orphan poisons via the missing terminal + row. + +Retry *idempotency* (harness-minted logical request ids + deterministic +`model_call_id`, collapsing identical retries into the same row instead of +poisoning) is an explicit follow-up; no retry outcome is silently wrong today. diff --git a/docs/guides/nano-swe-token-capture.md b/docs/guides/nano-swe-token-capture.md index 3822311ad4c..0612060abfe 100644 --- a/docs/guides/nano-swe-token-capture.md +++ b/docs/guides/nano-swe-token-capture.md @@ -1,9 +1,9 @@ -# Nano SWE RL with Gate-Authoritative Token Capture +# Nano SWE RL with Ledger-Authoritative Token Capture A reproducible 6-node recipe that runs agentic SWE RL on Nemotron-3-Nano-30B-A3B with **exact-token capture** enabled: the vLLM worker stages each model call's token delta durably into the TransferQueue data -plane, the Gym gate serves verified prefix token ids back on every follow-up +plane, the Gym capture ledger serves verified prefix token ids back on every follow-up call (token-in), and the trainer consumes rows rebuilt from the staged deltas. No token echo over HTTP, no re-tokenization of agent history — the tokens the engine sampled are byte-for-byte the tokens the trainer sees. @@ -45,29 +45,29 @@ The call flow for a single SWE rollout, end to end: 1. **Dispatch.** The `SingleControllerActor`'s rollout pump pulls a prompt group from the dataset and hands it to the NemoGym environment actor, which POSTs `/run` to the `swe_agents_train` Gym server. The run body - carries a fresh `ng_rollout_id`; RL **registers** the rollout with the gate - (control plane) before dispatch and mounts the returned rollout-local data - capability into the agent sandbox without serializing the secret. + carries a fresh `ng_rollout_id`. There is no registration step: the + rollout's ledger file is created lazily by its first committed call. 2. **Sandbox + agent.** The SWE harness materializes the SWE-bench instance in a sandbox and starts the OpenHands agent. The rollout id is snapshotted into the agent's config so every LLM request uses the rollout-prefixed - route and its data-capability header. The capability file is mode `0600` - and is removed when the request finishes. + token-capture route. 3. **Agent turn loop.** Each turn, OpenHands POSTs `/v1/chat/completions` with the *full rendered history* to the policy model server, which runs - in **gate mode**: - - the gate admits the call (rejects unregistered rollouts), fingerprints - the incoming history, and resolves which previously served call this - one continues (its *parent*) in the lineage index; + in **ledger mode**: + - the server fingerprints the incoming history and resolves which + committed ledger row this call continues (its *parent*); admission is a + pure function of that lineage result; - on a match it attaches the parent's **exact cumulative token ids** (`required_prefix_token_ids`) plus a capture context (`rollout_id`, `model_call_id`, `parent_call_id`, `prev_len`), and forwards the request to a vLLM engine; - - no match (a true root, or a rewritten/condensed history) starts a new - chain — a fallback, never an error. - Gate state and lineage are file-backed under the shared capture directory, - so registration, consecutive calls, and sealing may land on different - serving workers without losing ancestry. + - no assistant-authored history is a true text root; assistant history + without one verified committed parent is `UNRESOLVED` and fails closed + instead of silently starting a new chain. + Lineage and capture custody are stored together in a per-rollout + append-only JSONL file under the shared capture directory, so consecutive + calls may land on different serving workers without losing ancestry or + serializing unrelated rollouts. 4. **Generate + stage.** The vLLM worker splices the supplied prefix verbatim, renders only the new tail through the chat template, and generates. Before acknowledging the call it **stages the call's token @@ -75,18 +75,21 @@ The call flow for a single SWE rollout, end to end: (0.0 on carried prompt, 1.0 on generated), and per-token logprobs — to the TransferQueue staging partition (synchronous `tq_put`: bytes are durable before the response leaves the worker). The response back to the - gate carries only text plus token-light `CommitCoords` (~4 B/token). -5. **Commit.** The gate ingests the coords into its lineage index (this - *is* the authoritative commit — the next turn's parent resolution - depends on it), strips them, and returns a plain OpenAI-shaped - completion to the agent. Steps 3-5 repeat for every tool call the agent + ledger carries only text plus token-light `CommitCoords` (~4 B/token). +5. **Commit.** Gym reconstructs cumulative tokens from the coordinates and + appends one ledger row carrying both lineage and staging custody + (including the call's `logical_request_id`). This is the authoritative + commit the next turn resolves against; Gym strips the coordinates and + returns a plain OpenAI-shaped completion to the agent. Steps 3-5 repeat for every tool call the agent makes (tool execution happens agent-side between turns). -6. **Verify + seal.** When the agent finishes, the harness runs the - SWE-bench verifier to score the patch (reward 0/1). The `/run` response - returns the reward plus a **token-free `RolloutReceipt`**: the ordered - manifest of committed calls (model-call ids, staging keys, digests, weight - versions) and a `terminal_model_call_id` naming the chain that is the - training row. +6. **Verify + assemble.** When the agent finishes, the harness runs the + SWE-bench verifier to score the patch (reward 0/1) and reports the + terminal response id. RL fetches the rollout's **token-free manifest** + (`GET /training-token-capture/rollouts/{id}/manifest`) and assembles the + `RolloutReceipt` locally: the manifest of committed calls (model-call ids, + staging keys, digests, weight versions), a `terminal_model_call_id` + selected by the terminal logical request id, and fail-closed poisoning + (any failure row, or a missing terminal row, masks the rollout). 7. **Finalize.** The controller constructs a metadata-only `FinalizationRequest`, releases the rollout concurrency permit, and submits it to a fixed pool of CPU Ray actors. Each actor owns a connect-only TQ @@ -107,9 +110,8 @@ The call flow for a single SWE rollout, end to end: The token path in that flow, compressed: ``` -agent (nv-OpenHands) gate (vllm_model, gate mode) - prefixed path + data capability ───► admits registered calls, - fingerprints incoming history → +agent (nv-OpenHands) ledger (vllm_model, external staging) + rollout-prefixed capture path ───► fingerprints incoming history → resolves the parent call → sends the parent's exact prefix token ids │ required_prefix_token_ids @@ -120,23 +122,23 @@ vLLM worker: splices the prefix verbatim, renders only the new tail, acked — and returns token-light CommitCoords on the response │ coords (≈4 B/token) ▼ -gate ingests coords into its lineage index; when the rollout ends it -returns a token-free RolloutReceipt (a manifest of model_call_ids and staging -keys) on the /run response +ledger atomically publishes coords + lineage; when the rollout ends RL +fetches the token-free manifest (model_call_ids and staging keys) from the +control route and assembles the RolloutReceipt itself ▼ fixed CPU Ray finalizer pool: accepts metadata only, fetches staged deltas by key, verifies digests, rebuilds the exact row, and publishes it to TQ ``` The heavy bytes (token arrays, logprobs) move exactly once, worker→TQ, -node-locally. The gate hop and the `/run` response stay token-light. +node-locally. The ledger hop and the `/run` response stay token-light. The pieces, by repo: | Component | Where | |---|---| -| Gate mode, prefix serving, lineage | Gym `responses_api_models/vllm_model/app.py` + `nemo_gym/token_id_capture/gate.py` | -| Rollout attribution + capability custody | Gym capture middleware + `swe_agents` sandbox mount/header forwarding | +| Ledger mode, prefix serving, lineage | Gym `responses_api_models/vllm_model/app.py` + `nemo_gym/token_id_capture/lineage.py` | +| Rollout attribution | Gym capture middleware + `swe_agents` rollout-prefixed routing | | Wire schema (deltas, coords, receipts) | Gym `nemo_gym/token_id_capture/staging/records.py` | | Worker-side capture + prefix splice | `nemo_rl/models/generation/vllm/vllm_worker_async.py` | | Staging sink/source over TransferQueue | `nemo_rl/data_plane/tq_token_sink.py` | @@ -200,8 +202,8 @@ DRY_RUN=0 SC_EXP_NAME=my-capture-smoke NG_TIC_FP_CANONICAL=1 \ `2 × 4 = 8`. The finalizer pool is mandatory whenever capture is enabled, and `num_finalizer_workers` must be positive. There is no pool enable/disable or legacy-inline-finalizer flag. The smoke deliberately runs two Gym policy -model workers; their gate state and lineage are shared, so it does not rely on -single-worker request affinity. +model workers; their per-rollout ledger files are shared, so it does not rely +on single-worker request affinity. Different rollouts use different locks. The 1h49 allocation is suitable when the model and compilation caches are warm and automatically selects the `short` QOS in `ultra_launch.sh`. For a @@ -245,7 +247,7 @@ its absence broke a run: | `token_capture.enabled=true` | Capture never engages. The launcher flips the config's default. Enabling it always constructs the fixed CPU finalizer actor pool sized by `token_capture.num_finalizer_workers`; there is no inline fallback. | | `token_capture.defer_routed_experts_to_policy=true` + `+policy.router_replay.enabled=true` | The validated R3 posture keeps routed-expert tensors out of controller RPCs and canonical rows, then reconstructs them on policy workers from staged-fragment plans. Set both together. | | `async_rl.sampler.name=windowed` + `max_staleness_versions=1` | The smoke does not exercise the intended bounded-staleness sampling policy. | -| `NG_TIC_FP_CANONICAL=1` | Token-in silently degrades to ~0: reasoning models echo history with `` blocks stripped, the gate's fingerprint of the served turn never matches, and every call falls back to text mode. With it, `token_in_rate ≈ 0.9999`. | +| `NG_TIC_FP_CANONICAL=1` | Reasoning models otherwise echo history with `` blocks stripped, so the ledger cannot verify a unique parent and fails the call as `UNRESOLVED`. With canonical fingerprints, `token_in_rate ≈ 0.9999`. | | `NRL_DRIVER_PYTHONPATH=/opt/nemo-rl/3rdparty/Gym-workspace/Gym` | Driver `ModuleNotFoundError: nemo_gym` — the driver imports the staging record schema, and the baked driver venv has no nemo_gym. | | `NRL_DRIVER_PIP_INSTALL=orjson` | Driver `ModuleNotFoundError: orjson` — Gym's `token_id_capture/__init__` eagerly imports the store. | | `NRL_DRIVER_UV_RUN_FLAGS="--locked --no-sync"` | `uv run` otherwise replaces the prefetched driver environment and can give the driver a different Python/Ray version from the already-running Ray cluster. Lock mutation is forbidden; worker-specific environments are still rebuilt. | @@ -257,21 +259,21 @@ its absence broke a run: Config echo is not evidence. Check, in order: -1. **Gate metrics in the SC step output.** Grade from the SC worker `.out` - under `-logs/ray/session_*/logs/worker-*-.out` — the driver - log's actor-stdout forwarding is not reliable and can make a healthy run - look stalled: +1. **Ledger-derived admission counters in the finalize metrics.** Each + manifest row records its admission mode, and the finalizer aggregates them + per group into `step_metrics` (W&B prefix `train`): ``` - gate_metrics={'registered': 128.0, 'admitted': 7048.0, - 'token_in': 7042.0, 'text': 6.0, - 'capture_failed': 0.0, ...} + finalize/token_in_calls, finalize/text_root_calls, + finalize/token_in_rate, finalize/capture_poisoned_rollouts ``` - `token_in / admitted` should be ≥ 0.99 after the root calls. A rate near 0 - with a large `text` count means canonical fingerprints are off (see above). - `capability_rejected` or `unattributed_calls` above zero means the harness - is not forwarding the rollout capability/path correctly. + `finalize/token_in_rate` should be ≥ 0.99 after the root calls (each chain + opens with exactly one `text` root). A rate near 0 with a large + `text_root_calls` count means canonical fingerprints are off (see above). + Nonzero `capture_poisoned_rollouts` means calls are failing admission or + commit — check the model-server logs for `unresolved_parent` / + `worker_capture_failed` poison reasons. 2. **Finalizer-pool health.** In `step_metrics`, require `finalize/invalid_row_rate=0` and, for the R3 command above, diff --git a/docs/index.md b/docs/index.md index 257949f12f8..9fe88ddc5f7 100644 --- a/docs/index.md +++ b/docs/index.md @@ -153,7 +153,7 @@ Six-node SWE RL smoke on Nemotron-3-Nano-30B-A3B with rollouts flowing through t :link: guides/nano-swe-token-capture :link-type: doc -Exact-token SWE RL: the worker stages each call's token delta into TransferQueue and the gate serves verified prefixes back — no token echo, no re-tokenization. +Exact-token SWE RL: the worker stages each call's token delta into TransferQueue and the capture ledger serves verified prefixes back — no token echo, no re-tokenization. ::: :::{grid-item-card} {octicon}`plus-circle` Adding New Models @@ -384,6 +384,8 @@ design-docs/training-backends.md design-docs/sequence-packing-and-dynamic-batching.md design-docs/env-vars.md design-docs/nemo-gym-integration.md +design-docs/token-capture-ledger.md +design-docs/rollout-verification-boundary.md design-docs/modelopt-real-quant-architecture.md design-docs/nccl-reshard-refit.md ``` diff --git a/examples/configs/ultra/nano_swe_teacher_sc.yaml b/examples/configs/ultra/nano_swe_teacher_sc.yaml index ab9c5c22ac8..706242e5d21 100644 --- a/examples/configs/ultra/nano_swe_teacher_sc.yaml +++ b/examples/configs/ultra/nano_swe_teacher_sc.yaml @@ -75,7 +75,7 @@ loss_fn: # Enable a small KL (matches grpo_math_1B_megatron_single_controller.yaml). reference_policy_kl_penalty: 0.01 -# Gate-authoritative token capture (token-in/token-out via NeMo-Gym; see +# Ledger-authoritative token capture (token-in/token-out via NeMo-Gym; see # docs/guides/nano-swe-token-capture.md). Dormant by default: with # enabled=false every legacy codepath behaves exactly as before — flip to # true (or override token_capture.enabled=true) for the capture arm. @@ -88,7 +88,7 @@ token_capture: # retains these through policy consumption; direct mode clears in finalizer. staging_partition: "rollout_staging" # continue: a failed worker-side stage poisons the rollout (placeholder row); - # abort: fails the whole rollout at the gate. + # abort: fails the whole rollout in the ledger. on_capture_failure: "continue" # allow: train groups whose calls span a refit (staleness = group's oldest # call version); reject: placeholder such rollouts. @@ -96,16 +96,12 @@ token_capture: # Drop the whole group when fewer than this fraction of its rollouts # produced valid rows (null keeps every group). min_valid_fraction_per_group: null - # Gate registration expiry. staging_ttl_s is reserved configuration only; - # TQ does not currently provide an operational staging TTL/reaper. - registration_ttl_s: 3600.0 - staging_ttl_s: 3600.0 # Bearer token for Gym's token-capture control routes. null = minted per run. control_auth_token: null - # Hard deadline per control-plane call (gate death must surface as failed + # Hard deadline per control-plane call (control-plane death must surface as failed # dispatches + placeholders, not a silent retry stall). control_timeout_s: 60.0 - # Process-shared gate state, lineage, and capture root. null resolves to + # Process-shared per-rollout ledger and capture root. null resolves to # /gym_token_capture. capture_dir: null # Keep route bytes out of canonical rows and assemble them on policy workers. diff --git a/nemo_rl/algorithms/async_utils/replay_buffer.py b/nemo_rl/algorithms/async_utils/replay_buffer.py index 524873ad21b..a37cf5a61c6 100644 --- a/nemo_rl/algorithms/async_utils/replay_buffer.py +++ b/nemo_rl/algorithms/async_utils/replay_buffer.py @@ -657,7 +657,7 @@ def __init__( self._dp_client = dp_client self._partition_id = partition_id self._pad_value_dict = dict(pad_value_dict) - # Token-capture mode only (docs/design-docs/tq-gym-gate-authoritative.md): + # Token-capture mode only (docs/design-docs/token-capture-ledger.md): # the staging partition whose per-call delta rows `remove` must clear # alongside the canonical rows. None on the legacy path. self._staging_partition_id = staging_partition_id @@ -687,7 +687,7 @@ def reserve( weight_version: Weight version stamped on the slot. target_step: Training step this slot targets; only consulted by StalenessSampler.force_in_order. group_id: Per-group sample_id prefix; defaults to a fresh uuid4. - rollout_ids: Token-capture mode: the gate-registered rollout ids + rollout_ids: Token-capture mode: the ledger-registered rollout ids this slot dispatched, recorded so cleanup can name what it owns even before a receipt exists. diff --git a/nemo_rl/algorithms/single_controller.py b/nemo_rl/algorithms/single_controller.py index 2e3db2681e2..e604d8c5168 100644 --- a/nemo_rl/algorithms/single_controller.py +++ b/nemo_rl/algorithms/single_controller.py @@ -837,8 +837,6 @@ async def _train_pump(self) -> None: self._logger.log_metrics( timing_metrics, step=self._train_steps, prefix="timing/train" ) - if self._master_config.token_capture.enabled: - await self._log_gate_metrics() self._timer.reset() # min sample version refers to the version each consumed sample was @@ -960,27 +958,6 @@ async def _sync_weights( self._rollout_permitted.set() return aborted_stale_inflight_groups - async def _log_gate_metrics(self) -> None: - """Log the capture gate's cumulative § 8 counters + token_in_rate. - - Counters are cumulative over the run; ``gate/token_in_rate`` is the - cumulative marker-hit rate over all admitted model calls. Fetch - failures are logged and swallowed — metrics must never kill a step. - """ - try: - counters = await self._rollout_manager.gate_metrics() - except (RuntimeError, OSError) as error: - print(f"gate metrics fetch failed: {error}", flush=True) - return - if not counters: - return - calls = counters.get("admitted", 0) - gate_metrics: dict[str, float] = {k: float(v) for k, v in counters.items()} - if calls: - gate_metrics["token_in_rate"] = counters.get("token_in", 0) / calls - self._logger.log_metrics(gate_metrics, step=self._train_steps, prefix="gate") - print(f"gate_metrics={gate_metrics}", flush=True) - async def _advantage_stage(self, meta: KVBatchMeta) -> KVBatchMeta: """Fetch advantage inputs, compute advantages, and write them back. diff --git a/nemo_rl/algorithms/single_controller_utils/config.py b/nemo_rl/algorithms/single_controller_utils/config.py index aeda29e84c8..ca9a6973bde 100644 --- a/nemo_rl/algorithms/single_controller_utils/config.py +++ b/nemo_rl/algorithms/single_controller_utils/config.py @@ -54,12 +54,12 @@ class AsyncRLConfig(BaseModel, extra="allow"): class TokenCaptureConfig(BaseModel, extra="allow"): - """Gate-authoritative token capture (token-in/token-out via NeMo-Gym). + """Ledger-authoritative token capture (token-in/token-out via NeMo-Gym). Dormant by default: with ``enabled=False`` every legacy codepath behaves - exactly as before — no staging partition is registered, no gate is + exactly as before — no staging partition is registered, no ledger is installed, and rollouts ride the token-echo path. See - docs/design-docs/tq-gym-gate-authoritative.md. + docs/design-docs/token-capture-ledger.md. """ enabled: bool = False @@ -68,7 +68,7 @@ class TokenCaptureConfig(BaseModel, extra="allow"): staging_partition: str = "rollout_staging" # A failed worker-side stage poisons the rollout; "continue" serves the # completion and lets the finalizer emit a placeholder row, "abort" fails - # the whole rollout at the gate. + # the whole rollout in the ledger. on_capture_failure: Literal["continue", "abort"] = "continue" # "allow" trains groups whose calls span a refit (staleness accounted via # group_min_wv); "reject" placeholders them. Strict modes beyond the MVP @@ -77,19 +77,15 @@ class TokenCaptureConfig(BaseModel, extra="allow"): # Drop the whole group when fewer than this fraction of its rollouts # produced valid rows (None keeps every group). min_valid_fraction_per_group: Optional[float] = None - # Gate-side cleanup backstops. - registration_ttl_s: float = 3600.0 - staging_ttl_s: float = 3600.0 # Bearer token for Gym's token-capture control routes. None = # minted per run at setup; set explicitly only for multi-controller - # setups that must share one gate. + # setups that must share one ledger. control_auth_token: Optional[str] = None - # Hard deadline per control-plane call (S5 finding: gate death must + # Hard deadline per control-plane call (S5 finding: control-plane death must # surface as a failed dispatch, not a silent retry stall). control_timeout_s: float = 60.0 - # Directory for the Gym base capture layer the gate rides on (#2124-c1: - # the capture middleware only engages with a capture dir configured; the - # dir stays essentially empty on the gate path). None = derived at setup + # Root for Gym's per-rollout capture ledgers and base capture layer. None = + # derived at setup # under the run's log dir. capture_dir: Optional[str] = None # Keep routed_experts out of canonical rows and assemble them on policy diff --git a/nemo_rl/algorithms/single_controller_utils/setup.py b/nemo_rl/algorithms/single_controller_utils/setup.py index 0917f428f9b..27096f1e550 100644 --- a/nemo_rl/algorithms/single_controller_utils/setup.py +++ b/nemo_rl/algorithms/single_controller_utils/setup.py @@ -302,7 +302,7 @@ def _spinup_gym(master_config: MasterConfig, base_urls: list[str]) -> tuple[Any, enable_router_replay=enable_router_replay, routed_experts_dtype=routed_experts_dtype, use_fastokens=bool(policy_config["tokenizer"].get("use_fastokens")), - # Gate config rides into Gym's policy model server (§ 9.1). + # Ledger config rides into Gym's policy model server (§ 9.1). token_capture=( master_config.token_capture.model_dump() if master_config.token_capture.enabled @@ -416,7 +416,7 @@ def setup_single_controller( if not _should_use_nemo_gym(master_config): raise ValueError( "token_capture.enabled requires the NeMo-Gym rollout path " - "(env.should_use_nemo_gym=true) — the gate lives in Gym's " + "(env.should_use_nemo_gym=true) — the ledger lives in Gym's " "policy model server" ) if generation_config["backend"] != "vllm": @@ -439,7 +439,7 @@ def setup_single_controller( "nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker" ] = PY_EXECUTABLES.VLLM_GYM - # Fill the derived gate-hosting fields (see TokenCaptureConfig): a + # Fill the derived ledger-hosting fields (see TokenCaptureConfig): a # per-run control-plane bearer token and the process-shared capture # directory used by every Gym worker. if token_capture_cfg.control_auth_token is None: diff --git a/nemo_rl/data_plane/tq_token_sink.py b/nemo_rl/data_plane/tq_token_sink.py index 10c0ece0631..39822c2fbc2 100644 --- a/nemo_rl/data_plane/tq_token_sink.py +++ b/nemo_rl/data_plane/tq_token_sink.py @@ -14,7 +14,7 @@ """TransferQueue implementations of NeMo-Gym's token staging protocols. ``TQTokenSink``/``TQTokenSource`` are NeMo-RL's providers for the -gate-authoritative capture design (docs/design-docs/tq-gym-gate-authoritative.md): +ledger-authoritative capture design (docs/design-docs/token-capture-ledger.md): the sink is the worker-side write of one model call's token delta to the ``rollout_staging`` partition — the design's only heavy token hop — and the source is the finalizer's read-back of those rows by staging key. This module diff --git a/nemo_rl/environments/nemo_gym.py b/nemo_rl/environments/nemo_gym.py index dcfe8fd10ea..43f87ac3891 100644 --- a/nemo_rl/environments/nemo_gym.py +++ b/nemo_rl/environments/nemo_gym.py @@ -12,7 +12,6 @@ # See the License for the specific language governing permissions and # limitations under the License. import asyncio -import hashlib import math import os import subprocess @@ -129,20 +128,19 @@ class NemoGymConfig(TypedDict): tokenizer_config: NotRequired[ Optional[TokenizerConfig] ] # For processor reconstruction inside the actor - # Gate-authoritative token capture (token_capture.enabled): the dumped - # TokenCaptureConfig. Turns on the gate in Gym's policy model server, - # switches run_rollouts to receipt mode, and adds the register/seal/fail - # control-plane helpers. None/absent = legacy token-echo path. + # Ledger-authoritative token capture (token_capture.enabled): the dumped + # TokenCaptureConfig. Turns on external staging in Gym's policy model + # server, switches run_rollouts to receipt mode, and assembles receipts + # from the manifest control route. None/absent = legacy token-echo path. token_capture: NotRequired[Dict[str, Any] | None] -# Gym control-plane server name (the model server hosting the gate) and the +# Gym control-plane server name (the model server hosting the ledger) and the # opaque run-body key rollout ids ride on (Gym's ROLLOUT_ID_KEY_NAME): the # agent derives the id from the run body and stamps /ng-rollout/ on every # model call, so the TQ sample id IS the capture key end to end. _POLICY_SERVER_NAME = "policy_model" _NG_ROLLOUT_ID_BODY_KEY = "_ng_rollout_id" -_NG_CAPTURE_CAPABILITY_BODY_KEY = "_ng_capture_capability" _TOKEN_CAPTURE_CONTROL_PREFIX = "/training-token-capture/control" _TOKEN_CAPTURE_CONTROL_ENV = "NEMO_GYM_TOKEN_CAPTURE_CONTROL_TOKEN" @@ -479,12 +477,12 @@ def _spinup(self) -> None: "`rollout_max_attempts_to_avoid_lp_nan` must be at least 1" ) - # Gate-authoritative token capture: turn on the gate in the policy - # model server (via the policy_model global-config override block the - # env yamls already use) and disable the legacy token echo. Receipt - # mode is incompatible with re-dispatching a batch under the same - # rollout ids, so the NaN retry must be exactly 1 (create-only - # registration would fail the retry loudly anyway; fail at setup). + # Ledger-authoritative token capture: enable external staging in the + # policy model server (via the policy_model global-config override + # block the env yamls already use) and disable the legacy token echo. + # Receipt mode is incompatible with re-dispatching a batch under the + # same rollout ids — the retry's calls would resolve against the + # first attempt's ledger rows — so the NaN retry must be exactly 1. token_capture = self.cfg.get("token_capture") or None self._token_capture_enabled = bool( token_capture and token_capture.get("enabled") @@ -492,13 +490,12 @@ def _spinup(self) -> None: self._server_client = None self._control_headers: Dict[str, str] = {} self._control_timeout_s = 60.0 - self._control_owner_id = f"nemo-rl-{os.getpid()}-{id(self):x}" if self._token_capture_enabled: if self.rollout_max_attempts_to_avoid_lp_nan != 1: raise ValueError( "token_capture.enabled requires " "rollout_max_attempts_to_avoid_lp_nan == 1: a NaN retry " - "would re-register create-only rollout ids at the gate" + "would resolve against the first attempt's ledger rows" ) policy_overrides = ( initial_global_config_dict.setdefault("policy_model", {}) @@ -512,18 +509,14 @@ def _spinup(self) -> None: "all_agents": True, "rebuild_response": False, "dir": capture_dir, - # Both stores are process-shared. Every uvicorn worker builds - # its own handle over these same paths, so token-in ancestry - # remains valid when consecutive calls land on different - # workers. + # The lineage store is process-shared and doubles as the + # per-rollout capture ledger. Every uvicorn worker builds its + # own handle over the same root, so token-in ancestry remains + # valid when consecutive calls land on different workers. "lineage_store": ("nemo_gym.token_id_capture.lineage:FileLineageStore"), "lineage_store_kwargs": {"root": os.path.join(capture_dir, "lineage")}, - "gate": { - "enabled": True, - "state_store_path": os.path.join(capture_dir, "gate-state.json"), - "registration_ttl_s": token_capture["registration_ttl_s"], - "control_auth_token_env": _TOKEN_CAPTURE_CONTROL_ENV, - }, + "external_staging": True, + "control_auth_token_env": _TOKEN_CAPTURE_CONTROL_ENV, } # Gym resolves the credential inside each serving process. Keep # only the variable name in serialized config and inherit the @@ -533,8 +526,8 @@ def _spinup(self) -> None: "Authorization": f"Bearer {token_capture['control_auth_token']}" } # S5 chaos finding: Gym's shared request() retries connection - # errors indefinitely; a dead gate must surface as a failed - # dispatch (placeholders + TTL), not a silent stall. + # errors indefinitely; a dead control plane must surface as a + # failed manifest fetch (placeholder row), not a silent stall. self._control_timeout_s = float( token_capture.get("control_timeout_s") or 60.0 ) @@ -556,7 +549,7 @@ def _spinup(self) -> None: ) self.rch = RolloutCollectionHelper() - # ── gate control plane (token-capture mode) ───────────────────────────── + # ── ledger control plane (token-capture mode) ─────────────────────────── def _control_client(self): """Gym ServerClient resolving servers by name from the head server.""" @@ -583,61 +576,16 @@ async def _control(self, method: str, path: str, **kwargs: Any) -> dict: ) except asyncio.TimeoutError: raise RuntimeError( - f"gate control call {method} {path} exceeded " - f"{self._control_timeout_s}s (gate unreachable or stalled)" + f"ledger control call {method} {path} exceeded " + f"{self._control_timeout_s}s (control plane unreachable or stalled)" ) from None if response.status != 200: raise RuntimeError( - f"gate control call {method} {path} failed: " + f"ledger control call {method} {path} failed: " f"HTTP {response.status} {await response.text()}" ) return await response.json() - def _control_operation_id(self, operation: str, rollout_id: str) -> str: - payload = f"{self._control_owner_id}\0{operation}\0{rollout_id}".encode() - return f"{operation}-{hashlib.sha256(payload).hexdigest()}" - - async def register_rollouts(self, rollout_ids: list[str]) -> dict[str, str]: - """Register each rollout and return its worker-facing data capability.""" - capabilities: dict[str, str] = {} - for rollout_id in rollout_ids: - registration = await self._control( - "PUT", - f"{_TOKEN_CAPTURE_CONTROL_PREFIX}/rollouts/{rollout_id}", - json={ - "owner_id": self._control_owner_id, - "operation_id": self._control_operation_id("register", rollout_id), - }, - ) - capability = registration.get("data_capability") - if not isinstance(capability, str) or not capability: - raise RuntimeError( - f"gate registration for {rollout_id} returned no data capability" - ) - capabilities[rollout_id] = capability - return capabilities - - async def fail_rollouts(self, rollout_ids: list[str], *, reason: str) -> None: - """Best-effort gate cleanup for a cancelled/failed dispatch (§ 7).""" - for rollout_id in rollout_ids: - try: - await self._control( - "POST", - f"{_TOKEN_CAPTURE_CONTROL_PREFIX}/rollouts/{rollout_id}/fail", - json={ - "owner_id": self._control_owner_id, - "operation_id": self._control_operation_id("fail", rollout_id), - "reason": reason, - }, - ) - except (RuntimeError, OSError) as error: - # The registration TTL is the backstop when the gate is - # unreachable during teardown. - print(f"fail_rollout({rollout_id}) failed: {error}", flush=True) - - async def gate_metrics(self) -> dict: - return await self._control("GET", f"{_TOKEN_CAPTURE_CONTROL_PREFIX}/metrics") - async def run_rollouts( self, nemo_gym_examples: list[dict], @@ -661,16 +609,6 @@ async def run_rollouts( # examples carry no `input_image` items (text-only case). encode_images_in_examples(nemo_gym_examples) - if self._token_capture_enabled: - # Receipt mode: register the ids riding each row's run body - # before dispatch; seal at completion. - rollout_ids = [ - example[_NG_ROLLOUT_ID_BODY_KEY] for example in nemo_gym_examples - ] - capabilities = await self.register_rollouts(rollout_ids) - for example, rollout_id in zip(nemo_gym_examples, rollout_ids): - example[_NG_CAPTURE_CAPABILITY_BODY_KEY] = capabilities[rollout_id] - timer.start("_run_rollouts_total") nemo_gym_result_iterator = self.rch.run_examples( examples=nemo_gym_examples, head_server_config=self.head_server_config @@ -701,9 +639,10 @@ async def run_rollouts( with timer.time(label=f"{timer_prefix}/postprocess_results"): if self._token_capture_enabled: - # Receipt mode: seal at the gate; token-free result. The - # canonical row is rebuilt by the finalizer, so no - # message_log walk (and no NaN check) applies here. + # Receipt mode: fetch the ledger manifest and assemble the + # receipt locally; token-free result. The canonical row is + # rebuilt by the finalizer, so no message_log walk (and no + # NaN check) applies here. nemo_rl_result = await self._postprocess_receipt_mode( nemo_gym_row, nemo_gym_result ) @@ -749,11 +688,11 @@ async def run_rollouts( async def _postprocess_receipt_mode( self, nemo_gym_row: dict, nemo_gym_result: dict ) -> dict: - """Seal the rollout and return a token-free result (§ 9.1). + """Fetch the ledger manifest and assemble the receipt locally. The legacy token walk (and its contiguity assert) does not run: the - gate owns lineage now, output items carry no token arrays, and the - canonical row is rebuilt by the finalizer from staged deltas. The + capture ledger owns lineage, output items carry no token arrays, and + the canonical row is rebuilt by the finalizer from staged deltas. The Ray return carries only the receipt (~100 B/call) beside the agent-level result. """ @@ -762,35 +701,23 @@ async def _postprocess_receipt_mode( ) rollout_id = nemo_gym_row[_NG_ROLLOUT_ID_BODY_KEY] terminal_logical_request_id = nemo_gym_result.get("terminal_logical_request_id") - if not isinstance(terminal_logical_request_id, str) or not ( - terminal_logical_request_id - ): - await self.fail_rollouts( - [rollout_id], reason="missing_terminal_logical_request_id" - ) - return { - "message_log": [], - "input_message_log": [], - "full_result": nemo_gym_result, - "rollout_id": rollout_id, - "receipt": None, - } - try: - receipt = await self._control( - "POST", - f"{_TOKEN_CAPTURE_CONTROL_PREFIX}/rollouts/{rollout_id}/seal", - json={ - "owner_id": self._control_owner_id, - "operation_id": self._control_operation_id("seal", rollout_id), - "terminal_logical_request_id": terminal_logical_request_id, - "reward": float(nemo_gym_result.get("reward") or 0.0), - }, - ) - except (RuntimeError, OSError) as error: - # An unsealable rollout finalizes as a placeholder; the gate's - # registration TTL sweeps its state. - print(f"seal({rollout_id}) failed: {error}", flush=True) - receipt = None + receipt = None + if isinstance(terminal_logical_request_id, str) and terminal_logical_request_id: + try: + manifest = await self._control( + "GET", + f"{_TOKEN_CAPTURE_CONTROL_PREFIX}" + f"/rollouts/{rollout_id}/manifest", + ) + receipt = self._assemble_receipt( + rollout_id, + manifest, + terminal_logical_request_id=terminal_logical_request_id, + reward=float(nemo_gym_result.get("reward") or 0.0), + ) + except (RuntimeError, OSError) as error: + # An unfetchable manifest finalizes as a placeholder row. + print(f"manifest({rollout_id}) fetch failed: {error}", flush=True) return { "message_log": [], "input_message_log": [], @@ -799,6 +726,55 @@ async def _postprocess_receipt_mode( "receipt": receipt, } + @staticmethod + def _assemble_receipt( + rollout_id: str, + manifest: dict, + *, + terminal_logical_request_id: str, + reward: float, + ) -> dict: + """Build the token-free RolloutReceipt payload from a ledger manifest. + + The terminal row is the manifest row whose ``logical_request_id`` + equals the response id the agent actually kept. Retry duplicates are + dead-branch rows: they stay in the manifest (their staged rows are + fetched, verified, and cleaned) but never join the terminal chain — + ``verify_and_linearize`` tolerates rows unreferenced by the terminal + chain. Poisoning is fail-closed: any failure row, or no row for the + terminal request, masks the rollout. + """ + records = [dict(record) for record in manifest.get("records") or []] + failures = list(manifest.get("failures") or []) + deduped: dict[str, dict] = {} + for record in records: + deduped.setdefault(str(record.get("model_call_id")), record) + terminal_record = next( + ( + record + for record in deduped.values() + if record.get("logical_request_id") == terminal_logical_request_id + ), + None, + ) + failure_reason = None + if failures: + failure_reason = str(failures[0].get("reason") or "capture_failed") + elif terminal_record is None: + failure_reason = "missing_terminal_row" + return { + "rollout_id": rollout_id, + "reward": reward, + "terminal_model_call_id": ( + terminal_record.get("model_call_id") + if terminal_record is not None + else None + ), + "manifest": list(deduped.values()), + "capture_poisoned": failure_reason is not None, + "failure_reason": failure_reason, + } + def _postprocess_nemo_gym_to_nemo_rl_result( self, nemo_gym_row: dict, diff --git a/nemo_rl/experience/blackbox_finalizer.py b/nemo_rl/experience/blackbox_finalizer.py index bd195ada519..60fd5ff987b 100644 --- a/nemo_rl/experience/blackbox_finalizer.py +++ b/nemo_rl/experience/blackbox_finalizer.py @@ -13,7 +13,7 @@ # limitations under the License. """Blackbox finalization: token-free receipts + staged deltas -> canonical rows. -Orchestration only (docs/design-docs/tq-gym-gate-authoritative.md § 5, § 9.1): +Orchestration only (docs/design-docs/token-capture-ledger.md): per rollout, fetch the staged rows the receipt manifest names through the ``TokenSource``, re-verify them (digest recomputation over fetched values, shape/mask/finite-logprob checks, length chaining, weight-version tag @@ -530,6 +530,33 @@ def finalize_group( sum(len(row.staging_keys) for row in rows) / len(rows) ), } + # Ledger-derived admission counters (per group): each manifest row + # carries its admission mode. token_in_rate near 1.0 is the capture + # health signal (a text root only opens each chain); this replaces the + # deleted gate metrics route. + manifest_rows = [ + record + for receipt in receipts + if isinstance(receipt, dict) + for record in (receipt.get("manifest") or []) + if isinstance(record, dict) + ] + if manifest_rows: + token_in_calls = sum( + 1 for record in manifest_rows if record.get("mode") == "token_in" + ) + metrics["finalize/token_in_calls"] = float(token_in_calls) + metrics["finalize/text_root_calls"] = float( + len(manifest_rows) - token_in_calls + ) + metrics["finalize/token_in_rate"] = token_in_calls / len(manifest_rows) + metrics["finalize/capture_poisoned_rollouts"] = float( + sum( + 1 + for receipt in receipts + if isinstance(receipt, dict) and receipt.get("capture_poisoned") + ) + ) for row in rows: if not row.valid: print( @@ -660,7 +687,7 @@ def finalize_group( weight_version=group_min_wv, ) assert sample_ids == rollout_ids, ( - "canonical sample ids must equal the gate-registered rollout ids: " + "canonical sample ids must equal the ledger-registered rollout ids: " f"{sample_ids} != {rollout_ids}" ) _tensorize_ms = (time.perf_counter() - _tensorize_t0) * 1000.0 diff --git a/nemo_rl/experience/rollout_manager.py b/nemo_rl/experience/rollout_manager.py index 4d1faae6903..f8f3bbaa0c4 100644 --- a/nemo_rl/experience/rollout_manager.py +++ b/nemo_rl/experience/rollout_manager.py @@ -783,8 +783,6 @@ def __init__( self._tokenizer = tokenizer self._num_generations_per_prompt = num_generations_per_prompt self._tq_buffer = tq_buffer - # The NeMo-Gym env handle doubles as the gate control-plane proxy - # (gate_metrics / fail_rollouts) on the capture path. self._env_handles = task_to_env self._weight_version: int = 0 @@ -796,20 +794,6 @@ def set_weight_version(self, version: int) -> None: """ self._weight_version = int(version) - async def gate_metrics(self) -> Optional[dict[str, int]]: - """Fetch the capture gate's § 8 counters, or None off the capture path. - - Returns: - Cumulative gate counters (token_in, fallback_*, capture_failed, - registered/sealed/failed/expired) from Gym's token-capture control - metrics route, - or None when no NemoGym env handle is wired. - """ - env = self._env_handles.get("nemo_gym") if self._env_handles else None - if env is None: - return None - return await env.gate_metrics.remote() - async def run_rollout( self, input_sample: DatumSpec, *, rollout_ids: Optional[list[str]] = None ) -> PromptGroupRecord: @@ -926,10 +910,10 @@ async def generate_for_finalization( assert_metadata_only(request) return request except BaseException: + # Abandoned dispatch: no receipt will name these rollouts' staged + # rows, so they leak until the staging partition is torn down at + # run end (there is no prefix-clear primitive in the data plane + # yet). Their ledger files are inert — failure rows or missing + # terminal rows keep any later read fail-closed. self._tq_buffer.abort(group_id) - nemo_gym_env = self._env_handles.get("nemo_gym") - if nemo_gym_env is not None: - await nemo_gym_env.fail_rollouts.remote( - list(rollout_ids), reason="dispatch_failed" - ) raise diff --git a/nemo_rl/models/generation/vllm/vllm_generation.py b/nemo_rl/models/generation/vllm/vllm_generation.py index 2740aa5ea60..f71b329475e 100644 --- a/nemo_rl/models/generation/vllm/vllm_generation.py +++ b/nemo_rl/models/generation/vllm/vllm_generation.py @@ -536,12 +536,12 @@ def _post_init(self): def setup_token_capture( self, dp_cfg: dict[str, Any], staging_partition: str ) -> None: - """Install gate-authoritative token capture in every DP-leader worker. + """Install ledger-authoritative token capture in every DP-leader worker. Called once at setup when ``token_capture.enabled``; each async worker builds its in-worker data-plane client + TQTokenSink and makes the single Gym ``install_capture`` call (see - docs/design-docs/tq-gym-gate-authoritative.md § 9.1). + docs/design-docs/token-capture-ledger.md). """ assert self.cfg["vllm_cfg"]["async_engine"], ( "token capture requires the async vLLM engine (the capture host " diff --git a/nemo_rl/models/generation/vllm/vllm_worker_async.py b/nemo_rl/models/generation/vllm/vllm_worker_async.py index 61275293a64..2a1c3545fad 100644 --- a/nemo_rl/models/generation/vllm/vllm_worker_async.py +++ b/nemo_rl/models/generation/vllm/vllm_worker_async.py @@ -101,9 +101,9 @@ def __init__( self.base_url = None self.http_server = None - # Gate-authoritative token capture (dormant until the + # Ledger-authoritative token capture (dormant until the # setup_token_capture fan-out runs; see - # docs/design-docs/tq-gym-gate-authoritative.md § 9.1). The weight + # docs/design-docs/token-capture-ledger.md). The weight # version is stamped per model call at begin_call time and rotated by # the set_rollout_weight_version fan-out from the SC's _sync_weights. self.token_capture = None @@ -364,7 +364,7 @@ def install_token_capture(self, capture: Any) -> None: async def setup_token_capture( self, dp_cfg: dict[str, Any], staging_partition: str ) -> bool: - """Host gate-authoritative token capture in this worker. + """Host ledger-authoritative token capture in this worker. Fan-out target (token_capture.enabled only): builds the in-worker data-plane client and TQTokenSink, then makes the single @@ -396,11 +396,11 @@ async def set_rollout_weight_version(self, version: int) -> None: self._rollout_weight_version = int(version) def _begin_request_capture(self, request: Any, prompt_token_ids: list[int]) -> None: - """Admit one gate-forwarded call into the capture layer. + """Admit one ledger-forwarded call into the capture layer. Called from preprocess_chat once the exact engine prompt is known (post-splice in token-in mode, full render in text mode). No-op - unless capture is installed and the request carries the gate's + unless capture is installed and the request carries the ledger's ``ng_capture`` context. """ capture = self.token_capture @@ -750,7 +750,7 @@ class NeMoRLChatCompletionRequest( NeMoRLOpenAIChatRequestMixin, ChatCompletionRequest ): required_prefix_token_ids: Optional[List[int]] = None - # Gate-authoritative token capture: the call identity the gate + # Ledger-authoritative token capture: the call identity the ledger # attaches (rollout_id, call_id, parent_call_id, prev_len, mode). ng_capture: Optional[dict[str, Any]] = None diff --git a/swe_nano.env b/swe_nano.env index 274fc578f97..b4ada725428 100644 --- a/swe_nano.env +++ b/swe_nano.env @@ -22,6 +22,17 @@ # Point every variable in that block at a directory you own before launching. # ============================================================================= export NRL_FORCE_REBUILD_VENVS=false + +# The 2026-07-26 ultra-recipes container's baked NemoGym venv has no usable +# nemo_gym install; rebuild only that one by default (identical on both A/B +# arms). The capture launcher extends this list. +export NRL_FORCE_REBUILD_VENVS_LIST="${NRL_FORCE_REBUILD_VENVS_LIST:-nemo_rl.environments.nemo_gym.NemoGym}" + +# Secrets (WANDB_API_KEY, HF_TOKEN) live in an untracked mode-600 file so they +# never enter git history. Create swe_nano.secrets.env next to this file. +_SECRETS="$(dirname "${BASH_SOURCE[0]}")/swe_nano.secrets.env" +# shellcheck disable=SC1090 +[ -f "${_SECRETS}" ] && source "${_SECRETS}" export NCCL_MAX_NCHANNELS=1 export NCCL_NVLS_ENABLE=0 @@ -30,11 +41,13 @@ export NCCL_NVLS_ENABLE=0 # ----------------------------------------------------------------------------- # Training container (GB200 aarch64, prefetched venvs) and the nemo-skills # sandbox image. Both are plain squashfs files read by pyxis at job start. -export CONTAINER=/lustre/fsw/portfolios/llmservice/users/zhiyul/enroot-images/gitlab-master.nvidia.com+yifuw+images+rl+ultra_gb200_forcenccl_20260724_prefetch_venvs.2026-07-25.squashfs +# nightly-gym 08-10: vLLM 0.25.1 (the 07-26 bake's vLLM 0.20 routed-experts +# capturer overflows its slot buffer on hybrid Nemotron at scale). +export CONTAINER=/lustre/fsw/portfolios/llmservice/users/zhiyul/enroot-images/nvcr.io+nvidian+nemo-rl+nightly-gym.2026-08-10.squashfs export SANDBOX_CONTAINER=/lustre/fsw/portfolios/coreai/users/cye/enroot/nemo-rl:skills-sandbox-latest.squashfs # SWE prompt set (7816 instances). Read-only; no need to duplicate. -export DATA_DIR=/lustre/fsw/portfolios/coreai/users/cye/code/RL-debug-ultra/workspace/data +export DATA_DIR=/lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/data # SWE rewards come from executing each instance in its apptainer .sif image # (no GPU judges). Expects swerebench/ and swegym/ subdirectories. @@ -55,12 +68,12 @@ export MODEL_PATH=nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 # HF_HOME HuggingFace cache — ~60 GB for the Nano BF16 checkpoint # PERSISTENT_CACHE vLLM/Triton/Inductor compile caches reused across jobs # ----------------------------------------------------------------------------- -# Code from this checkout (the fsw path is a symlink to the fs1 RL-copy tree). -export CODE_DIR=/lustre/fsw/portfolios/llmservice/users/zhiyul/RL-copy -export WORKSPACE_DIR=${CODE_DIR}/workspace +# Code from this checkout (fsw path of the fs1 delta-staging tree). +export CODE_DIR=/lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-staging +export WORKSPACE_DIR=/lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace -export HF_HOME=/lustre/fsw/portfolios/llmservice/users/zhiyul/hf_cache -export PERSISTENT_CACHE=/lustre/fsw/portfolios/llmservice/users/zhiyul/persistent_cache +export HF_HOME=/lustre/fsw/portfolios/llmservice/users/pthombre/hf_cache +export PERSISTENT_CACHE=/lustre/fsw/portfolios/llmservice/users/pthombre/persistent_cache export NRL_MEGATRON_CHECKPOINT_DIR=${PERSISTENT_CACHE}/megatron_ckpt_cache # Slurm identity — change SLURM_ACCOUNT to an account you can charge. @@ -88,7 +101,7 @@ export WALLTIME=3:59:00 # empty submits with no --qos. Set SLURM_QOS=short (and WALLTIME under 2h) if you # want the priority boost and know your node budget is free. export SLURM_QOS= -export NRL_MAX_STEPS=5 +export NRL_MAX_STEPS="${NRL_MAX_STEPS:-5}" # Nano base checkpoint: MTP off, container's stock vLLM (no Ultra fork needed). export ENABLE_MTP_INFERENCE=0 @@ -96,7 +109,7 @@ export USE_CUSTOM_VLLM=0 # Mount RL-copy's own Gym (writable; SWE/OpenHands harness writes here at runtime). export EXTRA_MOUNTS=/lustre:/lustre,${CODE_DIR}/3rdparty/Gym-workspace/Gym:/opt/nemo-rl/3rdparty/Gym-workspace/Gym -export WANDB_PROJ=nano-swe-smoke +export WANDB_PROJ=${WANDB_PROJ:-nano-swe-smoke} # Start safe: inspect the resolved launch command before submission. export DRY_RUN=1 @@ -104,7 +117,7 @@ export USE_SNAPSHOT=0 # Default variant = async GRPO baseline. The tq/sc launchers override both. # EXP_NAME namespaces results/, ray_logs/ and the W&B run — rename it per user. -export EXP_NAME=nano-swe-teacher-qwen3mesh-zhiyul +export EXP_NAME=nano-swe-ledger-smoke-pthombre export CONFIG_PATH=${CODE_DIR}/examples/configs/ultra/nano_swe_teacher_qwen3mesh.yaml export TRAIN_PATH=${DATA_DIR}/swe.jsonl export VAL_PATH=${DATA_DIR}/swe.jsonl @@ -113,3 +126,23 @@ export RESULTS_DIR=${WORKSPACE_DIR}/results/${EXP_NAME} export BASE_LOG_DIR=${WORKSPACE_DIR}/ray_logs/${EXP_NAME} export NRL_WG_USE_RAY_REF=1 + +# nightly-gym 08-10 bakes TransferQueue 0.1.9 into the base env on every +# node, so the per-actor runtime_env pip pin (a compute-node GitHub fetch) +# is skipped. +export NRL_TQ_SKIP_RUNTIME_ENV_PIN=1 + +# Shared prewarmed uv cache (survives across jobs; removes compute-node +# GitHub dependence for venv rebuilds). Never /tmp: UV_CACHE_DIR rides into +# the driver's TRAIN_CMD (ultra_launch.sh) and UV_CACHE_DIR_OVERRIDE mounts +# the same cache at /root/.cache/uv on every node (ray.sub). +export UV_CACHE_DIR=/lustre/fsw/portfolios/llmservice/users/pthombre/uv +export UV_CACHE_DIR_OVERRIDE=/lustre/fsw/portfolios/llmservice/users/pthombre/uv + +# Forced venv rebuilds must install from the container's uv.lock, not +# re-resolve. +export NRL_VENV_SYNC_FROZEN=1 + +# A failed engine weight-sync must crash fast, not wedge the SC for the +# rest of the walltime. +export NRL_REFIT_ERRORS_FATAL=1 diff --git a/tests/unit/environments/test_nemo_gym_token_capture.py b/tests/unit/environments/test_nemo_gym_token_capture.py index a87b040f622..3c540577d74 100644 --- a/tests/unit/environments/test_nemo_gym_token_capture.py +++ b/tests/unit/environments/test_nemo_gym_token_capture.py @@ -22,35 +22,29 @@ def _capture_env() -> NemoGym: env_cls = NemoGym.__ray_metadata__.modified_class - env = object.__new__(env_cls) - env._control_owner_id = "owner-1" - return env - - -def test_register_rollouts_returns_capabilities_and_uses_idempotent_ownership() -> None: - env = _capture_env() - env._control = AsyncMock( - side_effect=[ - {"data_capability": "cap-r0"}, - {"data_capability": "cap-r1"}, - ] - ) - - capabilities = asyncio.run(env.register_rollouts(["r0", "r1"])) - - assert capabilities == {"r0": "cap-r0", "r1": "cap-r1"} - for rollout_id, call in zip(("r0", "r1"), env._control.await_args_list): - assert call.args == ( - "PUT", - f"/training-token-capture/control/rollouts/{rollout_id}", - ) - assert call.kwargs["json"]["owner_id"] == "owner-1" - assert call.kwargs["json"]["operation_id"].startswith("register-") + return object.__new__(env_cls) + + +def _manifest_record(call_id: str, *, logical_request_id: str, parent: str | None = None) -> dict: + prev_len = 0 if parent is None else 900 + return { + "model_call_id": call_id, + "parent_call_id": parent, + "prev_len": prev_len, + "delta_len": 100, + "cum_len": prev_len + 100, + "weight_version": 3, + "digest": "a" * 64, + "extras_digest": "b" * 64, + "staging_key": f"r0/{call_id}", + "mode": "text" if parent is None else "token_in", + "logical_request_id": logical_request_id, + } def test_receipt_postprocess_fails_closed_without_a_terminal_logical_id() -> None: env = _capture_env() - env.fail_rollouts = AsyncMock() + env._control = AsyncMock() result = asyncio.run( env._postprocess_receipt_mode( @@ -59,32 +53,96 @@ def test_receipt_postprocess_fails_closed_without_a_terminal_logical_id() -> Non ) ) - env.fail_rollouts.assert_awaited_once_with( - ["r0"], reason="missing_terminal_logical_request_id" - ) + env._control.assert_not_awaited() assert result["receipt"] is None -def test_receipt_postprocess_seals_the_trusted_logical_terminal() -> None: +def test_receipt_postprocess_fetches_manifest_and_selects_terminal_row() -> None: env = _capture_env() - env._control = AsyncMock(return_value={"rollout_id": "r0", "manifest": []}) + records = [ + _manifest_record("c1", logical_request_id="lr-1"), + _manifest_record("c2", logical_request_id="lr-2", parent="c1"), + ] + env._control = AsyncMock( + return_value={"rollout_id": "r0", "records": records, "failures": []} + ) result = asyncio.run( env._postprocess_receipt_mode( {"_ng_rollout_id": "r0"}, - { - "reward": 1.0, - "terminal_logical_request_id": "logical-terminal", - }, + {"reward": 1.0, "terminal_logical_request_id": "lr-2"}, ) ) call = env._control.await_args assert call.args == ( - "POST", - "/training-token-capture/control/rollouts/r0/seal", + "GET", + "/training-token-capture/control/rollouts/r0/manifest", + ) + receipt = result["receipt"] + assert receipt["rollout_id"] == "r0" + assert receipt["terminal_model_call_id"] == "c2" + assert receipt["capture_poisoned"] is False + assert receipt["failure_reason"] is None + assert receipt["reward"] == 1.0 + assert [r["model_call_id"] for r in receipt["manifest"]] == ["c1", "c2"] + + +def test_receipt_assembly_poisons_on_failure_rows() -> None: + env = _capture_env() + manifest = { + "rollout_id": "r0", + "records": [_manifest_record("c1", logical_request_id="lr-1")], + "failures": [{"model_call_id": "c2", "reason": "worker_capture_failed"}], + } + receipt = env._assemble_receipt( + "r0", manifest, terminal_logical_request_id="lr-1", reward=0.0 ) - assert call.kwargs["json"]["owner_id"] == "owner-1" - assert call.kwargs["json"]["operation_id"].startswith("seal-") - assert call.kwargs["json"]["terminal_logical_request_id"] == ("logical-terminal") - assert result["receipt"] == {"rollout_id": "r0", "manifest": []} + assert receipt["capture_poisoned"] is True + assert receipt["failure_reason"] == "worker_capture_failed" + + +def test_receipt_assembly_poisons_when_the_terminal_row_is_missing() -> None: + env = _capture_env() + manifest = { + "rollout_id": "r0", + "records": [_manifest_record("c1", logical_request_id="lr-1")], + "failures": [], + } + receipt = env._assemble_receipt( + "r0", manifest, terminal_logical_request_id="lr-lost", reward=0.0 + ) + assert receipt["capture_poisoned"] is True + assert receipt["failure_reason"] == "missing_terminal_row" + assert receipt["terminal_model_call_id"] is None + + +def test_receipt_assembly_keeps_dead_branch_siblings_in_the_manifest() -> None: + """A retry sibling stays enumerable (its staged row must be cleaned) but + never becomes the terminal call.""" + env = _capture_env() + records = [ + _manifest_record("c1", logical_request_id="lr-1"), + _manifest_record("c2", logical_request_id="lr-2", parent="c1"), + _manifest_record("c2r", logical_request_id="lr-2r", parent="c1"), + ] + manifest = {"rollout_id": "r0", "records": records, "failures": []} + receipt = env._assemble_receipt( + "r0", manifest, terminal_logical_request_id="lr-2r", reward=1.0 + ) + assert receipt["terminal_model_call_id"] == "c2r" + assert receipt["capture_poisoned"] is False + assert {r["model_call_id"] for r in receipt["manifest"]} == {"c1", "c2", "c2r"} + + +def test_receipt_postprocess_returns_placeholder_on_fetch_failure() -> None: + env = _capture_env() + env._control = AsyncMock(side_effect=RuntimeError("control plane down")) + + result = asyncio.run( + env._postprocess_receipt_mode( + {"_ng_rollout_id": "r0"}, + {"reward": 1.0, "terminal_logical_request_id": "lr-1"}, + ) + ) + assert result["receipt"] is None diff --git a/tests/unit/experience/test_rollout_manager.py b/tests/unit/experience/test_rollout_manager.py index b65a68ef48a..3bd614edbf4 100644 --- a/tests/unit/experience/test_rollout_manager.py +++ b/tests/unit/experience/test_rollout_manager.py @@ -1006,25 +1006,6 @@ def reserve( ) -class _FakeGymEnvHandle: - """NemoGym actor stand-in exposing fail_rollouts.remote.""" - - def __init__(self): - self.failed: list[tuple[list[str], str]] = [] - outer = self - - class _FailRollouts: - def remote(self, rollout_ids, reason): - outer.failed.append((list(rollout_ids), reason)) - - async def _done(): - return None - - return _done() - - self.fail_rollouts = _FailRollouts() - - def _receipt_record(rollout_ids, receipts): completions = [ Completion( @@ -1050,7 +1031,7 @@ def _make_capture_manager(buf, *, on_run=None, num_generations=2): mgr._tokenizer = None mgr._num_generations_per_prompt = num_generations mgr._tq_buffer = buf - mgr._env_handles = {"nemo_gym": _FakeGymEnvHandle()} + mgr._env_handles = {} mgr._weight_version = 7 class _CaptureImpl: @@ -1090,9 +1071,8 @@ def test_mints_ids_and_returns_metadata_request(self): # Finalization and commit are exclusively owned by the controller's # actor-pool path; the manager leaves the reservation unready. assert buf.commit_calls == [] - assert mgr._env_handles["nemo_gym"].failed == [] - def test_failed_dispatch_aborts_and_fails_gate_rollouts(self): + def test_failed_dispatch_aborts_the_reservation(self): buf = _FakeCaptureBuffer() async def _boom(_sample): @@ -1101,8 +1081,6 @@ async def _boom(_sample): mgr = _make_capture_manager(buf, on_run=_boom) with pytest.raises(RuntimeError, match="rollout exploded"): _run(mgr.generate_for_finalization({"prompt": "p"})) + # The slot is released; abandoned staged rows are swept with the + # staging partition at run end (no per-rollout control-plane call). assert len(buf.abort_calls) == 1 - (failed_ids, reason) = mgr._env_handles["nemo_gym"].failed[0] - (group_id,) = [buf.abort_calls[0]] - assert failed_ids == [f"{group_id}_g0", f"{group_id}_g1"] - assert reason == "dispatch_failed" diff --git a/token-capture-external-sink-recommendation.md b/token-capture-external-sink-recommendation.md new file mode 100644 index 00000000000..bb4ea7baefd --- /dev/null +++ b/token-capture-external-sink-recommendation.md @@ -0,0 +1,183 @@ +# PR 2278 Review: External Token Staging with `LineageStore` + +## Recommendation + +Use the existing `LineageStore` directly for parent resolution and cumulative +token tracking. Supporting an external staging sink does not require a separate +`RolloutCaptureGate` or `GateStateStore`. + +The current gate does not provide a second lineage algorithm. It resolves the +parent through `LineageStore`, duplicates the resulting call state in the gate, +and publishes successful commits back to `LineageStore`. This creates two +sources of truth for the same rollout ancestry. + +The recommended initial implementation is framework-authoritative: + +- Gym owns request correlation and lineage resolution. +- The inference worker owns durable staging through `StagingSink`. +- The external framework owns rollout completeness, terminal selection, + finalization, and cleanup. +- The finalizer verifies the staged records with `verify_and_linearize()` before + publishing a training row. + +## Why `LineageStore` Is Sufficient for Lineage + +`LineageStore` already provides the two operations needed by the serving path: + +1. `resolve(rollout_id, request_items)` identifies the verified parent and + returns its cumulative token IDs. +2. `record(...)` publishes a completed call so a later request, including one + handled by another model-server worker, can continue it. + +`FileLineageStore` supplies cross-process read-after-write visibility for +multi-worker model servers. The gate currently delegates lineage publication +back to this interface after it receives worker commit coordinates. + +Removing the gate does not change the lineage algorithm. It removes the +duplicated lifecycle database around it. + +## Simplified External-Sink Flow + +1. Gym assigns `rollout_id` and `model_call_id` in the request-scoped capture + context. +2. Gym calls `LineageStore.resolve()` before dispatch. +3. Gym sends the call identity, parent identity, and required prefix to the + inference worker. +4. The worker creates a `StagedCallRecord` and makes it durable through the + configured `StagingSink` before acknowledging success. +5. After successful staging, Gym computes the cumulative token sequence and + calls `LineageStore.record()`. +6. Gym removes token IDs, log probabilities, routed-expert values, and internal + coordinates before returning the model response to the agent. +7. The external framework reads staged records through `StagingSource`, selects + the terminal ancestry, and calls `verify_and_linearize()`. + +The ordering requirement is important: a call must not become a lineage parent +until its external staged record is durable. + +## What the Gate Adds + +The gate adds lifecycle and authorization behavior, not lineage behavior: + +- rollout registration and per-rollout data capabilities; +- admitted-but-not-committed call tracking; +- owner-bound and operation-bound retries; +- logical-request-to-model-call indexing; +- seal and failure transitions; +- receipt construction; +- TTL expiry, tombstones, cleanup queues, and metrics. + +These behaviors may be useful, but they should not be treated as prerequisites +for an external sink. If NeMo RL already owns rollout lifecycle and finalization, +duplicating those responsibilities inside Gym increases coupling without +improving lineage resolution. + +## Important Caveat: Root Versus Unresolved Lineage + +The existing lineage result uses `None` for several different cases: + +- the request is a genuine first call with no assistant-authored history; +- the assistant fingerprint is ambiguous; +- the request context digest no longer matches; +- a prior call cannot be found. + +The local capture builder can sometimes recover a missing parent through strict +token-prefix matching. The external finalizer follows explicit parent links and +cannot rely on that recovery. + +The external path should therefore distinguish: + +```text +ROOT -> admit as a text-mode root +MATCH -> admit as a token-in child +UNRESOLVED -> reject or poison token capture +``` + +Silently converting `UNRESOLVED` into a new root can turn earlier +policy-generated tokens into prompt tokens with mask zero and produce an +incorrect training trajectory. + +This is a lineage API correction and remains necessary whether or not a gate is +present. + +## Responsibilities Outside `LineageStore` + +`LineageStore` as currently defined does not store: + +- external staging keys; +- weight versions and complete commit manifests; +- admitted calls that never committed; +- terminal model-call identity; +- failed or sealed rollout state. + +For the framework-authoritative design, those responsibilities remain with the +external sink, source, controller, and finalizer. The staged record already +contains the parent link, lengths, weight version, and integrity digests needed +for final verification. + +If Gym must eventually own fail-closed receipt construction, extend the lineage +store into a per-rollout append-only capture ledger. Do not introduce a separate +global gate state file. A ledger could add `CallStarted`, `CallCommitted`, +`CallFailed`, and `RolloutSealed` events alongside the existing lineage data. + +## External Sink Contract Recommendations + +Keep the public integration surface small: + +- `StagingSink.stage(record) -> StageResult` +- `StagingSource.fetch(staging_keys) -> snapshots` +- versioned `StagedCallRecord`, `CallRecord`, and receipt types where needed; +- digest and conformance helpers; +- `verify_and_linearize()` as the final trust boundary; +- engine-specific capture adapters behind the generic capture protocol. + +For cleanup after a lost worker acknowledgment, prefer a deterministic staging +key derived from `rollout_id` and `model_call_id`, or allocate the key before +dispatch. An opaque key returned only after staging cannot be recovered when the +acknowledgment is lost. + +Authorization, if required for untrusted agent traffic, can be implemented as a +small stateless signed capability check. It does not require a complete rollout +state machine. + +## Suggested PR Scope + +The core Gym PR should contain: + +- staging protocols and versioned records; +- integrity and conformance helpers; +- the worker capture hook; +- the generic lineage integration; +- the external finalization verifier; +- a vLLM adapter behind the engine-neutral interface. + +Move framework- or harness-specific behavior into companion changes: + +- NeMo RL TransferQueue sink/source and finalizer wiring; +- SWE-specific credential-file routing and terminal discovery; +- rollout-controller policy, cleanup, and metrics; +- unrelated packaging exclusions or runtime-directory cleanup. + +## Decision Summary + +Both behaviors are required for a fully fail-closed system, but both current +components are not: + +| Concern | Recommended owner | +| --- | --- | +| Parent resolution and exact prefix | Existing `LineageStore` | +| Durable token/logprob/route staging | External `StagingSink` | +| Snapshot retrieval | External `StagingSource` | +| Terminal selection and rollout completeness | External framework | +| Final integrity verification | `verify_and_linearize()` | +| Separate Gym gate database | Remove from the initial external-sink path | + +The smallest useful change is therefore to keep `LineageStore` and the staging +contract, remove the independent gate state system, and leave lifecycle policy +with the framework that already owns the rollout. + +## Reviewed Change + +This recommendation is based on the five-commit delta in +[NVIDIA-NeMo/Gym PR 2278](https://github.com/NVIDIA-NeMo/Gym/pull/2278), from +base `d2123272` through head `1b342084`. diff --git a/token-capture-lineage-ledger-approach.md b/token-capture-lineage-ledger-approach.md new file mode 100644 index 00000000000..451b13898c1 --- /dev/null +++ b/token-capture-lineage-ledger-approach.md @@ -0,0 +1,443 @@ +# Token Capture Lineage Ledger — Approach Plan + +This plan replaces the `RolloutCaptureGate` / `GateStateStore` pair in NeMo Gym's +token-capture path with a small extension to the existing `LineageStore`, turning it +into a per-rollout append-only capture ledger. The external staging contract +(`StagingSink` / `StagingSource`), the worker capture path, and the +`verify_and_linearize()` trust boundary are unchanged. + +**Scope note:** when this change lands, the gate is **removed completely from the Gym +codebase** — `gate.py`, `gate_store.py`, the gate control routes, gate configuration, +and gate tests are deleted in the same change. The gate is not retained behind a +config flag and no backward-compatibility path is kept. Both components ship in the +same PR (NVIDIA-NeMo/Gym PR 2278), so there are no external deployments to migrate. + +## Motivation + +The gate does not provide a second lineage algorithm. Parent resolution runs upstream +through `LineageStore.resolve()`; the gate cross-checks that result against its own +copy of the call state and republishes successful commits back to +`LineageStore.record()`. This creates two sources of truth for the same ancestry: +each call's cumulative token IDs are stored both in `GateCallState` and in the +lineage JSONL. + +The gate's state store is also a scalability liability. `FileGateStateStore` +serializes the **entire global gate state** — every live rollout's cumulative token +arrays and stored request items — and atomically rewrites it under one exclusive +file lock. `commit_coords()` performs three such transactions per model call. For +long-context rollouts this is megabytes of fsynced rewrite per call, serialized +across all serving workers. + +What the gate legitimately provides — admission, retry idempotency, rollout +completeness, terminal selection, cleanup — is either a pure function of the lineage +result or belongs to the framework that already owns the rollout. This design moves +each responsibility to its natural owner and deletes the redundant state machine. + +## Design + +### 1. `LineageStore` becomes the capture ledger + +`FileLineageStore` already writes one append-only JSONL row per committed call +(`model_call_id`, fingerprint, context digest, cumulative token IDs, digest) with +per-rollout locked appends and cross-worker read-after-write visibility. Three +additions make it the single record of rollout capture state: + +1. **`record(...)` gains the token-free `CallRecord` fields.** The commit hook + already holds the worker's `CommitCoords`, so each row additionally stores + `parent_call_id`, `staging_key`, `weight_version`, `prev_len` / `delta_len` / + `cum_len`, `extras_digest`, `mode`, and `logical_request_id` (the request header + when present, else the response id — the same binding the gate's commit uses + today). Additive change; existing fields are untouched. +2. **`record_failure(rollout_id, model_call_id, reason)`** appends a failure row when + a call's coordinates come back `capture_failed` or the request dies after + admission. Failure rows carry no fingerprint, so `resolve()` never returns them + as parents — no filtering logic is needed. +3. **`manifest(rollout_id) -> list[CallRecord]`** reads the rows back token-free + (cumulative token IDs are stripped; token arrays stay off Gym's HTTP surface). + +The only duplication that remains between ledger rows and staged records (digest, +lengths, parent link) is intentional cross-attestation — it is exactly what +`verify_and_linearize()` checks one against the other. + +### 2. Gate-free admission in `resolve_parent()` + +When external staging is enabled, the model server builds the `CaptureAdmission` +directly from the lineage result. Admission is a pure function; no shared state is +required. The lineage outcome is a strict tri-state: + +| Lineage outcome | Admission | +| --- | --- | +| `ROOT` — empty assistant fingerprint, or unmatched fingerprint on a rollout with no ledger rows (seeded assistant history in the task prompt) | `text` mode, no parent | +| `MATCH` — unique fingerprint match with verified context digest | `token_in` mode, `required_prefix_token_ids` = parent's cumulative tokens | +| `UNRESOLVED` — non-empty fingerprint with no match, ambiguity, or digest mismatch | no admission; `record_failure()` poisons the call | + +`UNRESOLVED` is never silently converted into a new root: doing so would turn earlier +policy-generated tokens into mask-zero prompt tokens and corrupt the training row. +The seeded-history carve-out (unmatched fingerprint on an empty rollout is a `ROOT`) +requires a cheap "has any rows" check on the ledger. + +### 3. Gate-free commit + +Where the model server today hands `ng_commit_coords` to `gate.commit_coords()`, +it instead: + +- on `disposition == "staged"`: computes + `cumulative = context.parent_tokens + coords.token_ids_delta` and calls + `lineage_store.record(...)` with the extended row. This is the lineage-publication + block currently inside the gate, relocated — minus the state machine around it. +- on `disposition == "capture_failed"`, missing coordinates, or a request error after + admission: calls `record_failure(...)`. + +The ordering invariant the external sink requires — *a call must not become a +lineage parent until its staged record is durable* — holds structurally: the worker +stages through `StagingSink.stage()` before acknowledging, coordinates exist only +after the bytes are durable, and the ledger row (which is what makes a call +resolvable as a parent) is written only after the coordinates arrive. + +### 4. One read-only control route + +The register / seal / fail control routes are deleted (see removal section). They +are replaced by a single stateless read: + +``` +GET /training-token-capture/rollouts/{rollout_id}/manifest +``` + +which returns `manifest(rollout_id)`. The framework must not read the ledger JSONL +files directly — the route keeps `LineageStore` implementations swappable. + +### 5. Retry handling (fail-closed now, idempotent as a follow-up) + +For the initial change, `model_call_id` stays a per-request `uuid4` minted by the +capture middleware, and a ledger row's `logical_request_id` field is filled the way +the gate's commit binding fills it today: the client header when present, else the +vLLM response id. A retried HTTP request (Gym's `ServerClient` retries with backoff) +therefore lands as a *distinct* call. The ledger handles this fail-closed rather +than idempotently: + +- a retry whose first attempt never committed leaves a failure row (the request + died after admission) → the rollout is poisoned; +- a retry whose first attempt committed becomes a sibling row. If the regenerated + text differs, later calls resolve uniquely to the survivor and the losing sibling + is a dead branch in the manifest (never on the terminal chain); if the text is + identical, resolution is ambiguous → `UNRESOLVED` → poisoned. Terminal selection + still works because the agent reports the response id of the response it actually + kept, which matches exactly one row's `logical_request_id`. + +No retry outcome is silently wrong — the current gate's worst case (ambiguous +resolution silently falling back to a text-mode root) is closed by the `UNRESOLVED` +rule. What is deferred is *recovery*: making identical retries collapse into the +same row instead of poisoning. That follow-up is two touch points — the harness +mints a per-call `x-nemo-gym-logical-request-id` (reused verbatim on retry), and the +middleware derives `model_call_id` deterministically from it — after which +`record()`'s existing semantics absorb retries: an identical payload is a no-op, a +conflicting payload raises and the call is poisoned, and staging keys +(`{rollout_id}/{model_call_id}`) overwrite/no-op rather than orphaning a record. + +### 6. Framework-owned receipt, lifecycle, and cleanup + +NeMo RL (the rollout owner) assembles the `RolloutReceipt` locally at rollout end: + +- `manifest` = the fetched `CallRecord` list, deduped by `model_call_id`; +- `terminal_model_call_id` = the row whose `logical_request_id` matches the + rollout's terminal logical request; +- `capture_poisoned` = any failure row present, or no row for the terminal request. + +`verify_and_linearize(receipt, snapshots)` runs unchanged — it consumes a plain +value and does not care who built it. Cleanup is framework-driven: on publish or on +rollout failure/abandonment, NeMo RL prefix-clears `{rollout_id}/` in the staging +backend and drops the ledger file. No registration TTLs, tombstones, or +cleanup-manifest queues. + +## Complete gate removal (no backward compatibility) + +The same change that introduces the ledger **deletes the gate entirely** from Gym. +Nothing is kept behind a flag, deprecated, or retained for compatibility: + +- `nemo_gym/token_id_capture/gate.py` — deleted. +- `nemo_gym/token_id_capture/gate_store.py` (including `FileGateStateStore`, + `InMemoryGateStateStore`, `SharedGateState`, `GateRolloutState`, `GateCallState`, + `GateTombstone`, `CleanupManifest`, `RolloutRegistration`) — deleted. + `CallRecord` and `RolloutReceipt` survive: they live in `staging/records.py` and + are consumed by `verify_and_linearize()`. +- Control routes `PUT /rollouts/{id}`, `POST /rollouts/{id}/seal`, + `POST /rollouts/{id}/fail`, and the gate metrics route — deleted, replaced by the + single manifest route. +- `token_id_capture.gate.*` configuration (`enabled`, `state_store_path`, + `registration_ttl_s`, `tombstone_ttl_s`) — deleted, not deprecated. A config that + still sets these keys fails validation loudly rather than silently ignoring them. +- Gate wiring in `base_responses_api_model.py` (`RolloutCaptureGate` construction, + `_fail_uncommitted_gate_call`, `_reject_gate_streaming`, capability header + handling) — deleted. Streaming rejection is retained where external capture + requires it, keyed off the capture context rather than the gate. +- `CaptureContext.staging_gate` and `data_capability` fields — deleted. +- Gate unit tests (`test_token_capture_gate.py`, + `test_token_capture_gate_multiworker.py`) — deleted; their invariants that still + apply (admission tri-state, commit ordering, retry idempotency, poisoning) are + re-expressed as ledger and admission tests. + +Rationale for not keeping a compatibility path: the gate ships in the same unmerged +PR as the external sink, so there is no deployed consumer of the gate API. Retaining +it would preserve the dual-source-of-truth problem this design exists to remove and +double the surface that must be tested. + +Data-capability authorization is dropped with the gate. The NeMo RL deployment is a +trusted, framework-owned serving path. If untrusted agent traffic later needs +per-rollout authorization, a stateless signed capability check can be added without +any state store. + +## NeMo RL companion changes + +- `nemo_rl/environments/nemo_gym.py`: remove the register / seal / fail control-route + calls and capability plumbing; fetch the manifest at rollout end; keep configuring + `FileLineageStore` for the policy model server. +- `nemo_rl/experience/blackbox_finalizer.py`: build the `RolloutReceipt` from the + fetched manifest as described above; verification and linearization are unchanged. +- Rollout failure paths (`rollout_manager.py`): prefix-clear staged rows for the + rollout instead of draining gate cleanup manifests. +- `TQTokenSink` / `TQTokenSource` and the vLLM worker capture path + (`vllm_worker_async.py`): unchanged. + +## Example walkthrough + +A SWE-agent rollout `r42_g0` with three model calls, served by a Gym model server +with the ledger and a vLLM worker staging into TransferQueue. + +**Call 1 (root).** The agent server sends the task prompt; the server mints +`model_call_id=c1`, and commit binds `logical_request_id=lr-1` (the response id) +onto the ledger row. The request has no +assistant-authored turns → `ROOT` → `text` admission. The worker generates +(prompt 700 tokens, generation 200), stages +`StagedCallRecord(staging_key="r42_g0/c1", prev_len=0, delta_len=900, cum_len=900, +weight_version=17, ...)` durably into TransferQueue, then returns token-free +coordinates. The server appends ledger row 1 (lineage fields + `CallRecord` fields + +`lr-1`). The agent receives text only. + +**Call 2 (continuation).** The agent appends a 150-token tool result and sends +`lr-2`. The fingerprint uniquely matches row 1 and the context digest verifies → +`MATCH` → `token_in` admission with the 900-token required prefix. The worker +asserts the prompt begins with exactly that prefix, generates 120 tokens, stages +`r42_g0/c2` (`prev_len=900`, `delta_len=270` — 150 carry tokens mask 0 + 120 +generated mask 1, `cum_len=1170`). Ledger row 2 records `parent_call_id=c1`. + +**Call 3 (terminal).** Same shape via `lr-3` → `c3`, `cum_len=1290`, ledger row 3. +The rollout result returns to NeMo RL with the reward and +`terminal_logical_request_id=lr-3`. + +**Finalization.** +1. `GET .../rollouts/r42_g0/manifest` → three `CallRecord`s. +2. Build the receipt: terminal row is `lr-3` → `terminal_model_call_id=c3`; no + failure rows → not poisoned. +3. `StagingSource.fetch(["r42_g0/c1", "r42_g0/c2", "r42_g0/c3"])`, then + `verify_and_linearize(receipt, snapshots)`: digest recomputation, parent-graph + walk `c3 → c2 → c1 → root`, length chaining `900 + 270 = 1170`, + `1170 + 120 = 1290`, mask-order checks. Output: one contiguous 1290-token + training row, mask 1 over the 320 policy tokens. +4. Publish the row with the reward; prefix-clear `r42_g0/*` from TransferQueue and + drop the ledger file. + +## Failure semantics + +- **Capture fails mid-rollout** (e.g. call 2): the model call still succeeds for the + agent, but a failure row is written instead of a lineage row. Call 3 then misses + resolution with a non-empty assistant history → `UNRESOLVED` → failure row. + Finalization sees failure rows → poisoned → masked placeholder row (the group + still publishes exactly N rows); staging is prefix-cleared. +- **Terminal response lost, harness retries:** the retry is admitted as a new call + (uuid identity) and becomes a sibling of the lost attempt. The harness keeps the + retry's response, so its response id is reported as the terminal logical request + and receipt assembly selects the retry's row; the lost attempt is a dead branch + that never joins the terminal chain and is prefix-cleared with the rollout. Two + versions of the same call can never *both* train, and an ambiguous mid-rollout + sibling (identical regenerated text) poisons via `UNRESOLVED` instead of silently + becoming a root. (The deterministic-identity follow-up upgrades this to full + idempotent collapse.) +- **Crash after staging but before the ledger append:** the staged record exists but + the ledger has no row. If the call has descendants, they resolve `UNRESOLVED` and + poison; if it was terminal, receipt assembly finds no terminal row and poisons. + The orphaned staged record is removed by the prefix-clear. Fail-closed with zero + admitted-call bookkeeping. +- **Abandoned rollout:** NeMo RL's failure path prefix-clears TransferQueue and + drops the ledger file. No TTL machinery. + +## Constraints and caveats + +- **`InMemoryLineageStore` cannot serve the ledger role.** Its backing index evicts + rollouts under memory bounds, which is acceptable for a resolution cache but not + for a completeness record. External-sink mode requires a non-evicting store + (`FileLineageStore` or a framework-provided implementation); multi-worker serving + already requires this. The in-memory store remains for single-worker development + and unit tests only. +- **Receipt independence is reduced.** The manifest is derived from the same commit + stream that produced the staged records rather than from an independent state + machine. Per-record integrity is unaffected (digests are recomputed at + finalization); completeness rests on lineage poisoning plus the terminal-row + check, which the failure analysis above shows is fail-closed at the same points + the gate was. +- **Ledger row growth.** Rows store full cumulative token ID arrays (as the lineage + JSONL does today), so a rollout's ledger grows quadratically with call count. + Acceptable at current rollout lengths; if it becomes a problem, rows can store a + prefix reference instead of the full array without changing the contract. + +## PR scope + +Gym (PR 2278): + +- `LineageStore` protocol + `FileLineageStore`: extended `record()`, + `record_failure()`, `manifest()`, has-rows check. +- `sink.py`: gate-free admission tri-state in `resolve_parent()`. +- Commit hook: relocated lineage publication + failure recording. +- New manifest control route; `logical_request_id` stored on ledger rows (header or + response-id fallback). Deterministic `model_call_id` derivation is **out of scope** + (follow-up; see Implementation plan). +- **Full deletion of the gate** (files, routes, config, wiring, tests) as itemized + above. + +NeMo RL (companion): + +- Receipt assembly in the finalizer; manifest fetch in `nemo_gym.py`; removal of + register/seal/fail calls; prefix-clear cleanup on failure paths. + +## Implementation plan + +File-by-file changes, in landing order. Retry *idempotency* (client-minted logical +ids + deterministic `model_call_id`) is explicitly deferred — see the follow-up +section at the end. Gym paths are relative to the `3rdparty/Gym-workspace/Gym` +checkout; NeMo RL paths are relative to the repo root. Line references are against +the current state of both trees. + +### Gym (PR 2278) + +**1. `nemo_gym/token_id_capture/protocols.py` — widen the `LineageStore` protocol.** +`record()` (`protocols.py:74-82`) gains the token-free `CallRecord` fields: +`parent_call_id`, `staging_key`, `weight_version`, `prev_len`, `delta_len`, +`cum_len`, `extras_digest`, `mode`, `logical_request_id`. New protocol methods: +`record_failure(rollout_id, model_call_id, reason)`, +`manifest(rollout_id) -> list[CallRecord]`, and `has_rows(rollout_id) -> bool`. +Additive, so it lands standalone. + +**2. `nemo_gym/token_id_capture/lineage.py` — implement in both stores.** +- `FileLineageStore._record` (`lineage.py:520-527`) extends the six-key JSONL row + with the new fields. The existing same-id idempotency scan (`lineage.py:528-535` + — identical payload no-op, conflicting payload `ValueError`) is kept unchanged. +- `record_failure()` appends a row with `reason` and **no `fingerprint`**; since + `_resolve` (`lineage.py:472-490`) filters by fingerprint, failure rows can never + be returned as parents — no filtering logic needed. +- `manifest()` reads rows under the per-rollout lock, strips + `cumulative_token_ids`, returns validated `CallRecord`s. `has_rows()` is a locked + existence check. +- `InMemoryLineageStore` implements the same methods but is rejected in + external-sink mode (its `LineageIndex` evicts rollouts, `lineage.py:296-344`, + which breaks completeness); it remains for unit tests and single-worker dev. + +**3. `nemo_gym/base_responses_api_model.py` — de-wiring (no identity change).** +- `model_call_id = uuid4().hex` (`:1254`) **stays as-is** for now. +- Delete gate construction and control-route install (`:1481-1498`), the + `GateError` handler (`:1500-1510`), and capability-header handling in the + middleware (`:1263-1278`). +- Replace `_fail_uncommitted_gate_call` (`:1161-1176`, `finally` call site `:1286`) + with a `record_failure(reason="request_finished_without_staged_coordinates")` in + the same `finally` — a request that dies after admission must still poison. +- `CaptureContext` (`token_id_capture/sink.py:53-87`) drops `staging_gate` and + `data_capability`. +- `_reject_gate_streaming` (`:104-111`) is re-keyed off the capture context instead + of the gate; its three dialect call sites (`:235`, `:280`, `:322`) stay. + +**4. `nemo_gym/token_id_capture/sink.py` — tri-state admission in +`resolve_parent()`.** Replace the gate admission step (`sink.py:153-159`) with the +pure function of the lineage result described in Design §2: unique verified match → +`token_in`; empty fingerprint or unmatched fingerprint with `has_rows() == False` +(seeded history) → `text` root; anything else → no admission + +`record_failure(reason="unresolved_parent")`. This closes the gate's silent +root-fallback hole and is load-bearing for the deferred-retry story. + +**5. `responses_api_models/vllm_model/app.py` — gate-free commit hook.** +`_finalize_gate_capture` (`app.py:946-987`) keeps its shape; the gate call +(`:962-966`) becomes: +- `disposition == "staged"` → `cumulative = context.parent_tokens + + coords.token_ids_delta`, then the extended `lineage_store.record(...)` — the + publication block relocated from `gate.py:356-384`; +- `capture_failed`, missing coords, or any exception (including `record()`'s + conflict `ValueError`) → `record_failure(...)`. +The `logical_request_id` binding keeps today's fallback: +`context.logical_request_id or str(payload["id"])`. The existing catch-all that +never turns a valid completion into a harness failure (`:972-985`) stays. + +**6. `nemo_gym/token_id_capture/control_routes.py` — one read-only route.** +Delete `PUT /rollouts/{id}`, `POST .../seal`, `POST .../fail`, `GET /cleanup`, +`GET /metrics`, and the TTL sweeper task (`control_routes.py:74-164`). Add +`GET /training-token-capture/rollouts/{rollout_id}/manifest` returning +`manifest(rollout_id)`. `RolloutControlClient` (`:167-266`) shrinks to `manifest()`. + +**7. Deletions.** `gate.py`, `gate_store.py`, +`test_token_capture_gate*.py`; `token_id_capture.gate.*` config keys removed with +**loud validation failure** on leftovers; the `rebuild_response`-with-gate guard +(`token_id_capture/config.py:162-166`) re-keyed to external capture. +`CallRecord` / `RolloutReceipt` / `CommitCoords` stay in `staging/records.py`; +`verify_and_linearize()` is untouched. Gate test invariants that still apply +(tri-state admission, commit ordering, same-call commit idempotency, poisoning) are +re-expressed as ledger and admission tests. + +### NeMo RL (companion) + +**8. `nemo_rl/environments/nemo_gym.py`.** Delete `register_rollouts` +(`:600-618`), `fail_rollouts` (`:620-637`), `gate_metrics` (`:639-640`), capability +stamping in `run_rollouts` (`:664-672`), and the `gate:` config block (`:521-527`), +keeping the `FileLineageStore` configuration (`:519-520`). +`_postprocess_receipt_mode()` (`:753-812`) replaces the seal call (`:779-789`) with +the manifest `GET`, then builds the `RolloutReceipt` locally: terminal row = the +manifest row whose `logical_request_id` equals the reported +`terminal_logical_request_id` (a response id, unchanged agent behavior); +`capture_poisoned` = any failure row or no terminal row. **Open check:** retry +duplicates now appear as dead-branch rows in the manifest (distinct call ids, same +parent); receipt assembly must either prune to the terminal parent-chain or we must +confirm `_validate_manifest_graph` (`staging/rebuild.py:159`) tolerates rows +unreferenced by the terminal chain. + +**9. `nemo_rl/experience/blackbox_finalizer.py`.** Near-zero delta: the receipt +already arrives by value into `finalize_rollout` (`:241`); every downstream check, +`verify_and_linearize` (`:411-424`), placeholder masking (`:566-590`), and +`_clear_staging` (`:776-786`) are unchanged. Dead-branch rows are in the manifest, +so their staging keys are enumerated and cleaned — no new orphan class. + +**10. `nemo_rl/experience/rollout_manager.py`.** The abandonment path (`:929-935`) +drops `fail_rollouts.remote(...)`; cleanup becomes staging clear + ledger drop. The +`gate_metrics` proxy (`:799-811`) is removed or repointed at ledger-derived counters. + +### Landing order + +1→2 (additive ledger, standalone-testable) → 3→7 as one Gym change → 8→10 as the +NeMo RL companion → end-to-end via `docs/guides/nano-swe-token-capture.md`. + +### Open item to settle before step 10 + +**No prefix-clear primitive exists.** `TQTokenSink.clear()` is explicit-keys-only +(`nemo_rl/data_plane/tq_token_sink.py:280-290`); the layer below +(`data_plane/interfaces.py:415-440`) only offers a producer-local clear-all that may +silently no-op for non-producers. Manifest-enumerated keys cover every row the +ledger knows about, including dead branches. The remaining gap is the +crash-between-stage-and-ledger-append orphan (a staged row no manifest names). +Options: (a) add a prefix/scan delete to the TransferQueue data plane, (b) accept +the leak until partition teardown, (c) have the worker report the staging key in +failure coords so the orphan lands in a failure row and stays enumerable. + +### Follow-up (explicitly out of scope for this change) + +**Retry idempotency via deterministic identity.** The harness mints a unique +`x-nemo-gym-logical-request-id` per model call before dispatch, reuses it verbatim +on retry, and reports the terminal call's logical id (replacing the post-hoc +response-id derivation at `responses_api_agents/swe_agents/app.py:3119`); the +middleware then derives `model_call_id` deterministically from it (`:1254`). +Identical retries collapse into the same row and staging key instead of poisoning +or dead-branching; divergent retries poison via `record()`'s conflict rule. +Optionally layered on top: store the response payload on the ledger row and replay +it on retry (idempotency cache), eliminating divergent regeneration entirely. The +deferred surface is exactly two touch points, so nothing in this plan is redone. + +## Related documents + +- `token-capture-external-sink-recommendation.md` (repo root) — the review that + motivated this design; this document promotes its "per-rollout append-only capture + ledger" option from a future alternative to the recommended mechanism. +- `docs/guides/nano-swe-token-capture.md` — the recipe exercising this path end to + end. From 890a485c7f8816c26a850183786d92118a4d0cc0 Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Thu, 20 Aug 2026 05:09:00 -0700 Subject: [PATCH 32/52] fix(launcher): honour SC_EXP_NAME in the legacy nano SC launcher Co-Authored-By: Claude Fable 5 Signed-off-by: Prashant Thombre --- swe_nano_sc.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/swe_nano_sc.sh b/swe_nano_sc.sh index 594b91c4cc2..cc9f622c294 100644 --- a/swe_nano_sc.sh +++ b/swe_nano_sc.sh @@ -28,7 +28,7 @@ set -a source "${HERE}/swe_nano.env" # --- SingleController + TransferQueue overrides ------------------------------ -EXP_NAME=nano-swe-sc-tq-zhiyul +EXP_NAME="${SC_EXP_NAME:-nano-swe-sc-tq}" NRL_ENTRYPOINT="${CODE_DIR}/examples/run_grpo_single_controller.py" CONFIG_PATH="${CODE_DIR}/examples/configs/ultra/nano_swe_teacher_sc.yaml" RESULTS_DIR="${WORKSPACE_DIR}/results/${EXP_NAME}" From cd61d97847b59802e5e1a873e593fdf044931ad3 Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Thu, 20 Aug 2026 05:20:19 -0700 Subject: [PATCH 33/52] chore: bump Gym pin (lint fix) Co-Authored-By: Claude Fable 5 Signed-off-by: Prashant Thombre --- 3rdparty/Gym-workspace/Gym | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/3rdparty/Gym-workspace/Gym b/3rdparty/Gym-workspace/Gym index 1e1a16cb20c..74d4ef851b2 160000 --- a/3rdparty/Gym-workspace/Gym +++ b/3rdparty/Gym-workspace/Gym @@ -1 +1 @@ -Subproject commit 1e1a16cb20c749985464427048185ae215a5215f +Subproject commit 74d4ef851b29a0185256d256361b485ec8898d63 From b38b3c46f401236a9ae412eb3041f411a4e58304 Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Thu, 20 Aug 2026 05:31:51 -0700 Subject: [PATCH 34/52] fix(launcher): keep UV_CACHE_DIR_OVERRIDE off the prefetch-venvs container Co-Authored-By: Claude Fable 5 Signed-off-by: Prashant Thombre --- swe_nano.env | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/swe_nano.env b/swe_nano.env index b4ada725428..484439cf4d1 100644 --- a/swe_nano.env +++ b/swe_nano.env @@ -134,10 +134,11 @@ export NRL_TQ_SKIP_RUNTIME_ENV_PIN=1 # Shared prewarmed uv cache (survives across jobs; removes compute-node # GitHub dependence for venv rebuilds). Never /tmp: UV_CACHE_DIR rides into -# the driver's TRAIN_CMD (ultra_launch.sh) and UV_CACHE_DIR_OVERRIDE mounts -# the same cache at /root/.cache/uv on every node (ray.sub). +# the driver's TRAIN_CMD (ultra_launch.sh). Do NOT set UV_CACHE_DIR_OVERRIDE +# with the prefetch-venvs containers — it bind-mounts over /root/.cache/uv +# and severs the baked venvs' hardlinked packages (job 6357295: +# ModuleNotFoundError urllib3.exceptions in /opt/nemo_rl_venv). export UV_CACHE_DIR=/lustre/fsw/portfolios/llmservice/users/pthombre/uv -export UV_CACHE_DIR_OVERRIDE=/lustre/fsw/portfolios/llmservice/users/pthombre/uv # Forced venv rebuilds must install from the container's uv.lock, not # re-resolve. From 84feac48bd97710d8eb8faa5fe3a4e0f9eae40c3 Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Thu, 20 Aug 2026 06:31:48 -0700 Subject: [PATCH 35/52] fix(smoke): unblock both nano SWE smoke arms MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - venvs.py: port NRL_FORCE_REBUILD_VENVS_LIST from the swe-r3-capture campaign tree. The capture launcher has always exported it, but this tree never consumed it, so stale node-local VllmAsyncGenerationWorker venvs (no orjson) were silently reused — every 08-19/08-20 capture smoke died in setup_token_capture (jobs 6340634..6343270, 6358267). - nemo_gym.py: tool-call-only assistant output items carry content: None; the invalid-tool-call detector indexed it and killed every legacy-arm run (TypeError, jobs 6342333, 6358268). Treat None as no content message. Co-Authored-By: Claude Fable 5 Signed-off-by: Prashant Thombre --- nemo_rl/environments/nemo_gym.py | 9 ++++----- nemo_rl/utils/venvs.py | 9 +++++++++ 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/nemo_rl/environments/nemo_gym.py b/nemo_rl/environments/nemo_gym.py index 43f87ac3891..8ea4eb740e4 100644 --- a/nemo_rl/environments/nemo_gym.py +++ b/nemo_rl/environments/nemo_gym.py @@ -166,11 +166,10 @@ def _detect_invalid_tool_call_and_malformed_thinking( ) thinking_tags = thinking_tags or DEFAULT_THINKING_TAGS - is_output_message = ( - "content" in output_item_dict - and len(output_item_dict["content"]) > 0 - and "text" in output_item_dict["content"][0] - ) + # A tool-call-only assistant item carries content: None — not a final + # content message. + _content = output_item_dict.get("content") or [] + is_output_message = len(_content) > 0 and "text" in _content[0] # NeMo-Gym only attaches generation_token_ids to the last output item of a # model call (see vllm_model/app.py postprocess_chat_response). So this item # is guaranteed to be the final thing the model produced for this turn. diff --git a/nemo_rl/utils/venvs.py b/nemo_rl/utils/venvs.py index 3afa42c2e7f..a5a3eca9692 100644 --- a/nemo_rl/utils/venvs.py +++ b/nemo_rl/utils/venvs.py @@ -174,6 +174,15 @@ def create_local_venv_on_each_node(py_executable: str, venv_name: str): ray.get(pg.ready()) force_rebuild = os.environ.get("NRL_FORCE_REBUILD_VENVS", "false").lower() == "true" + # NRL_FORCE_REBUILD_VENVS_LIST: comma-separated venv names to rebuild even + # when the global flag is off — for containers whose baked venvs are only + # partially compatible with the checked-out branch. + rebuild_list = { + name + for name in os.environ.get("NRL_FORCE_REBUILD_VENVS_LIST", "").split(",") + if name + } + force_rebuild = force_rebuild or venv_name in rebuild_list # Launch one actor per node actors = [ _env_builder.options(placement_group=pg).remote( From 5c9801dfb1698ad1fac0f045f45ad25f8bbdae47 Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Thu, 20 Aug 2026 07:04:45 -0700 Subject: [PATCH 36/52] feat(data-plane): port NRL_TQ_SKIP_RUNTIME_ENV_PIN from the campaign tree Co-Authored-By: Claude Fable 5 Signed-off-by: Prashant Thombre --- nemo_rl/data_plane/adapters/transfer_queue.py | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/nemo_rl/data_plane/adapters/transfer_queue.py b/nemo_rl/data_plane/adapters/transfer_queue.py index 995cfa24c37..a125f2418e3 100644 --- a/nemo_rl/data_plane/adapters/transfer_queue.py +++ b/nemo_rl/data_plane/adapters/transfer_queue.py @@ -23,6 +23,7 @@ from __future__ import annotations import ipaddress +import logging import os import socket import subprocess @@ -159,6 +160,18 @@ class layout — if TQ restructures, this becomes a no-op with a if _TQ_RUNTIME_ENV_PATCHED: return + # Skip the pin when the deployment declares the base env already ships + # TQ on every node (container-baked, or single-node test runs whose venv + # is shared): the per-actor pip env Ray would build does not inherit the + # parent site-packages and breaks on nodes without network egress. + if os.environ.get("NRL_TQ_SKIP_RUNTIME_ENV_PIN", "") == "1": + logging.getLogger(__name__).info( + "NRL_TQ_SKIP_RUNTIME_ENV_PIN=1: not injecting the TransferQueue " + "pip pin into TQ actor runtime_envs (base-env TQ assumed)." + ) + _TQ_RUNTIME_ENV_PATCHED = True + return + runtime_env = {"pip": [_resolve_tq_pin()]} def _install(cls) -> bool: From 5bc2b50d1af4b56eba99e6d984230e82cfee65ea Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Thu, 20 Aug 2026 09:16:02 -0700 Subject: [PATCH 37/52] test(nemo-gym): regression cases for tool-call-only output items + A/B report - content: None / missing-content items return (False, False) from the invalid-tool-call detector (the crash that killed the legacy smoke arms). - reports/auto_research/lineage-ledger-0820: 5-step ledger-vs-legacy smoke A/B (jobs 6359951/6359952): both arms 5/5, capture dynamics strictly tighter (token_mult_prob_error 1.0137-1.0147 vs 1.020-1.066; gen_kl ~0.001 vs ~0.004), ledger 2821 rows with zero unresolved/worker failures. Co-Authored-By: Claude Fable 5 Signed-off-by: Prashant Thombre --- nemo_rl/environments/nemo_gym.py | 4 +- .../lineage-ledger-0820/experiments.tsv | 7 ++ .../lineage-ledger-0820/ledger-ab-report.md | 96 +++++++++++++++++++ session/20260820_004547/handoff.md | 18 ++++ session/20260820_004547/session_state.md | 37 +++++++ session/20260820_004547/timeline.md | 35 +++++++ .../unit/environments/test_nemo_gym_utils.py | 12 +++ 7 files changed, 207 insertions(+), 2 deletions(-) create mode 100644 reports/auto_research/lineage-ledger-0820/experiments.tsv create mode 100644 reports/auto_research/lineage-ledger-0820/ledger-ab-report.md create mode 100644 session/20260820_004547/handoff.md create mode 100644 session/20260820_004547/session_state.md create mode 100644 session/20260820_004547/timeline.md diff --git a/nemo_rl/environments/nemo_gym.py b/nemo_rl/environments/nemo_gym.py index 8ea4eb740e4..d9e1af4c41b 100644 --- a/nemo_rl/environments/nemo_gym.py +++ b/nemo_rl/environments/nemo_gym.py @@ -168,8 +168,8 @@ def _detect_invalid_tool_call_and_malformed_thinking( # A tool-call-only assistant item carries content: None — not a final # content message. - _content = output_item_dict.get("content") or [] - is_output_message = len(_content) > 0 and "text" in _content[0] + item_content = output_item_dict.get("content") or [] + is_output_message = len(item_content) > 0 and "text" in item_content[0] # NeMo-Gym only attaches generation_token_ids to the last output item of a # model call (see vllm_model/app.py postprocess_chat_response). So this item # is guaranteed to be the final thing the model produced for this turn. diff --git a/reports/auto_research/lineage-ledger-0820/experiments.tsv b/reports/auto_research/lineage-ledger-0820/experiments.tsv new file mode 100644 index 00000000000..c1aac70f43b --- /dev/null +++ b/reports/auto_research/lineage-ledger-0820/experiments.tsv @@ -0,0 +1,7 @@ +index branch parent_commit commit recipe metric_name metric_value elapsed_min launcher job_id command log_path status description +1 autoresearch/2026-08-20-lineage-ledger/impl c9602d447 1f9af7987 nano_swe_teacher_sc smoke_5step_dynamics PENDING swe batch 6358267 DRY_RUN=0 SC_EXP_NAME=ledger-capture-s43-0820 NG_TIC_FP_CANONICAL=1 WALLTIME=3:59:00 WANDB_PROJ=PR3456-Ledger-AB-0820 bash swe_nano_sc_capture.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 token_capture.num_finalizer_workers=2 token_capture.defer_routed_experts_to_policy=true token_capture.staging_partition=rollout_staging_ledger_smoke_0820 +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 +env.nemo_gym.model_endpoint_readiness_timeout_seconds=1800 policy.generation.vllm_cfg.reasoning_parser_plugin=/opt/nemo-rl/nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/results/ledger-capture-s43-0820 running Capture arm: ledger-authoritative token capture + R3, num_workers=2, seed 43, 5 steps +2 autoresearch/2026-08-20-lineage-ledger/impl c9602d447 890a485c7 nano_swe_teacher_sc smoke_5step_dynamics PENDING swe batch 6358268 DRY_RUN=0 SC_EXP_NAME=ledger-legacy-s43-0820 WALLTIME=3:59:00 WANDB_PROJ=PR3456-Ledger-AB-0820 bash swe_nano_sc.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 +env.nemo_gym.model_endpoint_readiness_timeout_seconds=1800 policy.generation.vllm_cfg.reasoning_parser_plugin=/opt/nemo-rl/nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/results/ledger-legacy-s43-0820 running Legacy arm: token-echo path + R3, num_workers=2, seed 43, 5 steps +3 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 (venv+legacy fix) nano_swe_teacher_sc smoke_5step_dynamics FAILED swe batch 6358267 (same as row 1) .../ledger-capture-s43-0820 crash setup_token_capture ModuleNotFoundError orjson: NRL_FORCE_REBUILD_VENVS_LIST unconsumed in this tree (stale node venv). Fixed in venvs.py. +4 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 (venv+legacy fix) nano_swe_teacher_sc smoke_5step_dynamics FAILED swe batch 6358268 (same as row 2) .../ledger-legacy-s43-0820 crash Pre-existing legacy bug: tool-call-only output item content=None -> TypeError in invalid-tool-call detector (also killed 0819 legacy runs). Fixed. +5 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 84feac48b nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 tmpe 1.0137-1.0147 gen_kl ~0.001 token_in_rate 0.95-0.99 59 swe batch 6359951 SC_EXP_NAME=ledger-capture-s43-0820-r2 ... (row 1 overrides, staging partition r3) .../ledger-capture-s43-0820-r2 keep Capture arm PASS: dynamics tighter than legacy; 16/40 rollouts fail-closed placeholders (per-call aborts; retry-idempotency follow-up); see ledger-ab-report.md +6 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 84feac48b nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 tmpe 1.020-1.066 gen_kl ~0.004 all-8-valid 67 swe batch 6359952 SC_EXP_NAME=ledger-legacy-s43-0820-r2 ... (row 2 overrides) .../ledger-legacy-s43-0820-r2 keep Legacy arm PASS: looser dynamics (echo re-tokenization); no rollout loss; see ledger-ab-report.md diff --git a/reports/auto_research/lineage-ledger-0820/ledger-ab-report.md b/reports/auto_research/lineage-ledger-0820/ledger-ab-report.md new file mode 100644 index 00000000000..68592ac2e02 --- /dev/null +++ b/reports/auto_research/lineage-ledger-0820/ledger-ab-report.md @@ -0,0 +1,96 @@ +# Lineage-Ledger Token Capture — 5-Step Smoke A/B (2026-08-20) + +**Goal:** validate the gate→ledger replacement (`token-capture-lineage-ledger-approach.md`) +end to end: external token capture ON (ledger) vs OFF (legacy token echo), both with +router replay (R3) enabled and the Gym policy model server at `num_workers=2`. + +**Setup** (identical across arms, per `docs/guides/nano-swe-token-capture.md`): +Nemotron-3-Nano-30B-A3B on 6 GB200 nodes (train 4 / gen 2), `grpo.seed=43`, +`num_prompts_per_step=2`, GBS 8, 5 steps, `+policy.router_replay.enabled=true`, +windowed sampler `max_staleness_versions=1`. Capture arm additionally: +`token_capture.enabled=true`, `defer_routed_experts_to_policy=true`, +`num_finalizer_workers=2`, `NG_TIC_FP_CANONICAL=1`. +Code: RL `autoresearch/2026-08-20-lineage-ledger/impl`, Gym `token-capture-lineage-ledger`. +W&B: `PR3456-Ledger-AB-0820`. + +| Arm | Job | Result | +|---|---|---| +| capture (ledger) | 6359951 | COMPLETED, 5/5 steps, 58:44 | +| legacy (token echo) | 6359952 | COMPLETED, 5/5 steps, 1:07:03 | + +## Training dynamics (per step, 1→5) + +| Metric | Capture (ledger) | Legacy (echo) | +|---|---|---| +| `token_mult_prob_error` | 1.0147, 1.0145, 1.0137, 1.0144, 1.0147 | 1.0199, 1.0319, 1.0657, 1.0347, 1.0283 | +| `gen_kl_error` | 0.00090, 0.00092, 0.00100, 0.00117, 0.00106 | 0.00216, 0.00403, 0.00371, 0.00377, 0.00384 | +| `reward` (mean) | 0, 0, 0.25, 0, 0 | 0, 0, 0.375, 0, 0 | +| `probs_ratio` | 1.0 all steps | 1.0 all steps | + +**Verdict: training dynamics are similar and the capture arm is strictly tighter** — +the same exact-token + exact-route property the gate-based runs showed +(prior campaign bands: capture 1.012–1.017 / ~0.001 vs legacy 1.023–1.027 / ~0.003). +Both arms are far below the `seq_logprob_error_threshold=2.0` health gate, rewards +spike on the same step (3), and the ratio stays pinned at 1.0. + +## Ledger health (capture arm) + +- **2,821 ledger rows across 44 rollouts; zero `unresolved_parent`, zero + `worker_capture_failed`, zero `invalid_worker_commit_coordinates`.** Lineage + resolution and tri-state admission worked flawlessly across 2 uvicorn workers + (no request affinity): `finalize/token_in_rate` 0.954–0.987 per group + (last step: 306 token-in calls vs 4 text roots), `routed_experts_row_coverage=1.0`. +- **16 poison rows, all `request_finished_without_staged_coordinates`** → + 16/40 trained rollouts masked as placeholders (`invalid_row_rate` 0–0.625 per + group; `num_valid_samples` 3–8 of 8). Root cause: ~0.57% of model calls + (16/2,821) finish without a worker ack — agent-side aborts/timeouts plus two + engine `ClientOSError` retry bursts — and one such call fail-closes its whole + rollout. With ~64 calls/rollout this amplifies to ~36% rollout loss. This is + the *designed* fail-closed behavior (identical semantics to the gate's + `fail_call`), and the documented retry-idempotency follow-up (harness-minted + logical ids + deterministic `model_call_id`) is the recovery path. The legacy + arm keeps such rollouts because re-tokenized echo text doesn't need per-call + acks — at the cost of the looser dynamics above. + +## Perf + +| Metric (step 5 / summary) | Capture | Legacy | +|---|---|---| +| `timing/train/total_step_time` (s) | 392.4 | 573.3 | +| `timing/train/exposed_generation` (s) | 368.6 | 542.3 | +| `timing/train/policy_training` (s) | 9.5 | 14.4 | +| `timing/train/valid_tokens_per_sec_per_gpu` | 6.84 | 8.08 | +| Row assembly (finalizer, per group, ms) | fetch+verify+linearize 250–650; tensorize 11–21; tq_put 41–44; total 313–725 | n/a (inline echo path) | +| `finalize/queue_wait_ms` | ≤0.013 | n/a | +| Wall clock, 5 steps e2e | 58:44 | 1:07:03 | + +Reading: the capture arm's off-hot-path finalizer costs are sub-second per group +and its queue never backs up. Its lower `valid_tokens_per_sec_per_gpu` is an +artifact of masked placeholder rows (fewer valid tokens per identical step), not +slower machinery — total step time is *shorter* than legacy, partly because +poisoned rollouts contribute less generation. At smoke scale (2 prompts/step) +these arms are not a rigorous throughput comparison; the prior campaign's +15-step pairs remain the perf reference. + +## Fixes landed to get here (all committed on the impl branch) + +1. `venvs.py`: `NRL_FORCE_REBUILD_VENVS_LIST` was never consumed in this tree — + every 08-19/08-20 capture smoke (8 jobs) died on stale worker venvs + (`orjson` missing in `setup_token_capture`). +2. `nemo_gym.py`: tool-call-only assistant items (`content: None`) crashed the + legacy postprocess (`TypeError`) — killed every legacy run on this branch; + regression tests added. +3. `swe_nano.env`: `UV_CACHE_DIR_OVERRIDE` must not mount over the prefetch-venvs + container's `/root/.cache/uv` (severs baked-venv hardlinks). Driver uv cache + pinned to `/lustre/fsw/portfolios/llmservice/users/pthombre/uv` (never /tmp). +4. `transfer_queue.py`: ported `NRL_TQ_SKIP_RUNTIME_ENV_PIN`. +5. `swe_nano_sc.sh`: honour `SC_EXP_NAME`. + +## Follow-ups + +- Retry idempotency (plan's explicit deferral): recover the ~36% rollout + poisoning from per-call aborts. +- Investigate the vLLM engine `ClientOSError` bursts (engine 13015) — connection + drops under concurrent capture traffic. +- CI (`/ok to test`) to run the Ray-heavy RL suites (blackbox_finalizer, + tq_token_sink, vllm hosting) that cannot run outside the container. diff --git a/session/20260820_004547/handoff.md b/session/20260820_004547/handoff.md new file mode 100644 index 00000000000..d3a53a68f15 --- /dev/null +++ b/session/20260820_004547/handoff.md @@ -0,0 +1,18 @@ +# Handoff + +## Resume From Here +Implementing token-capture-lineage-ledger-approach.md. Gym side DONE and committed (Gym branch `token-capture-lineage-ledger`, commit 1e1a16cb; gate deleted, ledger implemented, 436 Gym tests green). RL side code changes DONE but UNCOMMITTED on RL branch `autoresearch/2026-08-20-lineage-ledger/impl`: nemo_gym.py (external_staging config, manifest fetch + local receipt assembly via `_assemble_receipt`), rollout_manager.py (fail_rollouts/gate_metrics removed), single_controller.py (_log_gate_metrics removed), blackbox_finalizer.py (finalize/token_in_rate + capture_poisoned_rollouts metrics), config.py (TTL fields dropped), exemplar YAML, docs (design-docs/token-capture-ledger.md created; guide updated), rewritten tests (test_nemo_gym_token_capture.py, test_rollout_manager.py updates). RL unit tests running in background task baao872q9. + +## Next Actions +- Confirm RL unit tests green (env tests, blackbox_finalizer, rollout_manager, tq_token_sink, vllm_token_capture_hosting); fix failures; then commit RL side (sign-off + Co-Authored-By Claude Fable 5). +- Launch 5-step smoke A/B from repo root (fsw path): + - capture arm: `DRY_RUN=0 SC_EXP_NAME=ledger-capture-s43-0820 NG_TIC_FP_CANONICAL=1 WALLTIME=3:59:00 WANDB_PROJ=PR3456-Ledger-AB-0820 bash swe_nano_sc_capture.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 token_capture.num_finalizer_workers=2 token_capture.defer_routed_experts_to_policy=true token_capture.staging_partition=rollout_staging_ledger_smoke_0820 +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 +env.nemo_gym.model_endpoint_readiness_timeout_seconds=1800 policy.generation.vllm_cfg.reasoning_parser_plugin=/opt/nemo-rl/nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py` + - legacy arm: same via `swe_nano_sc.sh` (no NG_TIC_FP_CANONICAL, no token_capture overrides, keep num_workers=2 + R3 + sampler + seed 43, SC_EXP_NAME=ledger-legacy-s43-0820). + - dry-run of capture arm validated: UV cache resolves to /lustre/fsw/portfolios/llmservice/users/pthombre/uv (was /tmp; fixed via UV_CACHE_DIR/UV_CACHE_DIR_OVERRIDE in swe_nano.env). +- Compare token_mult_prob_error, gen_kl_error, reward across 5 steps + perf (row_assembly/* timings, step time, finalize/token_in_rate ≥0.99) per docs/guides/nano-swe-token-capture.md; write TSV ledger in reports/auto_research/lineage-ledger-0820/. + +## Watch Outs +- swe_nano.env was retargeted to pthombre paths + nightly-gym 08-10 container (commit with the work). swe_nano.secrets.env copied, chmod 600, git-excluded via .git/info/exclude. +- Prior 08-19 ledger smoke runs (pr3456-ledger-*-s43-0819*) in RL-pr3456-delta-smoke-workspace came from a DIFFERENT since-reverted implementation — reference only for launch shape, not results. +- Slurm submits from a networked shell; jobs monitored via slurm-broker MCP or squeue. +- Do not stop before both arms complete 5 steps and comparison is written (user stop rule). diff --git a/session/20260820_004547/session_state.md b/session/20260820_004547/session_state.md new file mode 100644 index 00000000000..5db2c73442b --- /dev/null +++ b/session/20260820_004547/session_state.md @@ -0,0 +1,37 @@ +# Session State + +- Session: 20260820_004547 +- Repo: /lustre/fs1/portfolios/llmservice/projects/llmservice_fm_text/users/pthombre/sweRun/RL-pr3456-delta-staging +- Branch: autoresearch/2026-08-18-nano35-dolphin-delta-r3/stage1-treatment-gate (start) +- Started: 2026-08-20 00:45 +- Updated: 2026-08-20 00:45 + +## Goal +Implement `token-capture-lineage-ledger-approach.md` (replace Gym's RolloutCaptureGate/GateStateStore with a LineageStore-based capture ledger; full gate deletion; NeMo RL companion changes). Validate via unit tests, then run a 5-step smoke A/B (external token capture on vs off, router replay enabled, Gym model server num_workers>1) and compare training dynamics + perf metrics per docs/guides/nano-swe-token-capture.md. uv cache: /lustre/fsw/portfolios/llmservice/users/pthombre/uv (never /tmp). + +## Current Subtask +Reading target code paths in Gym + RL (task #1). + +## Loaded Skills +- `nemo-rl-auto-research` — campaign workflow, branching, TSV ledger, stop rules. +- `nemo-rl-session-memory` — this record. + +## Current Status +- RL worktree has pre-existing UNCOMMITTED gate→ledger doc/comment renames across 12 files (verified: no functional changes) + untracked docs/design-docs/rollout-verification-boundary.md and two plan MDs. These are prep for this plan; carry them onto the impl branch. +- Gym checkout (3rdparty/Gym-workspace/Gym) clean at 10b34908 on token-capture-worker-custody-rebased; gate.py/gate_store.py still present. +- Prior campaign reference: /lustre/.../sweRun/RL/reports/auto_research/swe-r3-capture (CAMPAIGN.md read — smoke pattern: nano_swe_teacher_sc.yaml, pps=2/GBS=8/3-5 steps, Slurm, shared uv cache via NRL_UV_CACHE_DIR, NG_TIC_FP_CANONICAL=1, metrics token_mult_prob_error/gen_kl). + +## Plan +- [ ] #1 Read code paths, verify plan line refs +- [ ] #2 Gym ledger implementation + gate deletion + tests +- [ ] #3 NeMo RL companion changes +- [ ] #4 Unit tests green +- [ ] #5 5-step smoke A/B (capture on/off, R3 on, num_workers>1) +- [ ] #6 Compare dynamics + perf + +## Assumptions +- Smoke runs go to Slurm via the pattern in nano-swe-token-capture.md / prior campaign (slurm-broker MCP available). +- Pre-existing dirty rename files are intentional prep and should be committed with the work. + +## Blockers +- None known. diff --git a/session/20260820_004547/timeline.md b/session/20260820_004547/timeline.md new file mode 100644 index 00000000000..9bf8a6de934 --- /dev/null +++ b/session/20260820_004547/timeline.md @@ -0,0 +1,35 @@ +# Timeline + +## 2026-08-20 00:45 +- User asked: implement token-capture-lineage-ledger-approach.md, unit-test, then 5-step smoke A/B (capture on/off, R3 on, num_workers>1), compare dynamics+perf. +- Context gathered: plan doc; CAMPAIGN.md of prior swe-r3-capture campaign; Gym gate/lineage/sink/control_routes/records/rebuild/config/base_responses_api_model/vllm app; RL nemo_gym.py/blackbox_finalizer/rollout_manager/single_controller. +- Key findings: + - Pre-existing uncommitted RL changes = pure gate→ledger doc renames (safe to carry). + - `_validate_manifest_graph` (staging/rebuild.py:159) tolerates dead-branch rows → plan's step-8 open check resolved: no pruning needed in receipt assembly. + - RL vLLM worker imports only `token_id_capture.staging.*` → worker path untouched by gate deletion. + - Lineage JSONL files were never cleaned in gate mode either → skipping ledger-file drop is no regression (documented follow-up). + - Gate metrics route dies with the gate → replace with finalizer-derived mode counters (manifest rows carry mode) logged as finalize/token_in_rate. +- Design decisions: + - New Gym config flag `token_id_capture.external_staging: bool` replaces `gate:` block (extra="forbid" on TokenIdCaptureSettings makes leftover gate keys fail loudly). Requires enabled + rebuild_response=false; keeps control_auth_token_env for manifest-route bearer auth. + - Ledger row: existing 6 lineage keys + parent_call_id/staging_key/weight_version/prev_len/delta_len/cum_len/extras_digest/mode/logical_request_id + staging_digest (coords.digest; distinct from lineage digest). Failure rows: model_call_id+failure_reason, no fingerprint (resolve uses .get so they can't match). + - CallRecord gains optional logical_request_id; new RolloutManifest/ManifestFailure wire models in staging/records.py; manifest route GET /training-token-capture/rollouts/{id}/manifest. + - CaptureContext: staging_gate/data_capability deleted; adds external_staging bool + request_items (stashed in resolve_parent for commit-time record()). + - Data-capability plumbing fully removed (rollout_correlation, server_utils, base agents, swe sandbox mount); swe sandbox token-capture URL prefix re-keyed to agent-level token_id_capture enabled. + - Abandoned-rollout staging rows: accept leak until partition teardown (plan option b), fail_rollouts deleted. +- Result: starting Gym implementation. +- 2026-08-20 01:39:46 Gym side committed (1e1a16cb): ledger implemented, gate deleted, 436 Gym tests green (ledger 23, capture 112, swe 158, vllm 143). Starting RL companion. + +## 2026-08-20 06:30 +- RL companion committed (1f9af7987 + launcher fixes 890a485c7, uv-cache fix, Gym pin bumps; Gym lint fix 74d4ef85). +- Smoke A/B submitted: capture=6358267, legacy=6358268 (first pair 6357295/6357320 failed/cancelled — UV_CACHE_DIR_OVERRIDE mount severed the container venv's hardlinked packages; removed from swe_nano.env). +- RL unit tests: receipt-assembly 6/6 green (--noconftest). Ray-fixture suites hang on the login node (conftest autouse init_ray + TQ actors); moved to compute-node job 6359381. + +## 2026-08-20 07:40 +- Unit-test scope closed: Gym capture suites all green (436); RL receipt-assembly (6) + rollout-manager (16) green. Ray-heavy RL suites (blackbox_finalizer/tq_token_sink/vllm hosting) cannot run on login node (conftest autouse Ray wedge) nor bare/pyxis compute shells (uv-run worker env drift; /opt/nemo_rl_venv is ray.sub-materialized). They are exercised end-to-end by the smoke and will run in CI. +- Ported NRL_TQ_SKIP_RUNTIME_ENV_PIN into transfer_queue.py (committed) while diagnosing. +- Smoke r2: capture 6359951 TRAINING (step 2/5 at ~37 min — venv rebuild-list fix confirmed). Legacy 6359952 still queued. + +## 2026-08-20 09:10 +- BOTH SMOKE ARMS PASSED 5/5 steps. Capture: tmpe 1.0137-1.0147, gen_kl ~0.001, token_in_rate 0.95-0.99, 2821 ledger rows / 0 unresolved / 0 worker failures; 16 rollouts fail-closed on aborted calls (request_finished_without_staged_coordinates — retry-idempotency follow-up). Legacy: tmpe 1.020-1.066, gen_kl ~0.004, all 8 samples valid every step. Capture strictly tighter — matches gate-era campaign bands. +- Review of legacy fix done via /review-pr (local): fix correct; added 2 regression params to test_nemo_gym_utils (7/7 pass); renamed _content -> item_content. +- Report: reports/auto_research/lineage-ledger-0820/ledger-ab-report.md. diff --git a/tests/unit/environments/test_nemo_gym_utils.py b/tests/unit/environments/test_nemo_gym_utils.py index fa60df05fb1..904f3ec155f 100644 --- a/tests/unit/environments/test_nemo_gym_utils.py +++ b/tests/unit/environments/test_nemo_gym_utils.py @@ -55,6 +55,18 @@ True, False, ), + # A tool-call-only assistant item carries content: None — a structured + # (executed) call, never a penalty (regression: jobs 6342333/6358268). + ( + {"content": None, "tool_calls": [{"function": {"name": "bash"}}]}, + False, + False, + ), + ( + {}, + False, + False, + ), ], ) def test_detect_invalid_tool_call_and_malformed_thinking( From b39dbd8cdf6db249bdd28aa4b2d51399181aa7e0 Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Thu, 20 Aug 2026 09:54:49 -0700 Subject: [PATCH 38/52] fix(token-capture): do not poison rollouts on uncommitted off-chain calls MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A failure row with reason request_finished_without_staged_coordinates marks a call that never returned a completion (the ledger commit precedes the response leaving the server) and can never be a lineage parent (an uncommitted call has no row to resolve against). It is structurally off-chain and must not mask an otherwise complete, verified terminal chain. In the 0820 smoke A/B all 16/40 capture-arm placeholder rows came from exactly this: the rollout filled the 49,152-token context window and its doomed final call died without coordinates (failure row always last, chain verified to cum_len ~49k, no descendants). Legacy trains these truncated rollouts; the ledger arm now does too — with exact tokens. The fail-closed rule is unchanged for served-but-unstaged calls (worker_capture_failed / invalid_worker_commit_coordinates), and a terminal request that died uncommitted still poisons via the missing-terminal-row check. Co-Authored-By: Claude Fable 5 Signed-off-by: Prashant Thombre --- nemo_rl/environments/nemo_gym.py | 29 +++++++++-- .../test_nemo_gym_token_capture.py | 48 +++++++++++++++++++ 2 files changed, 73 insertions(+), 4 deletions(-) diff --git a/nemo_rl/environments/nemo_gym.py b/nemo_rl/environments/nemo_gym.py index d9e1af4c41b..b5db2a0ca04 100644 --- a/nemo_rl/environments/nemo_gym.py +++ b/nemo_rl/environments/nemo_gym.py @@ -143,6 +143,10 @@ class NemoGymConfig(TypedDict): _NG_ROLLOUT_ID_BODY_KEY = "_ng_rollout_id" _TOKEN_CAPTURE_CONTROL_PREFIX = "/training-token-capture/control" _TOKEN_CAPTURE_CONTROL_ENV = "NEMO_GYM_TOKEN_CAPTURE_CONTROL_TOKEN" +# Mirrors nemo_gym.token_id_capture.sink.UNCOMMITTED_CALL_REASON: the poison +# reason for an admitted call that finished without worker coordinates (no +# completion was ever served for it). +_UNCOMMITTED_CALL_REASON = "request_finished_without_staged_coordinates" def _detect_invalid_tool_call_and_malformed_thinking( @@ -740,8 +744,20 @@ def _assemble_receipt( dead-branch rows: they stay in the manifest (their staged rows are fetched, verified, and cleaned) but never join the terminal chain — ``verify_and_linearize`` tolerates rows unreferenced by the terminal - chain. Poisoning is fail-closed: any failure row, or no row for the - terminal request, masks the rollout. + chain. + + Poisoning is fail-closed with one carve-out. A failure row whose + reason is ``request_finished_without_staged_coordinates`` is a call + that never returned a completion (the ledger commit precedes the + response leaving the server) and can never be a lineage parent (an + uncommitted call has no row to resolve against) — e.g. the doomed + final call of a rollout that exhausted the model's context window. + Such rows are structurally off-chain and do not poison; if the + *terminal* request itself died this way, the missing-terminal-row + check below still masks the rollout. Every other failure reason + (``worker_capture_failed``, ``invalid_worker_commit_coordinates``) + marks a call whose completion WAS served — a hole in the chain — + and poisons. """ records = [dict(record) for record in manifest.get("records") or []] failures = list(manifest.get("failures") or []) @@ -756,9 +772,14 @@ def _assemble_receipt( ), None, ) + poisoning_failures = [ + failure + for failure in failures + if str(failure.get("reason") or "") != _UNCOMMITTED_CALL_REASON + ] failure_reason = None - if failures: - failure_reason = str(failures[0].get("reason") or "capture_failed") + if poisoning_failures: + failure_reason = str(poisoning_failures[0].get("reason") or "capture_failed") elif terminal_record is None: failure_reason = "missing_terminal_row" return { diff --git a/tests/unit/environments/test_nemo_gym_token_capture.py b/tests/unit/environments/test_nemo_gym_token_capture.py index 3c540577d74..a561e8cda24 100644 --- a/tests/unit/environments/test_nemo_gym_token_capture.py +++ b/tests/unit/environments/test_nemo_gym_token_capture.py @@ -102,6 +102,54 @@ def test_receipt_assembly_poisons_on_failure_rows() -> None: assert receipt["failure_reason"] == "worker_capture_failed" +def test_receipt_assembly_ignores_uncommitted_call_failures_off_the_terminal_chain() -> None: + """A call that died without coordinates never served a completion and can + never be a lineage parent (no committed row to resolve against), so it is + structurally off-chain — e.g. the doomed final call of a rollout that + exhausted the context window. It must not poison the verified chain.""" + env = _capture_env() + manifest = { + "rollout_id": "r0", + "records": [ + _manifest_record("c1", logical_request_id="lr-1"), + _manifest_record("c2", logical_request_id="lr-2", parent="c1"), + ], + "failures": [ + { + "model_call_id": "c3", + "reason": "request_finished_without_staged_coordinates", + } + ], + } + receipt = env._assemble_receipt( + "r0", manifest, terminal_logical_request_id="lr-2", reward=1.0 + ) + assert receipt["capture_poisoned"] is False + assert receipt["failure_reason"] is None + assert receipt["terminal_model_call_id"] == "c2" + + +def test_receipt_assembly_still_poisons_when_the_terminal_call_died_uncommitted() -> None: + """If the reported terminal request itself died without coordinates there + is no terminal row — the missing-terminal check must mask the rollout.""" + env = _capture_env() + manifest = { + "rollout_id": "r0", + "records": [_manifest_record("c1", logical_request_id="lr-1")], + "failures": [ + { + "model_call_id": "c2", + "reason": "request_finished_without_staged_coordinates", + } + ], + } + receipt = env._assemble_receipt( + "r0", manifest, terminal_logical_request_id="lr-2", reward=0.0 + ) + assert receipt["capture_poisoned"] is True + assert receipt["failure_reason"] == "missing_terminal_row" + + def test_receipt_assembly_poisons_when_the_terminal_row_is_missing() -> None: env = _capture_env() manifest = { From 53713adebfba29cdb0bc79b2fe196e1b432f5bcc Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Thu, 20 Aug 2026 11:17:14 -0700 Subject: [PATCH 39/52] test(nemo-gym): port nano-3.5 dolphin SC smoke launchers for multi-worker ledger revalidation Ported unchanged from nemo-rl-partial-rollout-recovery-refresh so the nano-35-rlvr-sc-tc-smoke shape (8 nodes, TP4xCP4xEP8, 8 prompts x 16 gens, policy_model num_workers=16 / reasoning_off 4) can rerun against the lineage-ledger token capture. Job 6300221 failed on this shape: the old per-process gate registry 409'd unregistered workers and raised UnknownRolloutError mid-flight. The recipe's rollout_checkpointing block is extra-allowed (ignored) in this tree. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre --- .../nano35_dolphin_launch.sh | 417 ++++++++++++++++++ .../nano35_dolphin_launch_sc.sh | 199 +++++++++ .../nano35_dolphin_launch_sc_smoke.sh | 118 +++++ .../nemotron-3.5-nano/rlvr_dolphin.yaml | 336 ++++++++++++++ .../nemotron-3.5-nano/rlvr_dolphin_sc.yaml | 341 ++++++++++++++ 5 files changed, 1411 insertions(+) create mode 100755 examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch.sh create mode 100755 examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch_sc.sh create mode 100755 examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch_sc_smoke.sh create mode 100644 examples/nemo_gym/nemotron-3.5-nano/rlvr_dolphin.yaml create mode 100644 examples/nemo_gym/nemotron-3.5-nano/rlvr_dolphin_sc.yaml diff --git a/examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch.sh b/examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch.sh new file mode 100755 index 00000000000..107efa32697 --- /dev/null +++ b/examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch.sh @@ -0,0 +1,417 @@ +#!/bin/bash +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -euo pipefail + +# ============================================================================= +# nano35_dolphin_launch.sh +# +# Nemotron 3.5 Nano — RLVR, legacy async-1, honest-dolphin warm start. +# Reproduces the internal reference run +# geshen-ultra-rl-nano-honest-dolphin-v10-iter6000-mopd-rlvr +# (launch_nano_honest_dolphin.sh + grpo_ultra_512n4g_bf16.yaml on +# nemo-rl-internal @ 97c55ee2) on public NeMo-RL main. +# +# This is a thin wrapper over examples/nemo_gym/nemotron-3-ultra/ultra_launch.sh. +# That launcher is fully parameterised and already handles code snapshotting, +# persistent-cache seeding, container mounts, Ray/Gym orchestration, and the +# OccupiedIdleGPUsJobReaper --comment exemption — so we set environment and +# delegate rather than forking 800+ lines. +# +# Usage. Judges run one of two ways; see the GenRM section below for the +# trade-off between them. +# +# Against a warm out-of-band GenRM pool (default; must already be serving): +# GENRM_BASE_URL=http://:9213/v1 \ +# bash examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch.sh +# +# Hosting GenRM and the NL2Bash judge inside the job, as the 6K recipe does: +# EXTERNAL_JUDGES=1 \ +# bash examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch.sh +# +# DRY_RUN=1 GENRM_BASE_URL=... bash .../nano35_dolphin_launch.sh # inspect only +# +# Extra Hydra overrides are forwarded verbatim: +# GENRM_BASE_URL=... bash .../nano35_dolphin_launch.sh grpo.max_num_steps=2 +# ============================================================================= + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" +cd "${REPO_ROOT}" # ultra_launch.sh derives PROJECT_ROOT from $PWD + +# ----------------------------------------------------------------------------- +# GenRM hosting. Two shapes, and this recipe supports both. +# +# EXTERNAL_JUDGES=1 is the 6K shape: GenRM and the NL2Bash judge are +# co-scheduled in a second Slurm hetgroup inside this job. The allocation +# wrapper brings up every replica plus one load balancer per pool, waits for +# health, then substitutes the resolved URLs into the driver command. The job +# owns its judges, so the reward model lands in provenance and no run can +# outlive, mismatch, or be starved by a pool it does not control. +# +# EXTERNAL_JUDGES=0 (default) keeps GenRM on a separately managed warm pool +# reached through GENRM_BASE_URL, and leaves the NL2Bash judge in the Gym pool. +# Partition `batch` caps at 4 h, so a warm pool amortizes the 470 GB bf16 +# Qwen3-235B load across a whole chain of jobs. The in-job path instead pays +# that load once per job — about ten minutes before training starts, measured +# on the 6K run — in exchange for being self-contained. +# +# To stand up the out-of-band pool (copy the dir first — it holds .lb_pid_*, +# logs/ and a flock'd registry, so running someone else's in place collides): +# +# cp -r /lustre/fs1/portfolios/llmservice/projects/llmservice_modelalignment_ppo/\ +# users/geshen/mopd_nano_fast/genrm_serving /genrm_serving +# cd /genrm_serving +# MODEL=/lustre/fsw/portfolios/llmservice/users/ansubramania/models/qwen235b_principle_comparison_genrm_step1230 \ +# ACCOUNT=nemotron_sw_post PARTITION=batch_long TIME=1-12:00:00 \ +# LB_PORT=9213 GENRM_GROUP_ID=nano35_dolphin \ +# ./genrm_server_manager.sh launch N +# ./genrm_server_manager.sh url +# +# NOTE: that script's default MODEL is the *ultra* GenRM (step_720) — override it. +# Each worker is 2 nodes x 4 GPUs at TP=8, separate from this job's 64 nodes. +# +# The `model` field must equal the pool's --served-model-name ("model" in +# genrm_worker.sh). ultra_launch.sh sets base_url XOR model, never both, so the +# name is pinned in rlvr_dolphin.yaml instead of passed here. +# ----------------------------------------------------------------------------- +export EXTERNAL_JUDGES="${EXTERNAL_JUDGES:-0}" +_DEFAULT_GYM_NODES=16 +if [[ "${EXTERNAL_JUDGES}" == "1" ]]; then + # Both pools reproduce the control run's serving shape, not just its GPU count, + # so only the judges' location changes. GenRM is 2 replicas at TP=8 spanning + # two nodes each, exactly as genrm_worker.sh ran them in the warm pool, filling + # the same four nodes. NL2Bash is eight TP=4 replicas, equalling the TP4 x DP8 + # deployment it replaces in Gym. + # + # GenRM's TP is the one number worth not "simplifying" to a single node. The + # checkpoint is 438 GB, so it is resident once per replica: 4 x TP4 would hold + # four copies (1752 GB) on the same 16 GPUs where 2 x TP8 holds two (876 GB), + # and the difference comes straight out of KV cache -- roughly 1.1 TB against + # 2.0 TB pool-wide. Four replicas would buy more schedulers and more concurrent + # slots, but GenRM's long prompts are KV-bound, so the control's shape wins. + export GENRM_MODEL="${GENRM_MODEL:-/lustre/fsw/portfolios/llmservice/users/ansubramania/models/qwen235b_principle_comparison_genrm_step1230}" + export GENRM_REPLICAS="${GENRM_REPLICAS:-2}" + export GENRM_TENSOR_PARALLEL_SIZE="${GENRM_TENSOR_PARALLEL_SIZE:-8}" + # The warm pool serves this checkpoint through the ultra_v3 parser plugin, so + # carry both the plugin and its name over rather than vLLM's built-ins. + export GENRM_REASONING_PARSER="${GENRM_REASONING_PARSER:-/lustre/fsw/portfolios/llmservice/users/lvega/evals/ultra_v3_reasoning_parser.py}" + export GENRM_REASONING_PARSER_NAME="${GENRM_REASONING_PARSER_NAME:-ultra_v3}" + # The control's genrm_worker.sh passes --enable-expert-parallel; the 6K + # deployment of this checkpoint does not. Follow the control. + export GENRM_ENABLE_EXPERT_PARALLEL="${GENRM_ENABLE_EXPERT_PARALLEL:-1}" + export NL2BASH_REPLICAS="${NL2BASH_REPLICAS:-8}" + export NL2BASH_TENSOR_PARALLEL_SIZE="${NL2BASH_TENSOR_PARALLEL_SIZE:-4}" + export EXTERNAL_VLLM_SEGMENT_SIZE="${EXTERNAL_VLLM_SEGMENT_SIZE:-2}" + # Gym drops to 8 nodes because the NL2Bash judge vacates exactly the 8 it was + # filling: TP=4 puts one replica on each four-GPU node, and DP=8 means eight + # of them. Every Gym node that was not serving NL2Bash stays, so the safety + # judge and the CPU-side env servers are untouched. + # + # That keeps the GPU allocation comparable to the control run. Akash's + # baseline was a 64-node job (5931924: 8 train + 40 gen + 16 gym) alongside a + # 4-node GenRM pool of its own (5931683 and 5931688, 2 nodes each, spanning + # the full training window) -- 68 nodes, 272 GB200 GPUs. This path is + # 8 + 40 + 8 in hetgroup 0 plus 12 service nodes, which is the same 68. + _DEFAULT_GYM_NODES=8 + # + # CHECK THIS FIRST IF A POOL FAILS TO START. serve_vllm_on_ray.py imports + # nemo_rl before vLLM's serve CLI, so pools must run the RL venv; they cannot + # use the Gym venv that rlvr_dolphin.yaml deliberately gives the in-Gym + # NL2Bash judge. On a vLLM 0.25 container that RL venv pairs vLLM with the + # openai release uv.lock pins, and `vllm serve` dies importing NamespaceTool + # from openai.types.responses (job 5943331). Judges share nothing with the + # trainer, so the fix is to serve them from an image whose RL venv is + # self-consistent rather than to match CONTAINER. Both pools default to + # CONTAINER in ultra_launch.sh and are overridable independently: + # GENRM_CONTAINER= NL2BASH_CONTAINER= +else + : "${GENRM_BASE_URL:?GENRM_BASE_URL must point to the external GenRM /v1 endpoint (or set EXTERNAL_JUDGES=1 to host GenRM in-job)}" + export GENRM_BASE_URL + unset GENRM_MODEL +fi + +# ----------------------------------------------------------------------------- +# Experiment identity +# EXP_NAME drives the W&B run name, the singleton job name, and the checkpoint +# and log dirs — so changing it starts a *new* run rather than resuming. +# ----------------------------------------------------------------------------- +export EXP_NAME="${EXP_NAME:-akamehra-nano35-honest-dolphin-v10-iter6000-rlvr-async1-tp4_cp4_ep8_pp1_gpp16_pps128_gbs2048}" +export CONFIG_PATH="${CONFIG_PATH:-examples/nemo_gym/nemotron-3.5-nano/rlvr_dolphin.yaml}" + +# ----------------------------------------------------------------------------- +# Model and data +# ----------------------------------------------------------------------------- +# honest-dolphin SFT v10 (closethink unmask, from midtrain 100B LC), iter_0006000. +export MODEL_PATH="${MODEL_PATH:-/lustre/fsw/portfolios/llmservice/users/venkats/training_actual_0603/nano_n3_post/checkpoints/nano-3.5-sft-v10-closethink-unmask-orig6k-from-midtrain-100B-lc-lr2e-5/eval/iter_0006000/hf}" + +# trusty_viper: 199,680 prompts / 24 agent families, carrying agent_ref per row. +# VAL_PATH intentionally equals TRAIN_PATH, as in the reference — validation is +# effectively disabled (grpo.val_period is very large) because the genrm cohort +# envs are train-only and would hang under eval. +_BLEND="${_BLEND:-/lustre/fs1/portfolios/llmservice/projects/llmservice_modelalignment_ppo/users/geshen/rl-data-tools/blends/curriculum_honest_dolphin_v41_trusty_viper.train.jsonl}" +export TRAIN_PATH="${TRAIN_PATH:-${_BLEND}}" +export VAL_PATH="${VAL_PATH:-${_BLEND}}" + +# ----------------------------------------------------------------------------- +# Judge checkpoints. Where they run depends on EXTERNAL_JUDGES above: the safety +# judge is always served from the Gym pool, and the NL2Bash judge joins it there +# unless the in-job hetgroup is enabled, in which case Gym skips its local +# launch and proxies to the pool's load balancer instead. +# ----------------------------------------------------------------------------- +export NL2BASH_JUDGE_MODEL="${NL2BASH_JUDGE_MODEL:-/lustre/fsw/portfolios/llmservice/users/ansubramania/models/Qwen3-235B-A22B-Instruct-2507-FP8}" +export SAFETY_JUDGE_MODEL="${SAFETY_JUDGE_MODEL:-/lustre/fsw/portfolios/llmservice/users/ansubramania/super_v3/model_checkpoints/Nemotron-Content-Safety-Reasoning-4B}" + +# ----------------------------------------------------------------------------- +# Containers +# Built 2026-08-14 with prefetched venvs, on vllm 0.25.1. +# +# The image supplies Megatron, and that is why this pin has to track the repo. +# ultra_launch.sh overlays only nemo_rl/, examples/configs and Gym from the +# worktree, so megatron.core always comes from the container. `46ab18ce1 ci: bump +# Megatron-Bridge to 0c565c9a0 (#3568)` landed on main 2026-08-11 and made +# megatron_policy_worker.py import FullyShardedDataParallelV1, which no image +# built before that date has -- including the 2026-07-30 one that used to be the +# default here and the 2026-08-07 main build. Both fail at +# `from megatron.core.distributed.fsdp.mcore_fsdp_adapter import ...` while the +# Megatron workers come up, ~20 min in (job 6231494). To check a candidate before +# spending an allocation on it: +# +# unsquashfs -l | grep mcore_fsdp_adapter +# unsquashfs -d /tmp/probe -f \ +# opt/nemo-rl/3rdparty/Megatron-Bridge-workspace/Megatron-Bridge/3rdparty/\ +# Megatron-LM/megatron/core/distributed/fsdp/mcore_fsdp_adapter.py +# +# Keep vllm >= 0.25.1 when moving this pin: vllm_worker_async.py imports +# ServingTokenization from vllm.entrypoints.serve.tokenize.serving, added in +# 0.25.x, and a 0.20.0 image kills every generation worker at engine init ~25 min +# in, after the judges and the Megatron workers have already loaded (jobs 5942981 +# and 5981739). +# +# The image's vllm and its nvidia-cutlass-dsl also have to agree, which is why +# this is the 08-14 build and not the 08-15 one that first got us past the +# Megatron import. The 08-15 image carries vllm 0.25.1+precompiled, whose +# vllm/vllm_flash_attn/cute/utils.py adds +# `from cutlass._mlir_helpers.arith import recast_type`; the cutlass-dsl 4.5.2 +# it ships exposes that module only as cutlass.base_dsl._mlir_helpers, so the +# import raises ModuleNotFoundError. utils.py is imported lazily, from the FA4 +# branch of flash_attn_varlen_func, so nothing surfaces until the first real +# attention forward: engine init and the weight sync both succeed, then every +# TP worker dies on the first rollout wave, EngineCore goes down, every +# generation 500s and the infra budget aborts the run ~3 min into serving (jobs +# 6233682 and 6234556, which is also where the scheduler's req_id_to_index +# KeyError came from -- collateral from the workers dying mid-batch, not a +# cause). FA4 is the default on these GB200s, so there is no getting to it. +# Add to the candidate probe above: +# +# unsquashfs -d /tmp/probe -f \ +# opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.\ +# VllmAsyncGenerationWorker/lib/python3.13/site-packages/vllm/vllm_flash_attn/\ +# cute/utils.py +# # follow the symlink into /root/.cache/uv, then: +# grep -n 'from cutlass' # must not name _mlir_helpers +# +# The 07-30 image passes that check but fails the Megatron one; 08-14 is the +# only build in yifuw's directory that passes both. +# ----------------------------------------------------------------------------- +if [[ -z "${CONTAINER:-}" ]]; then + : "${LUSTRE:?Set LUSTRE or provide CONTAINER explicitly}" + export CONTAINER="${LUSTRE}/enroot-images/nvcr.io/nvidian/nemo-rl:nightly-gym.squashfs" +else + export CONTAINER +fi + +# ----------------------------------------------------------------------------- +# Sandbox process DISABLED — this is what killed jobs 5726250 and 5732681. +# +# ray.sub launches the nemo-skills sandbox on every node with +# `--kill-on-bad-exit=1` (ray.sub:967), unlike the Ray worker step which uses +# `--kill-on-bad-exit=0` (ray.sub:1016). So a single node failing during sandbox +# startup makes srun tear down all 64 sandbox tasks; ray.sub then sees its +# background sandbox srun die and exits. At 64 nodes that happened on 3 of 3 +# attempts, on different nodes each time (nvl72133-T01, nvl72141-T07, +# nvl72126-T05/T17) — node exclusion cannot fix it. +# +# Only three Gym servers use the sandbox: competitive_coding_challenges (not in +# our blend), math_formal_lean and ns_tools. Both of the latter are in +# config_paths and are LEFT THERE deliberately: they construct their sandbox +# client lazily (math_formal_lean/app.py:387 builds it, :444 uses it inside a +# request handler; ns_tools only holds sandbox_host/port as config), so they +# boot fine without a sandbox and only contact it if a request routes to them. +# The dolphin blend routes to neither, so the sandbox is never needed. +# +# ray.sub gates everything sandbox-related on SANDBOX_CONTAINER && SANDBOX_COMMAND +# both being non-empty (ray.sub:559) — the ports dir, the 64-instance ready wait, +# and the srun itself. The unmodified ultra launcher replaces an empty +# SANDBOX_COMMAND with its default, so keep SANDBOX_CONTAINER empty instead. +# +# Side benefit: no 16 GB sandbox image extracted on 64 nodes, so faster startup. +# To re-enable (e.g. if a future blend uses Lean4), explicitly set +# SANDBOX_CONTAINER to the sandbox image path. +# ----------------------------------------------------------------------------- +if [[ -z "${SANDBOX_CONTAINER:-}" ]]; then + : "${LUSTRE:?Set LUSTRE or provide SANDBOX_CONTAINER explicitly}" + export SANDBOX_CONTAINER="${LUSTRE}/enroot-images/nvcr.io/nvidian/nemo-rl:skills-sandbox-latest.squashfs" +else + export SANDBOX_CONTAINER +fi + +# ----------------------------------------------------------------------------- +# Caches +# PERSISTENT_CACHE must be set explicitly: ultra_launch.sh requires it, and the +# internal reference's derivation (/lustre/fsw/portfolios/${ACCOUNT%%_*}/users/$USER) +# would resolve to /lustre/fsw/portfolios/nemotron/... which is read-only for us. +# HF_HOME is a *sibling* of the cache, not inside it, because the launcher purges +# vllm_compile_cache* under PERSISTENT_CACHE on every submission. +# HF_HOME also decides where the HF->Megatron conversion of the 62 GB checkpoint +# lands (get_megatron_checkpoint_dir falls back to $HF_HOME/nemo_rl), so keeping +# it on Lustre means the conversion is done once, not once per job. +# ----------------------------------------------------------------------------- +export PERSISTENT_CACHE="${PERSISTENT_CACHE:-/lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_llm/users/akamehra/.cache/nano35-dolphin}" +export HF_HOME="${HF_HOME:-/lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_llm/users/akamehra/hf_home}" + +# ----------------------------------------------------------------------------- +# Container mounts — REQUIRED. +# ultra_launch.sh starts MOUNTS empty and only appends three source overlays +# (nemo_rl, examples/configs, Gym). It never mounts /lustre, so without this the +# container cannot see the checkpoint, the blend jsonl, the judge models, +# HF_HOME or PERSISTENT_CACHE. The internal reference hardcoded this mount. +# ----------------------------------------------------------------------------- +export MOUNTS="${MOUNTS:-/lustre:/lustre}" + +# ----------------------------------------------------------------------------- +# Do not write bytecode into the Lustre-mounted source. +# +# nemo_rl is bind-mounted from Lustre into every container. Without this, all 64 +# nodes write .pyc back into that shared tree (194 files appeared during earlier +# runs, including generation/__pycache__/interfaces.cpython-313.pyc). That is +# metadata churn on the exact directories every node is importing from. +# +# Job 5742619 died when Ray unpickled VllmAsyncGenerationWorker on one node: +# __init__.py executed from the mount, then its sibling interfaces.py was not +# found — a per-node directory-view inconsistency, not a missing file. Reads +# alone are far safer than reads plus concurrent writes. +# ----------------------------------------------------------------------------- +export PYTHONDONTWRITEBYTECODE="${PYTHONDONTWRITEBYTECODE:-1}" + +# ----------------------------------------------------------------------------- +# examples/nemo_gym mount — REQUIRED, and the reason job 5730369 died with +# FileNotFoundError: /opt/nemo-rl/examples/nemo_gym/nemotron-3.5-nano/rlvr_dolphin.yaml +# +# ultra_launch.sh overlays only nemo_rl, examples/configs and Gym. Everything +# else under /opt/nemo-rl — including examples/nemo_gym, where this recipe and +# the nemotron-3-ultra base config live — comes from the container image. +# Two things are therefore invisible without this mount: +# 1. rlvr_dolphin.yaml itself (written here, never in any image), and +# 2. nemotron-3-ultra/student_rlvr1.yaml, which it inherits — that landed on +# main in 64cb9f985 (Jul 29-30) but this image was built Jul 26, so the +# image genuinely does not contain it (verified by inspecting the image). +# Mounting the directory fixes both, and also removes a version skew: nemo_rl +# would otherwise come from this checkout while examples/nemo_gym came from a +# Jul-26 image. +# +# Scoped to this one directory rather than mounting the repo root over +# /opt/nemo-rl, to avoid shadowing anything the image builds in place. +# ----------------------------------------------------------------------------- +_NEMO_GYM_MOUNT="${REPO_ROOT}/examples/nemo_gym:/opt/nemo-rl/examples/nemo_gym" +if [[ -n "${EXTRA_MOUNTS:-}" ]]; then + export EXTRA_MOUNTS="${EXTRA_MOUNTS},${_NEMO_GYM_MOUNT}" +else + export EXTRA_MOUNTS="${_NEMO_GYM_MOUNT}" +fi + +# ----------------------------------------------------------------------------- +# Snapshotting is OFF because tools/code_snapshot.sh copies only *git-tracked* +# files, and this recipe is untracked (`?? examples/nemo_gym/nemotron-3.5-nano/`). +# With snapshotting on, the mount above would point into a snapshot that does +# not contain the config. To restore frozen provenance, `git add` the recipe and +# set USE_SNAPSHOT=1. +# ----------------------------------------------------------------------------- +export USE_SNAPSHOT="${USE_SNAPSHOT:-0}" + +# ----------------------------------------------------------------------------- +# Results root — MUST be absolute. +# ultra_launch.sh defaults RESULTS_DIR to the relative "results/${EXP_NAME}". +# The host mkdir would land it in the repo, but TRAIN_CMD does `cd /opt/nemo-rl` +# inside the container, so the same relative string resolves to +# /opt/nemo-rl/results/... — the container's ephemeral overlay. Checkpoints +# would vanish at job end and the singleton auto-resume would never find them, +# so on a 4 h wall the run would restart from the SFT checkpoint forever. +# An absolute Lustre path fixes checkpoints, logs, ray_logs and slurm output. +# ----------------------------------------------------------------------------- +export RESULTS_DIR="${RESULTS_DIR:-/lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_llm/users/akamehra/runs/${EXP_NAME}}" + +# ----------------------------------------------------------------------------- +# SLURM +# Job shape: 8 train + 40 gen + 16 gym = 64 GB200 nodes (4 GPUs each), a 5:1 +# generation-to-training split. GenRM runs in its own allocation (2 replicas x +# 2 nodes), so the campaign footprint is 68 nodes. +# SEGMENT_SIZE=2 is the nano value; ultra defaults to 16. +# Partition `batch` caps at 4 h (batch_long is 7 d), so WALLTIME is 4 h and +# CHECKPOINTING_SAVE_BY keeps the reference's 25-minute teardown margin. +# ----------------------------------------------------------------------------- +export SLURM_ACCOUNT="${SLURM_ACCOUNT:-nemotron_sw_post}" +export SLURM_PARTITION="${SLURM_PARTITION:-batch}" +export WALLTIME="${WALLTIME:-4:00:00}" +# Async rollout collection can leave the training GPUs idle while judges work, +# and startup is idle end to end: job 5965796 needed 56 minutes to reach its +# first optimizer step and the reaper took it at 60. This is the name +# ultra_launch.sh actually reads -- JOB_REAPER_EXEMPT_IDLE_MINS was never +# consumed by anything, so the banner reported a number with no effect. +export JOB_REAPER_EXEMPT_MINS="${JOB_REAPER_EXEMPT_MINS:-120}" +export CHECKPOINTING_SAVE_BY="${CHECKPOINTING_SAVE_BY:-00:03:35:00}" +export NUM_TRAIN_NODES="${NUM_TRAIN_NODES:-8}" +export NUM_GEN_NODES="${NUM_GEN_NODES:-40}" +export NUM_GYM_NODES="${NUM_GYM_NODES:-${_DEFAULT_GYM_NODES}}" +export SEGMENT_SIZE="${SEGMENT_SIZE:-2}" + +# ----------------------------------------------------------------------------- +# W&B. WANDB_API_KEY must already be in the environment — ultra_launch.sh needs +# it. If it is exported only from ~/.zshrc, submit from zsh; a bash context +# will not see it. +# ----------------------------------------------------------------------------- +export WANDB_PROJ="${WANDB_PROJ:-ultra-streaming}" +export WANDB_ENTITY="${WANDB_ENTITY:-joc}" + +# MTP: head *training* is on via the config (5 repeated layers, loss 0.3, +# detached heads), matching the reference. MTP *speculative decoding* for vLLM +# is a separate, independent switch and is off, also matching the reference. +export ENABLE_MTP_INFERENCE="${ENABLE_MTP_INFERENCE:-0}" + +echo "================================================================" +echo " Nemotron 3.5 Nano — RLVR async-1 (honest-dolphin)" +echo "================================================================" +echo " Experiment : ${EXP_NAME}" +echo " Config : ${CONFIG_PATH}" +echo " Model : ${MODEL_PATH}" +echo " Blend : ${TRAIN_PATH}" +echo " Container : ${CONTAINER}" +echo " Cache : ${PERSISTENT_CACHE}" +echo " HF_HOME : ${HF_HOME}" +if [[ "${EXTERNAL_JUDGES}" == "1" ]]; then +echo " GenRM : in-job hetgroup — ${GENRM_REPLICAS} x TP=${GENRM_TENSOR_PARALLEL_SIZE}" +echo " ${GENRM_MODEL}" +echo " NL2Bash : in-job hetgroup — ${NL2BASH_REPLICAS} x TP=${NL2BASH_TENSOR_PARALLEL_SIZE}" +else +echo " GenRM : ${GENRM_BASE_URL} (external pool; served model name: model)" +echo " NL2Bash : served in the Gym pool" +fi +echo " SLURM : ${SLURM_ACCOUNT} / ${SLURM_PARTITION} / ${WALLTIME}" +echo " Reaper : ${JOB_REAPER_EXEMPT_MINS} min idle exemption" +echo " Nodes : ${NUM_TRAIN_NODES} train + ${NUM_GEN_NODES} gen + ${NUM_GYM_NODES} gym" +echo " W&B : ${WANDB_ENTITY}/${WANDB_PROJ}" +echo "================================================================" +echo "" + +exec bash examples/nemo_gym/nemotron-3-ultra/ultra_launch.sh "$@" diff --git a/examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch_sc.sh b/examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch_sc.sh new file mode 100755 index 00000000000..214f8309318 --- /dev/null +++ b/examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch_sc.sh @@ -0,0 +1,199 @@ +#!/bin/bash +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -euo pipefail + +# ============================================================================= +# nano35_dolphin_launch_sc.sh +# +# SingleController variant of nano35_dolphin_launch.sh: the same 64-node +# Nemotron 3.5 Nano RLVR pipeclean, driven by run_grpo_single_controller.py +# with streaming forward/backward, the TransferQueue data plane and +# shard-to-shard NCCL-reshard weight refit. +# +# Shape (GB200, 4 GPUs/node -> 256 GPUs), unchanged from the baseline so the +# two runs are comparable: +# 8 train + 40 generation + 16 gym = 64 nodes (5:1 generation-to-training) +# GenRM adds 4 nodes from its own allocation, so the campaign footprint is 68. +# +# This is a thin wrapper over nano35_dolphin_launch.sh, which already carries +# every site default (model, blend, judges, container, mounts, caches, Slurm). +# We only swap the config and the driver, so the two runs differ solely in the +# SC wiring. +# +# This branch supports durable trainer checkpoints plus periodic rollout/TQ +# snapshots. The defaults below exercise that production recovery path; set +# ROLLOUT_CHECKPOINT_INTERVAL_S=null to keep trainer checkpointing but disable +# periodic rollout snapshots for an ablation. +# +# Usage: +# GENRM_BASE_URL=http://:9213/v1 \ +# bash examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch_sc.sh +# +# DRY_RUN=1 GENRM_BASE_URL=... bash .../nano35_dolphin_launch_sc.sh +# +# GenRM must already be serving — see the GenRM section of the baseline script +# for how to stand up the external pool, or pass EXTERNAL_JUDGES=1 to host +# GenRM and the NL2Bash judge in a hetgroup inside this job instead, which is +# the shape the 6K recipe uses. +# +# Optional: +# NRL_MAX_STEPS=10 # short pipeclean +# STREAM_MIN_GROUPS=32 # async_rl.min_groups_for_streaming_train +# SAMPLER=in_order # in_order | weight_fifo | windowed +# MAX_LOOKAHEAD_VERSIONS=4 # the sampler's slack, whatever it spells it +# # 1 restores parity with the async-1 baseline +# BUFFER_RETENTION_MULTIPLIER=2 # max_buffered_rollouts only; gated samplers +# NUM_STORAGE_UNITS=16 # data_plane.num_storage_units +# REFIT_TRANSPORT=null # fall back to the full-tensor NCCL broadcast +# ROLLOUT_CHECKPOINT_INTERVAL_S=120 +# ROLLOUT_TELEMETRY_INTERVAL_S=30 +# +# Extra positional args are forwarded as Hydra overrides, after ours, so they win. +# ============================================================================= + +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd) + +export CONFIG_PATH="${CONFIG_PATH:-examples/nemo_gym/nemotron-3.5-nano/rlvr_dolphin_sc.yaml}" + +# The SC driver. This branch's Ultra launcher reads NRL_ENTRYPOINT. +# data_plane.enabled=true (set in the config) is mandatory for it. +export NRL_ENTRYPOINT="${NRL_ENTRYPOINT:-./examples/run_grpo_single_controller.py}" + +# Distinct from the baseline's EXP_NAME so this starts a new W&B run, run dir +# and singleton job name rather than colliding with the async-1 baseline. +export EXP_NAME="${EXP_NAME:-akamehra-nano35-honest-dolphin-v10-iter6000-rlvr-sc-tp4_cp4_ep8_pp1_gpp16_pps128_gbs2048}" + +# The SC knobs worth sweeping without editing the config. +# +# STREAM_MIN_GROUPS starts the optimizer step earlier on partial cohorts. +# NUM_STORAGE_UNITS is the untuned one at this data volume. It is passed as a +# Hydra override unconditionally below, so it beats the recipe and the value in +# rlvr_dolphin_sc.yaml is never what runs — keep the two in step. It shards a +# global pool rather than reserving per unit, so over-provisioning costs only a +# CPU actor each; the windowed sweep's 14336 peak rows are 1.4% of capacity. +# MAX_LOOKAHEAD_VERSIONS is the sampler's slack. in_order spells it +# max_lookahead_versions; weight_fifo/windowed spell it max_staleness_versions. +# The gated samplers also require max_buffered_rollouts >= prompts * (slack + 1). +STREAM_MIN_GROUPS="${STREAM_MIN_GROUPS:-32}" +NUM_STORAGE_UNITS="${NUM_STORAGE_UNITS:-16}" +MAX_LOOKAHEAD_VERSIONS="${MAX_LOOKAHEAD_VERSIONS:-4}" +ROLLOUT_CHECKPOINT_INTERVAL_S="${ROLLOUT_CHECKPOINT_INTERVAL_S:-300}" +ROLLOUT_TELEMETRY_INTERVAL_S="${ROLLOUT_TELEMETRY_INTERVAL_S:-60}" + +# Emit only fields belonging to the selected discriminated sampler config. +# Hydra's `+` form is required when switching away from the in_order block +# declared in rlvr_dolphin_sc.yaml. +SAMPLER="${SAMPLER:-in_order}" +case "${SAMPLER}" in + in_order) + _SAMPLER_OVERRIDES=( + "async_rl.sampler.name=in_order" + "async_rl.sampler.max_lookahead_versions=${MAX_LOOKAHEAD_VERSIONS}" + ) + ;; + weight_fifo) + _SAMPLER_OVERRIDES=( + "async_rl.sampler.name=weight_fifo" + "+async_rl.sampler.max_staleness_versions=${MAX_LOOKAHEAD_VERSIONS}" + ) + ;; + windowed) + _SAMPLER_OVERRIDES=( + "async_rl.sampler.name=windowed" + "+async_rl.sampler.max_staleness_versions=${MAX_LOOKAHEAD_VERSIONS}" + ) + ;; + *) + echo "SAMPLER must be in_order, weight_fifo or windowed, got '${SAMPLER}'" >&2 + exit 1 + ;; +esac + +# Shard-to-shard weight refit, on by default in this variant. It is still +# experimental, so keep the escape hatch one env var away: REFIT_TRANSPORT=null +# restores the full-tensor broadcast that rlvr_dolphin.yaml uses. +REFIT_TRANSPORT="${REFIT_TRANSPORT:-nccl_reshard}" + +_NUM_PROMPTS_PER_STEP="${_NUM_PROMPTS_PER_STEP:-128}" + +# Generation quota: the current cohort plus every lookahead cohort in flight at +# once. This is the number that must not move, because it is what the arms are +# compared on. +_MAX_INFLIGHT_PROMPTS=$(( _NUM_PROMPTS_PER_STEP * (MAX_LOOKAHEAD_VERSIONS + 1) )) + +# Retention headroom, as a multiple of that quota. +# +# _buffer_capacity is a per-group semaphore taken at dispatch and released on +# select, evict, or failure. Zero eviction deletes one of those three release +# paths, so groups that have finished generating but have not yet been trained +# on stay resident holding permits. At a multiplier of 1 they are holding +# permits out of the same pool that admission draws from, so the finished work +# crowds out new generation -- the fix for eviction creates a throughput +# problem one layer down. +# +# A multiplier above 1 gives retention its own headroom, which is what v1 does: +# late_arrival_slack=2 sizes its retention at P*lag*2 against a generation quota +# of P*lag. Retention strictly exceeding what admission can produce is the +# property that stops completed work from starving dispatch. +# +# This is NOT job 6014206 (768 buffered against 384 in flight, 1.8x slower). +# That arm ran WindowedSampler, which derives from BaseSampler and whose admit +# returns None immediately -- "dispatch is bounded by buffer capacity, not by +# version" -- so there the buffer was the only thing limiting dispatch and +# raising it raised dispatch. in_order and weight_fifo are gated samplers, so +# their dispatch windows remain bounded independently of buffer headroom. +BUFFER_RETENTION_MULTIPLIER="${BUFFER_RETENTION_MULTIPLIER:-1}" +_MAX_BUFFERED_ROLLOUTS=$(( _MAX_INFLIGHT_PROMPTS * BUFFER_RETENTION_MULTIPLIER )) + +if (( BUFFER_RETENTION_MULTIPLIER < 1 )); then + echo "BUFFER_RETENTION_MULTIPLIER must be >= 1, got ${BUFFER_RETENTION_MULTIPLIER}." >&2 + echo "Below 1 the buffer sits under the sampler's required floor and the train" >&2 + echo "pump waits for a batch the buffer is too small to ever hold." >&2 + exit 1 +fi + +if (( BUFFER_RETENTION_MULTIPLIER > 1 )) && [[ "${SAMPLER}" == "windowed" ]]; then + echo "BUFFER_RETENTION_MULTIPLIER=${BUFFER_RETENTION_MULTIPLIER} with SAMPLER=windowed is the 6014206 trap." >&2 + echo "WindowedSampler.admit returns None, so the buffer is its only dispatch" >&2 + echo "limit and raising it raises dispatch: that arm ran 1.8x slower. Only the" >&2 + echo "gated samplers (in_order and weight_fifo) can take a multiplier." >&2 + exit 1 +fi + +echo "================================================================" +echo " Nemotron 3.5 Nano — RLVR SingleController (honest-dolphin)" +echo "================================================================" +echo " Entrypoint : ${NRL_ENTRYPOINT}" +echo " Config : ${CONFIG_PATH}" +echo " Refit : ${REFIT_TRANSPORT}" +echo " Streaming : min ${STREAM_MIN_GROUPS} of ${_NUM_PROMPTS_PER_STEP} groups per dispatch" +echo " Sampler : ${SAMPLER} (slack ${MAX_LOOKAHEAD_VERSIONS})" +echo " Capacity : buffer ${_MAX_BUFFERED_ROLLOUTS} groups (x${BUFFER_RETENTION_MULTIPLIER}), ${_MAX_INFLIGHT_PROMPTS} in flight" +echo " TQ units : ${NUM_STORAGE_UNITS}" +echo " Rollout ckpt: interval=${ROLLOUT_CHECKPOINT_INTERVAL_S}s, telemetry=${ROLLOUT_TELEMETRY_INTERVAL_S}s" +echo "================================================================" +echo "" + +exec bash "${SCRIPT_DIR}/nano35_dolphin_launch.sh" \ + "async_rl.min_groups_for_streaming_train=${STREAM_MIN_GROUPS}" \ + "${_SAMPLER_OVERRIDES[@]}" \ + "async_rl.max_inflight_prompts=${_MAX_INFLIGHT_PROMPTS}" \ + "async_rl.max_buffered_rollouts=${_MAX_BUFFERED_ROLLOUTS}" \ + "data_plane.num_storage_units=${NUM_STORAGE_UNITS}" \ + "policy.generation.refit_transport=${REFIT_TRANSPORT}" \ + "rollout_checkpointing.interval_s=${ROLLOUT_CHECKPOINT_INTERVAL_S}" \ + "rollout_checkpointing.telemetry_interval_s=${ROLLOUT_TELEMETRY_INTERVAL_S}" \ + "$@" diff --git a/examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch_sc_smoke.sh b/examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch_sc_smoke.sh new file mode 100755 index 00000000000..8980df8760b --- /dev/null +++ b/examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch_sc_smoke.sh @@ -0,0 +1,118 @@ +#!/bin/bash +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# ============================================================================= +# Nemotron 3.5 Nano — 8-node SingleController pipeclean +# +# Scaled-down nano35_dolphin_launch_sc.sh for validating the stack end to end +# before spending a 68-node allocation: SingleController, the TransferQueue +# data plane, nccl_reshard refit with MTP-head gating, and the in-job judge +# hetgroup this recipe gained from the 6K side. +# +# Usage: +# bash examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch_sc_smoke.sh +# DRY_RUN=1 bash .../nano35_dolphin_launch_sc_smoke.sh # inspect only +# +# Training parallelism is deliberately NOT scaled. Four nodes is the floor that +# preserves it exactly: TP=4 x CP=4 x PP=1 fills 16 GPUs, and EP=8 still divides +# 16 at ETP=1, so every rank owns the same 16 of the 128 routed experts as the +# full-scale run. The only difference is DP, which drops from 2 to 1. +# Going to 2 training nodes would force CP below 4, and CP=4 is what makes the +# 73728-token context tractable, so the comparison would stop being meaningful. +# +# Sequence length is left at the production 73728 on purpose. Truncating it +# would risk failures in the blend rather than in the code under test, and +# generation length is bounded by EOS in practice, not by the cap. +# +# What gets scaled instead is the cohort, from 128 prompts to 8, and the step +# count. NRL_MAX_STEPS must stay >= 2: the first weight sync only happens +# between steps, and refit is the main thing this run exists to exercise. +# +# The default shape is 8 nodes total. Raise NUM_GEN_NODES / NUM_GYM_NODES to 2 +# each (10 nodes) if rollouts are throughput-bound or Gym's env servers are +# cramped on a single node. +# ============================================================================= + +set -euo pipefail + +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd) + +# 4 + 1 + 1 in hetgroup 0, plus 2 service nodes below = 8. Both counts have to +# stay divisible by their segment size, which is 2 on nano. +export NUM_TRAIN_NODES="${NUM_TRAIN_NODES:-4}" +export NUM_GEN_NODES="${NUM_GEN_NODES:-1}" +export NUM_GYM_NODES="${NUM_GYM_NODES:-1}" +export SEGMENT_SIZE="${SEGMENT_SIZE:-2}" + +# Judges in-job, which is the path that has never been run and the main reason +# this profile exists. One replica each at the production TP, so the servers +# are sharded exactly as they would be at full scale -- there are just fewer. +# This also frees Gym down to a single node, since only the safety judge +# (TP=1, DP=1) still runs there. +export EXTERNAL_JUDGES="${EXTERNAL_JUDGES:-1}" +export GENRM_REPLICAS="${GENRM_REPLICAS:-1}" +export GENRM_TENSOR_PARALLEL_SIZE="${GENRM_TENSOR_PARALLEL_SIZE:-4}" +export NL2BASH_REPLICAS="${NL2BASH_REPLICAS:-1}" +export NL2BASH_TENSOR_PARALLEL_SIZE="${NL2BASH_TENSOR_PARALLEL_SIZE:-4}" +export EXTERNAL_VLLM_SEGMENT_SIZE="${EXTERNAL_VLLM_SEGMENT_SIZE:-2}" + +# 8 prompts x 16 generations. Over DP=1 that is 128 sequences per rank against +# 1024 at full scale, so this is strictly lighter per GPU, not heavier. +NUM_PROMPTS_PER_STEP="${NUM_PROMPTS_PER_STEP:-8}" +TRAIN_GLOBAL_BATCH_SIZE="${TRAIN_GLOBAL_BATCH_SIZE:-$((NUM_PROMPTS_PER_STEP * 16))}" + +# Production lookahead, so the sampler and its capacity checks behave as they +# would at scale; the buffer floor scales with the cohort. +MAX_LOOKAHEAD_VERSIONS="${MAX_LOOKAHEAD_VERSIONS:-4}" +export MAX_LOOKAHEAD_VERSIONS +export _NUM_PROMPTS_PER_STEP="${NUM_PROMPTS_PER_STEP}" + +# Start the optimizer on a quarter cohort so streaming is actually exercised +# rather than trivially satisfied by the full cohort arriving at once. +export STREAM_MIN_GROUPS="${STREAM_MIN_GROUPS:-$((NUM_PROMPTS_PER_STEP / 4))}" +export NUM_STORAGE_UNITS="${NUM_STORAGE_UNITS:-2}" + +# The full-scale recipe inherits akamehra's results and cache directories from +# the reference run, and they are not group-writable. Point both at the +# submitter's own scratch so this profile runs as-is for whoever launches it. +_USER_SCRATCH="/lustre/fsw/portfolios/coreai/users/${USER}" +export RESULTS_DIR="${RESULTS_DIR:-${_USER_SCRATCH}/runs/nano35-sc-pipeclean-n8}" +export PERSISTENT_CACHE="${PERSISTENT_CACHE:-${_USER_SCRATCH}/.cache/nano35-dolphin}" + +export EXP_NAME="${EXP_NAME:-${USER}-nano35-sc-pipeclean-n8}" +export NRL_MAX_STEPS="${NRL_MAX_STEPS:-3}" +export WALLTIME="${WALLTIME:-2:00:00}" +export SLURM_QOS="${SLURM_QOS:-short}" +# Run the live worktree rather than a submission-time copy, so a fix can be +# retried without re-snapshotting between attempts. +export USE_SNAPSHOT="${USE_SNAPSHOT:-0}" + +if (( NRL_MAX_STEPS < 2 )); then + echo "ERROR: NRL_MAX_STEPS=${NRL_MAX_STEPS} never reaches a weight sync, which" >&2 + echo " leaves nccl_reshard refit -- the point of this run -- untested." >&2 + exit 1 +fi + +echo "Nemotron 3.5 Nano — SingleController pipeclean profile" +echo " hetgroup 0: train=${NUM_TRAIN_NODES}, generation=${NUM_GEN_NODES}, Gym=${NUM_GYM_NODES}" +echo " hetgroup 1: GenRM=${GENRM_REPLICAS}xTP${GENRM_TENSOR_PARALLEL_SIZE}, NL2Bash=${NL2BASH_REPLICAS}xTP${NL2BASH_TENSOR_PARALLEL_SIZE}" +echo " parallelism: TP4 x CP4 x EP8 x PP1 (ETP1) — unchanged, DP 2 -> 1" +echo " batch: prompts=${NUM_PROMPTS_PER_STEP}, generations=16, global=${TRAIN_GLOBAL_BATCH_SIZE}" +echo " steps: ${NRL_MAX_STEPS}" + +exec bash "${SCRIPT_DIR}/nano35_dolphin_launch_sc.sh" \ + "grpo.num_prompts_per_step=${NUM_PROMPTS_PER_STEP}" \ + "policy.train_global_batch_size=${TRAIN_GLOBAL_BATCH_SIZE}" \ + "$@" diff --git a/examples/nemo_gym/nemotron-3.5-nano/rlvr_dolphin.yaml b/examples/nemo_gym/nemotron-3.5-nano/rlvr_dolphin.yaml new file mode 100644 index 00000000000..b51704eb103 --- /dev/null +++ b/examples/nemo_gym/nemotron-3.5-nano/rlvr_dolphin.yaml @@ -0,0 +1,336 @@ +# ============================================================================= +# Nemotron 3.5 Nano — RLVR, legacy async-1, honest-dolphin warm start +# ============================================================================= +# Reproduces the internal reference run +# geshen-ultra-rl-nano-honest-dolphin-v10-iter6000-mopd-rlvr +# (launch_nano_honest_dolphin.sh + examples/configs/grpo_ultra_512n4g_bf16.yaml +# on nemo-rl-internal @ 97c55ee2, branch geshen/nano_training) +# on public NeMo-RL main. +# +# Inherits examples/nemo_gym/nemotron-3-ultra/student_rlvr1.yaml. "ultra" there +# names the RLVR *recipe lineage* — the 34-env Gym blend, GenRM comparison +# judging, judge topology, reward penalties, async-1 — not the model. The +# reference does the same thing: it runs the nano checkpoint against the ultra +# config and overrides parallelism. Everything ultra-shaped is overridden below. +# +# Model: NemotronH hybrid-Mamba MoE, 52 layers, hidden 2688, 128 routed experts +# top-6 + 1 shared, 32 heads / 2 KV, mamba n_groups=8, MTP 1 layer, vocab 131072. +# +# Node budget (GB200, 4 GPU/node), sbatch --nodes=64: +# 8 train + 40 generation -> cluster.num_nodes = 48 +# 16 gym -> env.nemo_gym.num_gpu_nodes = 16 +# +# A 5:1 generation-to-training split. GenRM is served from a SEPARATE +# allocation (2 replicas x 2 nodes = 4 nodes), so the campaign footprint is 68 +# nodes even though this job asks for 64. +# ============================================================================= +defaults: ../nemotron-3-ultra/student_rlvr1.yaml + +# ----------------------------------------------------------------------------- +# Cluster — ultra was 256 nodes / segment 16; nano uses 48 actor nodes / segment 2 +# ----------------------------------------------------------------------------- +cluster: + gpus_per_node: 4 + num_nodes: 48 + segment_size: 2 + +checkpointing: + # 4 h wall on partition `batch` (batch_long's 7 d is not what we're using), + # minus the reference's 25 min teardown margin. + checkpoint_must_save_by: "00:03:35:00" + # Reference used 10 against a 24 h wall. At 4 h a slow step can mean a job + # restarts having saved nothing and re-does the same work forever. Raise back + # toward 10 once step time is measured. + save_period: 5 + save_optimizer: true + +grpo: + # Stability setting: cut the prompt groups while retaining 16 generations per + # prompt. This yields 2048 samples/step and an async replay-buffer capacity + # of 256 groups. + num_prompts_per_step: 128 + # Validation is effectively off in the reference; genrm cohort envs are + # train-only and would hang under eval. + val_period: 10000000 + # REQUIRED by GRPOConfig, and absent from the inherited student_rlvr1.yaml — + # upstream inconsistency: 83753ed56 (Jul 29-30) added val_start_at as a required + # field while 64cb9f985 added the ultra recipes without it, so student_rlvr1.yaml + # on main cannot be validated by MasterConfig. Killed job 5733424 with + # "ValidationError: grpo.val_start_at Field required". + # -1 disables the delay, matching grpo_nanov3.yaml and grpo_math_1B.yaml. + val_start_at: -1 + +# ----------------------------------------------------------------------------- +# Policy — nano parallelism (reference: TP=4 CP=4 EP=16 PP=1 ETP=1) +# ----------------------------------------------------------------------------- +policy: + model_name: /lustre/fsw/portfolios/llmservice/users/venkats/training_actual_0603/nano_n3_post/checkpoints/nano-3.5-sft-v10-closethink-unmask-orig6k-from-midtrain-100B-lc-lr2e-5/eval/iter_0006000/hf + + # Must equal num_prompts_per_step * num_generations_per_prompt: 128 * 16. + # Over DP=2 (32 training GPUs / TP4 x CP4 x PP1) this is 1024 sequences per + # data-parallel rank — the same per-rank load as the 16-node / GBS 4096 shape + # this replaces, so per-GPU memory pressure is unchanged. + train_global_batch_size: 2048 + + # Ultra base is 49152. Reference nano runs 73728. + max_total_sequence_length: 73728 + + megatron_cfg: + tensor_model_parallel_size: 4 + expert_tensor_parallel_size: 1 + # 128 routed experts / EP8 = 16 experts per rank. Matches the control run + # (job 5931924) so the judge relocation is the only variable between them. + expert_model_parallel_size: 8 + pipeline_model_parallel_size: 1 + context_parallel_size: 4 + moe_token_dispatcher_type: "alltoall" + moe_flex_dispatcher_backend: "alltoall" + # MTP head training is inherited unchanged from the base and matches the + # reference exactly: mtp_num_layers 5, loss_scaling 0.3, use_repeated_layer + # true, detach_heads true. detach_heads keeps the MTP loss off the backbone, + # so it cannot perturb GRPO. use_repeated_layer reconciles 5 with the + # checkpoint's num_nextn_predict_layers=1 (one layer's weights reused 5x). + + generation: + # Base hardcodes 49152; re-tie to the policy value. + max_new_tokens: ${policy.max_total_sequence_length} + + vllm_cfg: + tensor_parallel_size: 4 + # Reference runs EP=1 for nano. vLLM's EP = DP * TP, and EP > TP is + # blocked upstream (NVIDIA-NeMo/RL#1101). + expert_parallel_size: 1 + max_model_len: ${policy.max_total_sequence_length} + # 0.85 -> 0.80. Job 5744748 died in weight refit: + # update_weights_from_collective: CUDA out of memory. + # Tried to allocate 3.69 GiB. GPU has 184.31 GiB total, 1.61 GiB free; + # this process holds 163.22 GiB. + # The OOM killed that vLLM rank, which was a participant in the NCCL + # collective, so MegatronPolicyWorker rank=0/22 aborted with it. + # 0.80 frees ~9 GiB per GPU against a 3.69 GiB staging buffer. + # The 40-node smoke run refit fine at 0.85 because it had 32 generation + # ranks; the buffer grows with that count. This shape has 160, more than + # the 128 that OOM'd, so 0.80 is a floor to watch rather than a settled + # value. Costs some KV cache, so slightly lower generation throughput. + gpu_memory_utilization: 0.80 + # nano_v3 reasoning parser + qwen3_coder tool parser are inherited and are + # correct for this checkpoint: its chat template emits and the + # XML form. + + vllm_kwargs: + # Hybrid Mamba allocates one state slot per running sequence. Base uses + # 256 (sized for ultra); the reference nano run uses 64 at 73728 tokens. + max_num_seqs: 64 + # Required on vLLM 0.25 and NOT accepted by 0.20 -- this branch only runs + # against a 0.25 container. Left at the default "auto", 0.25 picks the + # FlashInfer TRT-LLM BF16 MoE, whose expert weights are 3D grouped + # tensors under the new routed_experts submodule. Refit then hands them to + # a loader that rejects the expert dimension: job 5944595 died in + # update_weights_from_collective with "shard_dim=0 is not a valid data + # dimension for a 3D tensor (expected 1 or 2)". flashinfer_cutlass keeps + # the layout refit can address and is the backend the 64-node Ultra smoke + # run (job 5939591) refit through step 4 with on this same container. + moe_backend: flashinfer_cutlass + + colocated: + enabled: false + resources: + gpus_per_node: 4 + # Overridden by the launch script via NUM_GEN_NODES. + num_nodes: 40 + +# ----------------------------------------------------------------------------- +# Data — the launcher overrides these; defaults point at the reference blend. +# trusty_viper carries agent_ref per row, so NeMo-Gym routes each row to its +# family directly (no BlendDispatchAgent — that is a nemogym2mrl construct). +# ----------------------------------------------------------------------------- +data: + train: + data_path: /lustre/fs1/portfolios/llmservice/projects/llmservice_modelalignment_ppo/users/geshen/rl-data-tools/blends/curriculum_honest_dolphin_v41_trusty_viper.train.jsonl + validation: + data_path: /lustre/fs1/portfolios/llmservice/projects/llmservice_modelalignment_ppo/users/geshen/rl-data-tools/blends/curriculum_honest_dolphin_v41_trusty_viper.train.jsonl + +env: + nemo_gym: + num_gpu_nodes: 16 + + # ------------------------------------------------------------------------- + # Env blend. Identical to the base except rdkit_chemistry, which the base + # lists but the pinned Gym submodule (473f446) does not ship — it would fail + # at Gym startup. Unused by this blend, so dropped. + # ------------------------------------------------------------------------- + config_paths: + - responses_api_models/vllm_model/configs/vllm_model_for_training.yaml + - resources_servers/math_with_judge/configs/math_with_judge.yaml + - resources_servers/code_gen/configs/code_gen.yaml + - resources_servers/workplace_assistant/configs/workplace_assistant.yaml + - resources_servers/mcqa/configs/mcqa.yaml + - resources_servers/instruction_following/configs/instruction_following.yaml + - resources_servers/equivalence_llm_judge/configs/lc_judge.yaml + - resources_servers/calendar/configs/calendar.yaml + - resources_servers/genrm_compare/configs/genrm_compare.yaml + - resources_servers/equivalence_llm_judge/configs/nl2bash-equivalency.yaml + - resources_servers/equivalence_llm_judge/configs/equivalence_llm_judge.yaml + - resources_servers/single_step_tool_use_with_argument_comparison/configs/single_step_tool_use_with_argument_comparison.yaml + - resources_servers/reasoning_gym/configs/reasoning_gym.yaml + - resources_servers/terminus_judge/configs/terminus_judge_string_only.yaml + - resources_servers/ns_tools/configs/ns_tools.yaml + - resources_servers/math_formal_lean/configs/math_formal_lean_multi_turn.yaml + - resources_servers/multichallenge/configs/multichallenge.yaml + - resources_servers/inverse_if/configs/inverse_if.yaml + - resources_servers/single_step_tool_use_with_argument_comparison/configs/search_pivot_single_step_tool_use_with_argument_comparison.yaml + - resources_servers/single_step_tool_use_with_argument_comparison/configs/toolcall_schema_single_step_tool_use_with_argument_comparison.yaml + - resources_servers/single_step_tool_use_with_argument_comparison/configs/swe_pivot_single_step_tool_use_with_argument_comparison.yaml + - resources_servers/abstention/configs/abstention.yaml + - resources_servers/nvarc/configs/inductive.yaml + - resources_servers/nvarc/configs/transductive.yaml + - resources_servers/single_step_tool_use_with_argument_comparison/configs/droid_pivot_single_step_tool_use_with_argument_comparison.yaml + - resources_servers/equivalence_rule/configs/lc.yaml + - resources_servers/ether0/configs/ether0.yaml + - resources_servers/structured_outputs/configs/structured_outputs_json_yaml_xml_v1.yaml + - resources_servers/structured_outputs/configs/structured_outputs_v3.yaml + - resources_servers/format_verification/configs/freeform_formatting.yaml + - resources_servers/format_verification/configs/citation_format.yaml + - resources_servers/jailbreak_detection/configs/jailbreak_detection_nemotron_combined_reward_tp8.yaml + - resources_servers/indirect_prompt_injection/configs/indirect_prompt_injection.yaml + + # ------------------------------------------------------------------------- + # GenRM — EXTERNAL. The launcher sets base_url to the compute-side load + # balancer. `model` must match the external vLLM --served-model-name. + # Local vLLM settings below are inert whenever base_url is set. + # ------------------------------------------------------------------------- + genrm_model: + _override_: true + responses_api_models: + genrm_model: + entrypoint: app.py + api_key: dummy_key + base_url: null # set by the launcher via GENRM_BASE_URL + model: "model" + uses_reasoning_parser: true + return_token_id_information: false + debug: true + ray_worker_py_executable: /opt/ray_venvs/nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker/bin/python + # Aligned to nl2bash_judge_model where their roles overlap, except for + # GenRM-specific data parallelism, parsing, batching, and load format. + # Job 5746440's external GenRM predecessor died with a Triton illegal + # memory access; that incident is separate from job 5755734's internal + # nl2bash collective stall. + vllm_serve_env_vars: + VLLM_RAY_DP_PACK_STRATEGY: strict + NCCL_MNNVL_ENABLE: "1" + # Disable the blocked-GEMM MoE kernels. Job 5743685 died here: + # AssertionError: K must be divisible by blockK + # Error executing method 'load_model' + # genrm_model was the only vLLM service with none of these set, and + # it inherits VLLM_USE_FLASHINFER_MOE_FP8=1 from the driver's + # TRAIN_CMD. nl2bash is the control: same Qwen3-235B-A22B family, + # same TP4, loads fine with all four disabled. safety disables + # DeepGEMM too. This makes GenRM match them. + VLLM_USE_DEEP_GEMM: "0" + VLLM_MOE_USE_DEEP_GEMM: "0" + VLLM_USE_FLASHINFER_MOE_FP8: "0" + VLLM_USE_FLASHINFER_MOE_FP16: "0" + vllm_serve_kwargs: + attention_backend: FLASH_ATTN # was TRITON_ATTN; matches nl2bash + tensor_parallel_size: 4 + data_parallel_size: 4 + data_parallel_size_local: 1 + pipeline_parallel_size: 1 + enable_expert_parallel: true # matches nl2bash on the same MoE family + reasoning_parser: deepseek_r1 # GenRM-specific, nl2bash has none + gpu_memory_utilization: 0.85 + max_model_len: 131072 + max_num_seqs: 256 + max_num_batched_tokens: 8192 + enable_prefix_caching: true + enable_chunked_prefill: true + # Back to 112 (upstream default). The os error 108 in job 5744482 + # was traced to a single bad node, nvl72138-T01 (10.109.24.155), + # which also caused the ModuleNotFoundError in 5742619 and 5745204 — + # not loader-thread pressure. That node is now excluded at submit. + model_loader_extra_config: + enable_multithread_load: true + num_threads: 112 + load_format: auto + compilation_config: # matches nl2bash + cudagraph_capture_sizes: [1, 2, 4, 8, 16, 32] + + genrm_compare_resources_server: + resources_servers: + genrm_compare: + # Base ships 0.1 / 0.25 (retuned for ultra). Reference dolphin used + # 0.12 / 0.12 — restored, since we are reproducing that run. + group_reasoning_length_penalty_coeff: 0.12 + group_answer_length_penalty_coeff: 0.12 + + # Qwen3-235B-A22B-Instruct-2507-FP8. TP4 x DP8 = 32 GPUs = 8 of the 16 gym + # nodes. Base ships DP4 (sized for 256n ultra); reference uses DP8. + # Backs equivalence_llm_judge, lc_judge, math_with_judge, multichallenge, + # inverse_if and abstention — so this sizing gates six families. + nl2bash_judge_model: + responses_api_models: + local_vllm_model: + model: /lustre/fsw/portfolios/llmservice/users/ansubramania/models/Qwen3-235B-A22B-Instruct-2507-FP8 + # Launch this judge from the Gym venv, not the RL one. On a 0.25 + # container the RL venv pairs vLLM 0.25 with the openai version + # uv.lock pins, and `vllm serve` then fails to import NamespaceTool + # from openai.types.responses (job 5943331). The Gym venv is + # self-consistent. Judges need no weight sync, so they gain nothing + # from sharing the RL venv. + ray_worker_py_executable: /opt/gym_venvs/responses_api_models/local_vllm_model/.venv/bin/python + vllm_serve_env_vars: + # Matches the known-working internal dolphin/bear judge recipe. + NCCL_MNNVL_ENABLE: "0" + # Keep Ray's compiled-DAG wait aligned with vLLM's distributed wait. + RAY_CGRAPH_get_timeout: "2400" + # Off for every judge that runs from the Gym venv; see the note on + # safety_judge_model below. + VLLM_USE_FASTOKENS: "0" + vllm_serve_kwargs: + tensor_parallel_size: 4 + data_parallel_size: 8 + distributed_timeout_seconds: 2400 + # Back to 112 (upstream default); see genrm note above. + model_loader_extra_config: + enable_multithread_load: true + num_threads: 112 + + # Nemotron-Content-Safety-Reasoning-4B — 1 GPU, per the reference. + safety_judge_model: + responses_api_models: + local_vllm_model: + model: /lustre/fsw/portfolios/llmservice/users/ansubramania/super_v3/model_checkpoints/Nemotron-Content-Safety-Reasoning-4B + vllm_serve_env_vars: + # Some images bake VLLM_USE_FASTOKENS=1 into /etc/environment, so + # every process inherits it, but only the RL venv carries the wheel + # (fastokens 0.3.1). This judge has no ray_worker_py_executable, so + # it runs from the Gym venv, whose vLLM ships the integration module + # (vllm/tokenizers/fastokens.py) without the package behind it, and + # exits at import with "The 'fastokens' package (>= 0.2.0) is + # required when VLLM_USE_FASTOKENS=1" -- taking Gym spinup down with + # it (job 6232802, on the 08-15 build). The Jul-30 image's Gym vLLM + # had no fastokens support at all and ignored the flag, which is why + # this appeared only once the pin moved forward. Pinned off rather + # than left to the image so a future pin cannot reintroduce it. + # Merged onto the base's env vars rather than replacing them. + VLLM_USE_FASTOKENS: "0" + vllm_serve_kwargs: + tensor_parallel_size: 1 + data_parallel_size: 1 + +# ----------------------------------------------------------------------------- +# Logger — log_dir / wandb.name are set by the launcher. +# ----------------------------------------------------------------------------- +logger: + wandb_enabled: true + tensorboard_enabled: true + monitor_gpus: true + wandb: + project: "ultra-streaming" + +# reward_penalties inherited unchanged and already match the reference: +# penalize_duplicated_reasoning / empty_final_answer / unwanted_tokens / +# malformed_think_tag all true, token_ids {unwanted: [2], think_open: 12, +# think_close: 13}. This is the upstreamed form of the internal config's +# top-level penalize_* + token_ids block. diff --git a/examples/nemo_gym/nemotron-3.5-nano/rlvr_dolphin_sc.yaml b/examples/nemo_gym/nemotron-3.5-nano/rlvr_dolphin_sc.yaml new file mode 100644 index 00000000000..b3d203dd62e --- /dev/null +++ b/examples/nemo_gym/nemotron-3.5-nano/rlvr_dolphin_sc.yaml @@ -0,0 +1,341 @@ +# ============================================================================= +# Nemotron 3.5 Nano — RLVR on the SingleController path +# ============================================================================= +# Overlay on rlvr_dolphin.yaml that swaps the legacy async-GRPO driver for the +# SingleController (SC) architecture: streaming forward/backward, the +# TransferQueue data plane, gated in-order rollout sampling, and shard-to-shard +# NCCL-reshard weight refit. +# +# Everything about the model, parallelism, batch shape, sequence length, node +# split and the Gym blend is inherited unchanged, so a run of this config is +# directly comparable against a rlvr_dolphin.yaml run on the same 64 nodes +# (8 train + 40 generation + 16 gym, plus GenRM's own 4). +# +# Entrypoint: examples/run_grpo_single_controller.py (NOT run_grpo_nemo_gym.py). +# nano35_dolphin_launch_sc.sh sets that via NRL_ENTRYPOINT. +# +# Group vs sample units: async_rl counts PROMPT GROUPS, not generations. One +# optimizer cohort here is 2048 / 16 = 128 groups. +# ============================================================================= +defaults: rlvr_dolphin.yaml + +# ============================================================================= +# Checkpointing — on, because a 100-step run outlives the 4 h wall +# ============================================================================= +# The SC path saves and resumes now: _save_checkpoint writes policy weights, the +# optimizer, the tokenizer, the dataloader state, the replay buffer and the +# step/epoch/consumed-sample counters, and setup_single_controller resumes from +# get_latest_checkpoint_path. Restoring the buffer matters more here than the +# weights do -- a resumed run comes back with its lag-4 staleness distribution +# instead of ramping from zero, so the convergence curve has no seam at the job +# boundary. The buffer restore is skipped if the sampler name changed, so every +# job in a chain must keep the same async_rl.sampler.name. +# +# grpo.max_num_steps is compared against the *restored* step count, so it is a +# whole-campaign target: NRL_MAX_STEPS=100 once, and each requeued job continues +# toward it rather than running 100 more steps. +# +# Inherited checkpoint settings are overridden for SC recovery: +# +# metric_name student_rlvr1.yaml sets "val:total_reward/mean", which +# validate_single_controller_config rejects outright -- SC has +# no validation loop, so a "val:" metric is never collected and +# top-k retention would silently degrade to a no-op. null here +# makes retention purely recency-based, which is what we want. +# save periods Full trainer state is saved every step. Periodic rollout +# snapshots after bootstrap require a durable checkpoint at +# the matching trainer step, so a wider period would leave +# most of a short pipeclean without an eligible anchor. +# keep_top_k Inherited as 1000000, i.e. keep everything. With no metric, +# "best" is recency, so this keeps the newest 3 periodic +# checkpoints and lets the rest go. +# +# checkpoint_must_save_by comes from the launcher (CHECKPOINTING_SAVE_BY, +# 00:03:35:00) and remains a wall-clock backstop. +# ============================================================================= +checkpointing: + enabled: true + metric_name: null + save_period: 1 + ft_save_period: 1 + ft_keep_latest_k: 1 + keep_top_k: 3 + save_optimizer: true + +grpo: + # setup_single_controller rejects val_period > 0 outright, so the baseline's + # "effectively disabled" 10000000 is not enough — validation has to be off. + # val_start_at: -1, val_at_start: false and val_at_end: false are inherited + # and already satisfy the rest of the guard. + val_period: 0 + + # Required because loss_fn.reference_policy_kl_penalty is 0.0 (inherited from + # student_rlvr1.yaml). No reference model is built, but the SC train pump + # still computes reference logprobs unless this is set, and then fails with + # AttributeError: 'MegatronPolicyWorker' has no 'reference_state_dict'. + skip_reference_policy_logprobs_calculation: true + + # run_grpo_single_controller.py raises unless this is null; the SC knobs live + # under async_rl. student_rlvr1.yaml interpolates + # ${add:${grpo.async_grpo.max_trajectory_age_steps}, 1} for num_groups_nemo_rl + # in two places, so both are pinned to the value that interpolation resolved + # to (max_trajectory_age_steps: 1) under env.nemo_gym below. Nothing may + # reference this block once it is null or config load fails. + async_grpo: null + +policy: + # --------------------------------------------------------------------------- + # Speculative-decoding draft head. run_grpo_single_controller.py reads + # config.policy["draft"]["enabled"] unconditionally, so omitting this block is + # a KeyError before Ray starts. The nemo_gym runner guards it with an `in` + # check, which is why neither rlvr_dolphin.yaml nor student_rlvr1.yaml needs it. + # + # Unrelated to the MTP head this recipe trains: MTP head training is on via + # megatron_cfg (5 repeated layers, loss 0.3, detached), and MTP speculative + # decoding in vLLM is off (ENABLE_MTP_INFERENCE=0), both inherited. + # --------------------------------------------------------------------------- + draft: + enabled: false + model_name: null + loss_weight: 0.1 + num_layers: null + aux_layer_indices: null + + generation: + # ------------------------------------------------------------------------- + # Pinned, not interpolated. rlvr_dolphin.yaml ties max_new_tokens to + # ${policy.max_total_sequence_length}, which also feeds vllm_cfg.max_model_len + # — so the generation budget and the context window are the same number and + # a prompt at exactly that length can never emit a token. It is rejected + # deterministically instead, and because Gym prepends system prompts and tool + # schemas at rollout time and rollouts are multi-turn, no first-turn filter + # bounds it: that pairing rejected 2108 prompts in one 8-node ultra probe arm + # and 1344 in another at a 16384 window. + # + # 73728 is the value the interpolation already resolves to here, kept + # identical on purpose so these arms stay comparable with the v1 and windowed + # arms they are measured against. The pin buys decoupling, not headroom: what + # actually keeps rejections at zero on this recipe is that the window is 4.5x + # the ultra one and no row of the dolphin blend approaches it (job 5995586 + # logged no context rejections at this window). Anyone lowering + # max_total_sequence_length here must now choose a smaller max_new_tokens + # deliberately, which is the step the ultra fan-in config skipped. + # ------------------------------------------------------------------------- + max_new_tokens: 73728 + + # ------------------------------------------------------------------------- + # Weight refit transport. The inherited default (null) broadcasts every full + # parameter tensor from the 32 training ranks to all 160 generation ranks. + # nccl_reshard instead moves each parameter shard-to-shard between the two + # parallelism layouts over NCCL's M2N reshard, so no rank ever materializes + # a full tensor. The bulk path covers the MoE FFN projections, which are + # 97-98% of the weights at EP=16. + # + # Declared here rather than passed with `+` because refit_transport is + # NotRequired in VllmConfig; with the key present, REFIT_TRANSPORT=null in + # nano35_dolphin_launch_sc.sh falls back to the broadcast path. + # + # This config satisfies every config-checkable constraint in + # check_nccl_reshard_refit_support: non-colocated, Megatron train, vLLM gen, + # ETP=1, PP=1 with no custom layout, BF16 end to end, no EPLB, and vLLM + # expert_parallel_size=1 (allowed alongside any TP; the other legal value is + # EP == TP). + # + # STILL WORTH CHECKING ON THE FIRST RUN: if the container lacks the nccl4py + # M2N integration, xferdtensor.py logs "nccl.m2n.reshard not found" and + # silently falls back to a Python implementation. Grep the logs for it. + # + # The MTP-head risk flagged here before is now handled: this checkpoint's HF + # export does emit MTP expert projections (mtp.layers.N.mixer.experts.*), and + # they did take the bulk path and fail (job 5943756, "layer_prefix mismatch: + # mtp != backbone"). is_nccl_reshard_param now routes anything under mtp. to + # the misc path, matching what the broadcast baseline does with them. + # REFIT_TRANSPORT=null remains the escape hatch. + # ------------------------------------------------------------------------- + refit_transport: nccl_reshard + + # vllm_cfg.gpu_memory_utilization stays at the inherited 0.80. That was + # lowered from 0.85 for the broadcast path's staging buffer (job 5744748 + # OOM'd in update_weights_from_collective at 128 generation ranks, and this + # shape has 160) — the exact pressure nccl_reshard removes. 0.85 may be + # recoverable here once a reshard run is confirmed healthy; left alone so + # the two recipes differ only in the transport. + + vllm_cfg: + # ----------------------------------------------------------------------- + # Samples vllm:num_requests_running / num_requests_waiting / + # kv_cache_usage_perc off each engine so a step's exposed_generation can + # be attributed. Every run so far shows generation as the binding + # constraint (0.18 groups/s produced against a 0.26 groups/s trainer + # appetite) without showing why: a compute-bound fleet and an + # under-subscribed one are indistinguishable from the trainer's side, and + # only the second gets faster when max_inflight_prompts rises. A waiting + # queue that is busy most of the step means the engines are the ceiling + # and deeper lag cannot help. + # + # Matches the nemotron-3-ultra recipes, which all enable this. Costs one + # Prometheus snapshot per engine every 0.5 s on a daemon thread. + # ----------------------------------------------------------------------- + enable_vllm_metrics_logger: true + vllm_metrics_logger_interval: 0.5 + +# ============================================================================= +# Loss — off-policy correction, required by every arm past lag 1 +# ============================================================================= +# student_rlvr1.yaml pins the importance ratio at 1, which makes +# use_importance_sampling_correction inert: the sampled tokens come from an +# older policy but the loss is computed as though they came from the current +# one. A lag sweep run that way measures the uncorrected loss, not the +# staleness, so the pin is lifted here for every sampler. This costs the +# prev_logprobs forward pass that the inherited pin was skipping. +# ============================================================================= +loss_fn: + force_on_policy_ratio: false + +# ============================================================================= +# SingleController async RL — required by MasterConfig +# ============================================================================= +async_rl: + # in_order is checkpoint-replay capable and is the gated sampler used by the + # partial-rollout recovery implementation on this branch. The launcher can + # switch to weight_fifo or windowed while emitting that sampler's correctly + # named slack field. + sampler: + name: in_order + max_lookahead_versions: 4 + + recompute_kv_cache_after_weight_updates: false + + # Begin forward/backward once this many groups are ready rather than waiting + # for all 128; the remainder streams in behind it and accumulates into the + # single GBS=2048 optimizer step. 32 is 25% of the cohort, the ratio the + # validated smaller-scale streaming runs used to get 2-3 chunks per step. + min_groups_for_streaming_train: 32 + + # Let the current cohort and all four lookahead cohorts generate + # concurrently: num_prompts_per_step * (max_staleness_versions + 1) = 128 * 5. + max_inflight_prompts: 640 + + # Hard floor, not a preference: validate_sampler_buffer_capacity raises if + # this is below num_prompts_per_step * (max_staleness_versions + 1) = 640, + # because the rollout pump would deadlock waiting for buffer slots. It scales + # with the slack, so raising one raises both. + # + # Keep it EQUAL to max_inflight_prompts, never above. The buffer permit is + # acquired before dispatch, so it is a dispatch semaphore rather than passive + # capacity: job 6014206 set 768 against 384 in flight and ran 1.8x slower. + max_buffered_rollouts: 640 + + # Rollout fault tolerance. Gym retries dropped connections itself, but hands + # us env-server 500s and truncated response bodies, and one of those used to + # abort the whole run: jobs 5966951 and 5967263 died that way after one and + # six steps while the legacy stack absorbed 400 of the same faults and kept + # training. + # + # Retrying matters more here than it does in v1. A dropped prompt leaves its + # batch a group short, and the in_order sampler only ever offers the trainer + # groups stamped for the current step, so that step can never close: job + # 5982927 wedged exactly this way after step 7 when two prompts hit 500s from + # a single sick env server, then sat idle for 104 minutes. Keeping the batch + # whole is worth waiting for; the trainer falls back to a shortened batch only + # once every attempt is gone. + # + # Keys follow the post-#3589 taxonomy: infra failures (dead shard, timeout, + # transport) re-dispatch onto a different shard, data failures do not. The two + # budgets are independent counters, not a total and a sub-total. + rollout_failure: + # 5 infra attempts with a 1s base gives 1s/2s/4s/8s, bounded at 30s. v1's + # AsyncTrajectoryCollector retries a comparable number of times, so neither + # stack absorbs more than the other. + max_infra_attempts_per_prompt: 5 + max_data_attempts_per_prompt: 2 + backoff_base_s: 1.0 + max_backoff_s: 30.0 + + # This branch has no replacement-reserve policy. Fail after exhausting the + # per-prompt budgets instead of silently accepting an in_order cohort hole. + max_skipped_prompts: 0 + + # Deadlines. This is the NeMo-Gym rollout path, so only the nemo_gym block + # applies; validate_single_controller_config rejects a populated native + # block here rather than letting it look effective. Nothing below SC bounds + # a rollout -- Gym retries its server-to-server hops in an uncapped loop + # against an aiohttp session with no ClientTimeout -- so without this a + # wedged env server holds its in_order slot until the wall clock. + nemo_gym: + rollout_timeout_s: 1800.0 + + # Must outlast every deadline above, which _check_watchdog_outlasts_rollouts + # enforces: below it, a rollout that is merely slow reads as a stall. + watchdog: + interval_s: 30.0 + stall_timeout_s: 2400.0 + stall_action: warn + gym_subprocess_check: true + +# ============================================================================= +# NeMo-Gym rollout actor +# ============================================================================= +env: + nemo_gym: + # Pinned literals for what student_rlvr1.yaml interpolated off + # grpo.async_grpo.max_trajectory_age_steps, which is null on this path. + policy_model: + responses_api_models: + vllm_model: + num_groups_nemo_rl: 2 + policy_model_reasoning_off: + responses_api_models: + vllm_model: + num_groups_nemo_rl: 2 + +# ============================================================================= +# TransferQueue data plane — mandatory for the SC path +# ============================================================================= +# run_grpo_single_controller.py refuses to start unless enabled is true, and +# DataPlaneConfig requires every key below with no Python-side defaults. +# global_segment_size / local_buffer_size are read only when backend is +# mooncake_cpu, but must still be present. +# ============================================================================= +data_plane: + enabled: true + impl: transfer_queue + backend: "simple" + storage_capacity: 1000000 + # Sharding only — storage_capacity above is global and TQ splits it, as + # ceil(total_storage_size / num_data_storage_units), so this trades unit count + # against per-unit size and reserves nothing extra. Raising it costs one CPU + # actor each and buys read/write fan-out. + # + # 16 is headroom, not a fix for anything measured. The windowed sweep sizes + # the buffer as num_prompts_per_step * (staleness + 1), so the arms hold 256 / + # 384 / 640 / 896 groups resident — up to 14336 rows at staleness 6, which is + # 40% past the 10240 the previous value of 8 was reasoned against. Peak + # resident is still only ~1.4% of the 1000000-row global capacity, so the + # concern is the actors' aggregate throughput rather than their space. Held + # identical across arms deliberately: varying it would confound the staleness + # comparison the sweep exists to make. + num_storage_units: 16 + claim_meta_poll_interval_s: 0.5 + # Mooncake-only: _init_tq reads these solely on the mooncake_cpu branch, and + # this recipe is backend "simple". Inert here, kept for backend switches. + global_segment_size: 549755813888 # 512 GiB + local_buffer_size: 68719476736 # 64 GiB + # Required for restoring replay rows and token-capture receipts alongside + # trainer state and for periodic rollout snapshots. + checkpointing_enabled: true + +# Gate-authoritative token capture gives every logical sibling a durable TQ +# receipt that can be reused after restart. +token_capture: + enabled: true + on_capture_failure: continue + mixed_weight_version_policy: allow + +# Periodic data-plane/ledger snapshots. The launcher exposes both intervals as +# environment variables and forwards them as Hydra overrides. +rollout_checkpointing: + interval_s: 120 + telemetry_interval_s: 30 + keep_latest_k: 2 + restore_mode: latest From cbadca28b5968ead21fe5c299a89651d830da2ff Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Thu, 20 Aug 2026 11:19:21 -0700 Subject: [PATCH 40/52] test(nemo-gym): batch wrapper for the multi-worker ledger smoke revalidation Mirrors failed job 6300221 (nano-35-rlvr-sc-tc-smoke) with num_workers left at the recipe's 16/4 to exercise the multi-worker capture path on the lineage ledger. pthombre paths, proven 0820 launch plumbing, shared uv cache (never /tmp). Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre --- nano35_ledger_smoke.sh | 89 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100644 nano35_ledger_smoke.sh diff --git a/nano35_ledger_smoke.sh b/nano35_ledger_smoke.sh new file mode 100644 index 00000000000..dd1d8137231 --- /dev/null +++ b/nano35_ledger_smoke.sh @@ -0,0 +1,89 @@ +#!/bin/bash +# ============================================================================= +# nano35_ledger_smoke.sh — BATCH rerun of the nano-35-rlvr-sc-tc-smoke shape +# (failed job 6300221) on the lineage-ledger token-capture code. +# +# Purpose: regression validation of the multi-worker capture path. Job 6300221 +# died at the first rollout batch because the old per-process gate registry +# (register-before-dispatch) split across policy_model num_workers=16 / +# policy_model_reasoning_off num_workers=4 — 409 Conflict on unregistered +# workers, then UnknownRolloutError from ingest_coords mid-flight. The ledger +# replaced that with a process-shared FileLineageStore, so the SAME shape with +# num_workers deliberately left at 16/4 must now pass. +# +# Deviations from the original run, both deliberate: +# - container: zhiyul nightly-gym.2026-08-10 (the bake this branch's launch +# fixes were validated against), not amahishi's nightly-gym squashfs. +# - paths/W&B/secrets: pthombre-owned. +# Everything else (node shape, judges, batch geometry, walltime, QOS, model, +# blend, num_workers defaults) mirrors job 6300221. +# +# Run from a NETWORKED shell at the repo root (fsw twin path): +# DRY_RUN=1 bash nano35_ledger_smoke.sh # inspect only +# DRY_RUN=0 bash nano35_ledger_smoke.sh # submit +# ============================================================================= +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +_DRY_RUN_IN="${DRY_RUN:-}" + +set -a +# Secrets (WANDB_API_KEY, HF_TOKEN): untracked mode-600 file, never in git. +# shellcheck disable=SC1091 +source "${HERE}/swe_nano.secrets.env" + +# ---- container / shared read-only assets ------------------------------------ +CONTAINER=/lustre/fsw/portfolios/llmservice/users/zhiyul/enroot-images/nvcr.io+nvidian+nemo-rl+nightly-gym.2026-08-10.squashfs +SANDBOX_CONTAINER=/lustre/fsw/portfolios/coreai/users/cye/enroot/nemo-rl:skills-sandbox-latest.squashfs +# Model/blend/judge paths are baked into the dolphin launcher + recipe +# defaults (all verified readable); HF_HUB_CACHE keeps reading zhiyul's +# already-downloaded hub shards. +HF_HUB_CACHE=/lustre/fsw/portfolios/llmservice/users/zhiyul/hf_cache/hub + +# ---- per-user write paths ---------------------------------------------------- +WORKSPACE_DIR=/lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace +HF_HOME=/lustre/fsw/portfolios/llmservice/users/pthombre/hf_cache +PERSISTENT_CACHE=/lustre/fsw/portfolios/llmservice/users/pthombre/persistent_cache +NRL_MEGATRON_CHECKPOINT_DIR=${PERSISTENT_CACHE}/megatron_ckpt_cache + +EXP_NAME="${SC_EXP_NAME:-nano35-rlvr-sc-tc-smoke-ledger}" +RESULTS_DIR=${WORKSPACE_DIR}/results/${EXP_NAME} +BASE_LOG_DIR=${WORKSPACE_DIR}/ray_logs/${EXP_NAME} +WANDB_PROJ=nano-35-rlvr + +# ---- Slurm shape: exactly job 6300221 ---------------------------------------- +SLURM_PARTITION=batch +SLURM_ACCOUNT=nemotron_sw_post +SLURM_QOS=short +GPUS_PER_NODE=4 # GB200 NVL72 +WALLTIME=2:00:00 +# Smoke wrapper defaults reproduce the rest: 4 train + 1 gen + 1 gym nodes, +# EXTERNAL_JUDGES=1 (GenRM 1xTP4 + NL2Bash 1xTP4 hetgroup), 8 prompts x 16 +# generations, NRL_MAX_STEPS=3, STREAM_MIN_GROUPS=2, NUM_STORAGE_UNITS=2. +# num_workers stays at the recipe's 16/4 — the regression trigger under test. + +USE_SNAPSHOT=0 # live worktree, same as the original run + +# ---- launch plumbing proven by the 0820 ledger A/B smokes -------------------- +# (swe_nano.env / swe_nano_sc_capture.sh; see session/20260820_004547) +NRL_FORCE_REBUILD_VENVS=false +NRL_FORCE_REBUILD_VENVS_LIST="nemo_rl.environments.nemo_gym.NemoGym,nemo_rl.models.generation.vllm.vllm_worker_async.VllmAsyncGenerationWorker" +NRL_DRIVER_PYTHONPATH="/opt/nemo-rl/3rdparty/Gym-workspace/Gym" +NRL_DRIVER_PIP_INSTALL="orjson" +NRL_DRIVER_UV_RUN_FLAGS="--locked --no-sync" +NRL_TQ_SKIP_RUNTIME_ENV_PIN=1 +NRL_VENV_SYNC_FROZEN=1 +NRL_WG_USE_RAY_REF=1 +NRL_REFIT_ERRORS_FATAL=1 +# Shared prewarmed uv cache — NEVER /tmp; and never UV_CACHE_DIR_OVERRIDE with +# prefetch-venvs containers (it severs the baked venvs' hardlinks). +UV_CACHE_DIR=/lustre/fsw/portfolios/llmservice/users/pthombre/uv + +DRY_RUN="${_DRY_RUN_IN:-1}" +set +a + +# Same trailing override as the original invocation. rollout_checkpointing is +# an extra-allowed (ignored) block in this tree; kept for command fidelity. +exec bash "${HERE}/examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch_sc_smoke.sh" \ + rollout_checkpointing.restore_mode=none \ + "$@" From 02d700023c58b9018ac12f800bf39798c44bdcdf Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Thu, 20 Aug 2026 11:23:35 -0700 Subject: [PATCH 41/52] fix(launcher): honor NRL_ENTRYPOINT, driver deps and external UV_CACHE_DIR in ultra_launch Ports the recovery-refresh tree's driver plumbing the dolphin SC launchers depend on: NRL_ENTRYPOINT (SC driver instead of the hardcoded async one), NRL_DRIVER_PIP_INSTALL / NRL_DRIVER_PYTHONPATH (Gym import + orjson on the driver), NRL_DRIVER_UV_RUN_FLAGS (--locked --no-sync on prefetch-venv containers). Also lets an externally supplied UV_CACHE_DIR (shared Lustre cache) take precedence over the per-job /tmp cache. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre --- examples/nemo_gym/nemotron-3-ultra/ultra_launch.sh | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/examples/nemo_gym/nemotron-3-ultra/ultra_launch.sh b/examples/nemo_gym/nemotron-3-ultra/ultra_launch.sh index ed7e64268fd..24cce7205fc 100755 --- a/examples/nemo_gym/nemotron-3-ultra/ultra_launch.sh +++ b/examples/nemo_gym/nemotron-3-ultra/ultra_launch.sh @@ -855,7 +855,16 @@ export SETUP_COMMAND # learning rate, etc.) live in CONFIG_PATH. The launcher only passes the # per-run overrides: cluster shape, paths, judge endpoints, logging. # ============================================================================= +# A shared UV cache (e.g. prewarmed on Lustre) takes precedence; the /tmp +# fallback expands ${SLURM_JOB_ID} at runtime, hence the single quotes. +if [ -n "${UV_CACHE_DIR:-}" ]; then + _TRAIN_UV_CACHE_DIR="${UV_CACHE_DIR}" +else + _TRAIN_UV_CACHE_DIR='/tmp/nemo-gym-uv-cache-${SLURM_JOB_ID:-default}' +fi TRAIN_CMD="cd ${CODE_ROOT} && date ; \ +${NRL_DRIVER_PIP_INSTALL:+uv pip install --python /opt/nemo_rl_venv/bin/python ${NRL_DRIVER_PIP_INSTALL} ; }\ +${NRL_DRIVER_PYTHONPATH:+PYTHONPATH=${NRL_DRIVER_PYTHONPATH} }\ OMP_NUM_THREADS=16 \ RAY_DEDUP_LOGS=1 \ WANDB_INIT_TIMEOUT=300 \ @@ -864,7 +873,7 @@ NRL_VLLM_CACHE_SEED_DIR=${NRL_VLLM_CACHE_SEED_DIR} \ DG_JIT_CACHE_DIR=${NRL_VLLM_LOCAL_CACHE_DIR}/deep_gemm \ TORCHINDUCTOR_CACHE_DIR=${INDUCTOR_CACHE_DIR} \ TRITON_CACHE_DIR=${TRITON_CACHE_DIR} \ -UV_CACHE_DIR=/tmp/nemo-gym-uv-cache-\${SLURM_JOB_ID:-default} \ +UV_CACHE_DIR=${_TRAIN_UV_CACHE_DIR} \ UV_LOCK_TIMEOUT=1800 \ RAY_ENABLE_UV_RUN_RUNTIME_ENV=0 \ UV_HTTP_TIMEOUT=10 \ @@ -875,7 +884,7 @@ NRL_WG_USE_RAY_REF=1 \ HF_HOME=${HF_HOME:-} \ HF_TOKEN=${HF_TOKEN:-} \ NRL_USE_FASTOKENS=${NRL_USE_FASTOKENS:-1} \ -uv run ./examples/nemo_gym/run_grpo_nemo_gym.py \ +uv run ${NRL_DRIVER_UV_RUN_FLAGS:-} ${NRL_ENTRYPOINT:-./examples/nemo_gym/run_grpo_nemo_gym.py} \ --config ${CONFIG_PATH} \ policy.model_name=${MODEL_PATH} \ cluster.num_nodes=${NUM_ACTOR_NODES} \ From cf6d7913520bd0cfac9bffd134601f8316c6affd Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Thu, 20 Aug 2026 11:29:42 -0700 Subject: [PATCH 42/52] =?UTF-8?q?docs(report):=20r3=20rerun=20=E2=80=94=20?= =?UTF-8?q?carve-out=20recovers=20half=20the=20masked=20rows,=20residue=20?= =?UTF-8?q?analyzed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Signed-off-by: Prashant Thombre --- .../lineage-ledger-0820/experiments.tsv | 1 + .../lineage-ledger-0820/ledger-ab-report.md | 33 +++++++++++++++ session/20260820_004547/handoff.md | 22 +++++----- session/20260820_004547/timeline.md | 1 + session/20260820_111240/handoff.md | 16 ++++++++ session/20260820_111240/session_state.md | 41 +++++++++++++++++++ 6 files changed, 103 insertions(+), 11 deletions(-) create mode 100644 session/20260820_111240/handoff.md create mode 100644 session/20260820_111240/session_state.md diff --git a/reports/auto_research/lineage-ledger-0820/experiments.tsv b/reports/auto_research/lineage-ledger-0820/experiments.tsv index c1aac70f43b..ab7479f4d64 100644 --- a/reports/auto_research/lineage-ledger-0820/experiments.tsv +++ b/reports/auto_research/lineage-ledger-0820/experiments.tsv @@ -5,3 +5,4 @@ index branch parent_commit commit recipe metric_name metric_value elapsed_min la 4 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 (venv+legacy fix) nano_swe_teacher_sc smoke_5step_dynamics FAILED swe batch 6358268 (same as row 2) .../ledger-legacy-s43-0820 crash Pre-existing legacy bug: tool-call-only output item content=None -> TypeError in invalid-tool-call detector (also killed 0819 legacy runs). Fixed. 5 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 84feac48b nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 tmpe 1.0137-1.0147 gen_kl ~0.001 token_in_rate 0.95-0.99 59 swe batch 6359951 SC_EXP_NAME=ledger-capture-s43-0820-r2 ... (row 1 overrides, staging partition r3) .../ledger-capture-s43-0820-r2 keep Capture arm PASS: dynamics tighter than legacy; 16/40 rollouts fail-closed placeholders (per-call aborts; retry-idempotency follow-up); see ledger-ab-report.md 6 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 84feac48b nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 tmpe 1.020-1.066 gen_kl ~0.004 all-8-valid 67 swe batch 6359952 SC_EXP_NAME=ledger-legacy-s43-0820-r2 ... (row 2 overrides) .../ledger-legacy-s43-0820-r2 keep Legacy arm PASS: looser dynamics (echo re-tokenization); no rollout loss; see ledger-ab-report.md +7 autoresearch/2026-08-20-lineage-ledger/impl 5bc2b50d1 b39dbd8cd nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 valid 32/40 (was 24/40) tmpe 1.0137-1.0171 72 swe batch short-QOS 6365594 SC_EXP_NAME=ledger-capture-s43-0820-r3 WALLTIME=1:59:00 ... (row 5 overrides, staging partition r4) .../ledger-capture-s43-0820-r3 keep Carve-out validated: +8 valid rows; residue 9x missing_terminal_row (2 first-call deaths + ~7 harness-reported doomed-attempt terminal ids); dynamics unchanged diff --git a/reports/auto_research/lineage-ledger-0820/ledger-ab-report.md b/reports/auto_research/lineage-ledger-0820/ledger-ab-report.md index 68592ac2e02..11ba6ff9ace 100644 --- a/reports/auto_research/lineage-ledger-0820/ledger-ab-report.md +++ b/reports/auto_research/lineage-ledger-0820/ledger-ab-report.md @@ -94,3 +94,36 @@ these arms are not a rigorous throughput comparison; the prior campaign's drops under concurrent capture traffic. - CI (`/ok to test`) to run the Ray-heavy RL suites (blackbox_finalizer, tq_token_sink, vllm hosting) that cannot run outside the container. + +## Addendum: r3 rerun with the off-chain carve-out (job 6365594) + +`_assemble_receipt` now ignores `request_finished_without_staged_coordinates` +failure rows off the terminal chain (commit b39dbd8cd). Capture arm rerun, +same posture, short QOS, COMPLETED 5/5 in 1:11:49. + +| Metric | r2 (blanket poison) | r3 (carve-out) | +|---|---|---| +| valid rows / 40 | 24 (60%) | **32 (80%)** | +| rejection reason | 16× failure-row poison | 9× missing_terminal_row only | +| `token_mult_prob_error` | 1.0137–1.0147 | 1.0137–1.0171 | +| `gen_kl_error` | ~0.001 | ~0.001 | +| `finalize/token_in_rate` | 0.95–0.99 | 0.98–0.99 | +| ledger census | 2,821 rows / 16 uncommitted failures | 3,525 rows / 19 uncommitted failures | + +The carve-out worked exactly as designed: zero rollouts were rejected for +failure rows. The remaining 9 `missing_terminal_row` rejections split into: + +- 2 rollouts whose *first* call died (empty ledger — nothing to train; + correctly masked). +- ~7 overflow rollouts where the terminal id OpenHands reported does not + match any committed row: the harness derives it from the last llm_completion + file (`swe_agents/app.py:3101`), which for these episodes corresponds to the + doomed final attempt rather than the last successful completion. (In the + other ~9 overflow rollouts this run, the last file was the last successful + call and they trained — the carve-out's win.) + +Options for the residue (not taken here): (a) harness-side — report the last +*successful* response id when the final call errors; (b) receipt-side — fall +back to the deepest committed row when the terminal is missing and all +failures are uncommitted. (b) weakens the agent-kept-response attestation, so +(a) is the better follow-up alongside retry idempotency. diff --git a/session/20260820_004547/handoff.md b/session/20260820_004547/handoff.md index d3a53a68f15..6d06541a1f3 100644 --- a/session/20260820_004547/handoff.md +++ b/session/20260820_004547/handoff.md @@ -1,18 +1,18 @@ # Handoff ## Resume From Here -Implementing token-capture-lineage-ledger-approach.md. Gym side DONE and committed (Gym branch `token-capture-lineage-ledger`, commit 1e1a16cb; gate deleted, ledger implemented, 436 Gym tests green). RL side code changes DONE but UNCOMMITTED on RL branch `autoresearch/2026-08-20-lineage-ledger/impl`: nemo_gym.py (external_staging config, manifest fetch + local receipt assembly via `_assemble_receipt`), rollout_manager.py (fail_rollouts/gate_metrics removed), single_controller.py (_log_gate_metrics removed), blackbox_finalizer.py (finalize/token_in_rate + capture_poisoned_rollouts metrics), config.py (TTL fields dropped), exemplar YAML, docs (design-docs/token-capture-ledger.md created; guide updated), rewritten tests (test_nemo_gym_token_capture.py, test_rollout_manager.py updates). RL unit tests running in background task baao872q9. +CAMPAIGN COMPLETE. The lineage-ledger plan is implemented, unit-tested, and validated by a passing 5-step smoke A/B. + +- Gym branch `token-capture-lineage-ledger` (head: lint fix on 1e1a16cb): gate deleted, ledger implemented, 436 tests green. +- RL branch `autoresearch/2026-08-20-lineage-ledger/impl` (head 5bc2b50d1): receipt assembly from manifest, gate plumbing removed, ledger metrics in finalizer, launch-infra fixes (venv rebuild-list, UV cache, content-None legacy fix, TQ pin skip, SC_EXP_NAME), docs + design doc, A/B report. +- Smoke A/B (seed 43, R3 on, num_workers=2, 5 steps): capture 6359951 PASS (58:44), legacy 6359952 PASS (1:07:03). Capture dynamics strictly tighter (tmpe 1.0137-1.0147 vs 1.020-1.066; gen_kl ~0.001 vs ~0.004). Ledger: 2821 rows, 0 unresolved/worker failures, token_in_rate 0.95-0.99. +- Full report: reports/auto_research/lineage-ledger-0820/ledger-ab-report.md; ledger TSV in the same dir. ## Next Actions -- Confirm RL unit tests green (env tests, blackbox_finalizer, rollout_manager, tq_token_sink, vllm_token_capture_hosting); fix failures; then commit RL side (sign-off + Co-Authored-By Claude Fable 5). -- Launch 5-step smoke A/B from repo root (fsw path): - - capture arm: `DRY_RUN=0 SC_EXP_NAME=ledger-capture-s43-0820 NG_TIC_FP_CANONICAL=1 WALLTIME=3:59:00 WANDB_PROJ=PR3456-Ledger-AB-0820 bash swe_nano_sc_capture.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 token_capture.num_finalizer_workers=2 token_capture.defer_routed_experts_to_policy=true token_capture.staging_partition=rollout_staging_ledger_smoke_0820 +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 +env.nemo_gym.model_endpoint_readiness_timeout_seconds=1800 policy.generation.vllm_cfg.reasoning_parser_plugin=/opt/nemo-rl/nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py` - - legacy arm: same via `swe_nano_sc.sh` (no NG_TIC_FP_CANONICAL, no token_capture overrides, keep num_workers=2 + R3 + sampler + seed 43, SC_EXP_NAME=ledger-legacy-s43-0820). - - dry-run of capture arm validated: UV cache resolves to /lustre/fsw/portfolios/llmservice/users/pthombre/uv (was /tmp; fixed via UV_CACHE_DIR/UV_CACHE_DIR_OVERRIDE in swe_nano.env). -- Compare token_mult_prob_error, gen_kl_error, reward across 5 steps + perf (row_assembly/* timings, step time, finalize/token_in_rate ≥0.99) per docs/guides/nano-swe-token-capture.md; write TSV ledger in reports/auto_research/lineage-ledger-0820/. +- Push both branches and update Gym PR #2278 / RL PR #3456; run CI (/ok to test) — CI covers the Ray-heavy RL suites that cannot run outside the container. +- Follow-up items (report §Follow-ups): retry idempotency (recovers ~36% fail-closed rollout loss from per-call aborts), vLLM engine ClientOSError bursts, ledger-file drop on publish. ## Watch Outs -- swe_nano.env was retargeted to pthombre paths + nightly-gym 08-10 container (commit with the work). swe_nano.secrets.env copied, chmod 600, git-excluded via .git/info/exclude. -- Prior 08-19 ledger smoke runs (pr3456-ledger-*-s43-0819*) in RL-pr3456-delta-smoke-workspace came from a DIFFERENT since-reverted implementation — reference only for launch shape, not results. -- Slurm submits from a networked shell; jobs monitored via slurm-broker MCP or squeue. -- Do not stop before both arms complete 5 steps and comparison is written (user stop rule). +- swe_nano.env now targets pthombre paths + nightly-gym 08-10; never set UV_CACHE_DIR_OVERRIDE with prefetch-venvs containers; uv cache = /lustre/fsw/portfolios/llmservice/users/pthombre/uv (never /tmp). +- swe_nano.secrets.env is untracked (git-excluded via .git/info/exclude), mode 600. +- Lustre git operations intermittently time out (~2 min) — retry with timeout. diff --git a/session/20260820_004547/timeline.md b/session/20260820_004547/timeline.md index 9bf8a6de934..17939ac0db3 100644 --- a/session/20260820_004547/timeline.md +++ b/session/20260820_004547/timeline.md @@ -33,3 +33,4 @@ - BOTH SMOKE ARMS PASSED 5/5 steps. Capture: tmpe 1.0137-1.0147, gen_kl ~0.001, token_in_rate 0.95-0.99, 2821 ledger rows / 0 unresolved / 0 worker failures; 16 rollouts fail-closed on aborted calls (request_finished_without_staged_coordinates — retry-idempotency follow-up). Legacy: tmpe 1.020-1.066, gen_kl ~0.004, all 8 samples valid every step. Capture strictly tighter — matches gate-era campaign bands. - Review of legacy fix done via /review-pr (local): fix correct; added 2 regression params to test_nemo_gym_utils (7/7 pass); renamed _content -> item_content. - Report: reports/auto_research/lineage-ledger-0820/ledger-ab-report.md. +- 2026-08-20 11:29:36 r3 rerun 6365594 COMPLETED 5/5 (1:11:49): carve-out validated — valid rows 32/40 (was 24/40), zero failure-row rejections; residue 9x missing_terminal_row (2 first-call deaths + ~7 doomed-attempt terminal ids from harness). Dynamics unchanged (tmpe 1.0137-1.0171, gen_kl ~0.001). diff --git a/session/20260820_111240/handoff.md b/session/20260820_111240/handoff.md new file mode 100644 index 00000000000..6a2734b08bb --- /dev/null +++ b/session/20260820_111240/handoff.md @@ -0,0 +1,16 @@ +# Handoff + +## Resume From Here +Multi-worker ledger revalidation job **6368085** (het 6368085-6368086) submitted 2026-08-20 11:24, PENDING on batch/short. Branch `autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation` (3 commits: launcher port 53713adeb, wrapper cbadca28b, ultra_launch fixes 02d700023). Mirrors failed job 6300221 with num_workers=16/4 intact. + +## Next Actions +- Poll: `squeue -j 6368085`; once running, watch + `/lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/ray_logs/nano35-rlvr-sc-tc-smoke-ledger/6368085-logs/ray-driver.log` +- Verdict greps on driver log: `grep -c '409 Conflict'` (must be 0), `grep -c UnknownRolloutError` (must be 0), `Collecting rollouts` reaches 16/16, >=2 steps + nccl_reshard sync, `capture_dir/lineage/*.lineage.jsonl` populated, no `manifest(...) fetch failed`. +- On verdict: add row to reports/auto_research/lineage-ledger-0820/experiments.tsv. + +## Watch Outs +- UV cache must stay /lustre/fsw/portfolios/llmservice/users/pthombre/uv — never /tmp, never UV_CACHE_DIR_OVERRIDE with prefetch containers. +- Do NOT pin num_workers=1 — 16/4 is the regression trigger under test. +- rollout_checkpointing.* overrides are ignored (extra-allow) in this tree — expected. +- Secrets in swe_nano.secrets.env (untracked, mode 600). diff --git a/session/20260820_111240/session_state.md b/session/20260820_111240/session_state.md new file mode 100644 index 00000000000..ac8a6ac8302 --- /dev/null +++ b/session/20260820_111240/session_state.md @@ -0,0 +1,41 @@ +# Session State + +- Session: 20260820_111240 +- Repo: /lustre/fs1/portfolios/llmservice/projects/llmservice_fm_text/users/pthombre/sweRun/RL-pr3456-delta-staging (fsw twin: /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-staging) +- Branch: autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation +- Started: 2026-08-20 11:12 +- Updated: 2026-08-20 11:25 + +## Goal +Rerun the failed nano-35-rlvr-sc-tc-smoke shape (job 6300221, amahishi) with identical params — critically policy_model num_workers=16 / reasoning_off 4 — on the lineage-ledger token-capture code, to prove the multi-worker gate failure (409 Conflict + UnknownRolloutError from the per-process LineageRegistry) cannot recur. + +## Current Subtask +Submit the smoke job and monitor to verdict. + +## Loaded Skills +- `launch-nemo-rl` — debugging playbook (k8s-focused; this run is Slurm). +- `nemo-rl-auto-research` — campaign workflow, TSV ledger, branching. +- `nemo-rl-session-memory` — this record. + +## Current Status +- Root cause confirmed: job 6300221's gate registry was per-uvicorn-process; register PUT hit 1 of 16 workers, data-plane calls load-balanced → 409 at prepare, UnknownRolloutError at ingest after fail_rollouts cleanup. Current code (Gym 1e1a16cb ledger) has no registration and uses FileLineageStore under a shared root with fcntl locks. +- Ported examples/nemo_gym/nemotron-3.5-nano/ (5 files) from nemo-rl-partial-rollout-recovery-refresh (commit 53713adeb). +- Wrote nano35_ledger_smoke.sh batch wrapper (commit cbadca28b): pthombre paths, zhiyul nightly-gym.2026-08-10 container, proven 0820 launch plumbing, UV_CACHE_DIR=/lustre/fsw/portfolios/llmservice/users/pthombre/uv (user directive: NEVER /tmp). +- Fixed examples/nemo_gym/nemotron-3-ultra/ultra_launch.sh (commit 02d700023): NRL_ENTRYPOINT, NRL_DRIVER_PIP_INSTALL/PYTHONPATH, NRL_DRIVER_UV_RUN_FLAGS, external UV_CACHE_DIR precedence. +- Dry run clean: SC driver + --locked --no-sync, lustre UV cache, num_workers 16/4 from student_rlvr1.yaml defaults, all model/data/judge paths readable. + +## Plan +- [x] Port launchers, write wrapper, dry-run +- [ ] DRY_RUN=0 submit; record job id +- [ ] Monitor ray-driver.log; verdict greps +- [ ] Record row in reports/auto_research/lineage-ledger-0820/experiments.tsv + +## Pass Criteria +0x "409 Conflict", 0x UnknownRolloutError in driver log; Collecting rollouts reaches 16/16; >=2 steps incl. nccl_reshard sync; lineage/*.lineage.jsonl populated; no manifest fetch failures. + +## Assumptions +- rollout_checkpointing.* overrides are extra-allowed no-ops in this tree (MasterConfig extra="allow"). +- Container deviation (nightly-gym.2026-08-10 vs amahishi's bake) is required for this branch's validated venv plumbing. + +## Blockers +- None known. From cc33c363a062e2c26ec79c9539fbc59521bd3065 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Thu, 20 Aug 2026 11:43:54 -0700 Subject: [PATCH 43/52] fix(smoke): disable trainer checkpointing for the ledger revalidation run This tree's SingleController raises NotImplementedError on checkpointing.enabled=true (job 6368085 died at setup in 14 min); the recovery-refresh branch supports it but checkpointing is orthogonal to the multi-worker capture path under test. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre --- nano35_ledger_smoke.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/nano35_ledger_smoke.sh b/nano35_ledger_smoke.sh index dd1d8137231..62f59cf2966 100644 --- a/nano35_ledger_smoke.sh +++ b/nano35_ledger_smoke.sh @@ -84,6 +84,10 @@ set +a # Same trailing override as the original invocation. rollout_checkpointing is # an extra-allowed (ignored) block in this tree; kept for command fidelity. +# checkpointing.enabled=false: this tree's SingleController raises +# NotImplementedError on trainer checkpointing (the recovery-refresh branch +# supports it); irrelevant to the multi-worker capture path under test. exec bash "${HERE}/examples/nemo_gym/nemotron-3.5-nano/nano35_dolphin_launch_sc_smoke.sh" \ rollout_checkpointing.restore_mode=none \ + checkpointing.enabled=false \ "$@" From baced0988d7c573e11185cd038a54de8982ed541 Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Thu, 20 Aug 2026 12:26:11 -0700 Subject: [PATCH 44/52] fix(launcher): mount worktree project files and workspace members in examples ultra_launch Node-side venv rebuilds run 'uv sync --locked' against /opt/nemo-rl; with only nemo_rl/ and Gym overlaid, the rebased Gym pyproject paired with the container's baked uv.lock fails the lock check (job 6368547 died in _env_builder with CalledProcessError). Ports the root ultra_launch.sh mount block: pyproject.toml, uv.lock, .python-version and the 3rdparty workspace members. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre --- .../nemo_gym/nemotron-3-ultra/ultra_launch.sh | 27 ++++++++++++++++--- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/examples/nemo_gym/nemotron-3-ultra/ultra_launch.sh b/examples/nemo_gym/nemotron-3-ultra/ultra_launch.sh index 24cce7205fc..5a7f1ef0009 100755 --- a/examples/nemo_gym/nemotron-3-ultra/ultra_launch.sh +++ b/examples/nemo_gym/nemotron-3-ultra/ultra_launch.sh @@ -753,14 +753,33 @@ if [[ -d "${OVERLAY_SOURCE}/nemo_rl" ]]; then _append_mount "${OVERLAY_SOURCE}/nemo_rl:/opt/nemo-rl/nemo_rl" echo " Mount: nemo_rl → /opt/nemo-rl/nemo_rl" fi +# The worktree's project files must ride along with the overlaid sources: +# node-side venv rebuilds run `uv sync --locked` against /opt/nemo-rl, and a +# mounted Gym/pyproject paired with the container's baked uv.lock fails the +# lock check (job 6368547). +for _project_file in pyproject.toml uv.lock .python-version; do + if [[ -f "${OVERLAY_SOURCE}/${_project_file}" ]]; then + _append_mount "${OVERLAY_SOURCE}/${_project_file}:/opt/nemo-rl/${_project_file}" + echo " Mount: ${_project_file} → /opt/nemo-rl/${_project_file}" + fi +done if [[ -d "${OVERLAY_SOURCE}/examples/configs" ]]; then _append_mount "${OVERLAY_SOURCE}/examples/configs:/opt/nemo-rl/examples/configs" echo " Mount: configs → /opt/nemo-rl/examples/configs" fi -if [[ -d "${OVERLAY_SOURCE}/3rdparty/Gym-workspace/Gym" ]]; then - _append_mount "${OVERLAY_SOURCE}/3rdparty/Gym-workspace/Gym:/opt/nemo-rl/3rdparty/Gym-workspace/Gym" - echo " Mount: Gym → /opt/nemo-rl/3rdparty/Gym-workspace/Gym" -fi +_local_project_paths=( + "3rdparty/TensorRT-LLM-workspace" + "3rdparty/Automodel-workspace/Automodel" + "3rdparty/Megatron-Bridge-workspace/Megatron-Bridge" + "3rdparty/Gym-workspace/Gym" + "research/template_project" +) +for _local_project_path in "${_local_project_paths[@]}"; do + if [[ -d "${OVERLAY_SOURCE}/${_local_project_path}" ]]; then + _append_mount "${OVERLAY_SOURCE}/${_local_project_path}:/opt/nemo-rl/${_local_project_path}" + echo " Mount: ${_local_project_path} → /opt/nemo-rl/${_local_project_path}" + fi +done if [[ "${USE_SNAPSHOT}" == "1" ]]; then _append_mount "${SNAPSHOT_DIR}:${SNAPSHOT_DIR}" From 6f3bf5a1f749c7c9b9d5ea4fc7e45e09e676086b Mon Sep 17 00:00:00 2001 From: Pranav Thombre Date: Thu, 20 Aug 2026 12:56:55 -0700 Subject: [PATCH 45/52] fix(refit): route mtp.-prefixed params to the misc broadcast path Ported from the recovery-refresh tree: nano-3.5's MTP head exports HF names with a bare mtp. prefix, which matched the FFN suffix whitelist and tripped the single-layer-prefix assertion in prepare_nccl_reshard_refit_info (job 6370717: 'layer_prefix mismatch: mtp != backbone'). vLLM keeps the MTP drafter separate and updates it through load_weights, so these belong on the misc packed_broadcast path. Co-Authored-By: Claude Fable 5 Signed-off-by: Pranav Thombre --- nemo_rl/weight_sync/nccl_reshard_utils.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/nemo_rl/weight_sync/nccl_reshard_utils.py b/nemo_rl/weight_sync/nccl_reshard_utils.py index 8dd978e6772..f4914ac3136 100644 --- a/nemo_rl/weight_sync/nccl_reshard_utils.py +++ b/nemo_rl/weight_sync/nccl_reshard_utils.py @@ -191,9 +191,13 @@ def is_nccl_reshard_param(param_name: str) -> bool: ``load_weights`` path. Shared-expert FFN weights (``*.shared_expert.*``) are routed to misc path. + Bare ``mtp.``-prefixed HF names are routed to misc too, because vLLM keeps + the MTP drafter separate and updates it through ``load_weights``. """ if "shared_expert" in param_name: return False + if param_name.startswith("mtp."): + return False return param_name.endswith(FFN_PROJ_WEIGHT_SUFFIXES) or param_name.endswith( FFN_GROUPED_EXPERT_SUFFIXES ) From c8ca62e7f720d44a71c7e1fcd66cf44e7e7c74a3 Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Thu, 20 Aug 2026 17:09:53 -0700 Subject: [PATCH 46/52] feat(token-capture): heuristic terminal selection for undeclared rollouts Rollouts whose harness reports no terminal_logical_request_id previously finalized as missing_receipt placeholders (every rollout of the nano35 rlvr smoke, job 6371991). _postprocess_receipt_mode now always fetches the manifest; _assemble_receipt falls back to Gym's select_terminal_call when no terminal is declared, with strict precedence declared > heuristic > mask: a declared id that matches no row still masks (never falls back), and any ambiguous manifest shape masks with the selection reason. The receipt records the path in terminal_selection and the finalizer emits finalize/heuristic_terminal_count|fraction per group. Pin bump: Gym token-capture-lineage-ledger + admitted_at ledger column (G1) and select_terminal_call + RolloutReceipt.terminal_selection (G2). Signed-off-by: Prashant Thombre --- 3rdparty/Gym-workspace/Gym | 2 +- docs/design-docs/token-capture-ledger.md | 12 +++ docs/guides/nano-swe-token-capture.md | 5 + nemo_rl/environments/nemo_gym.py | 95 +++++++++++++------ nemo_rl/experience/blackbox_finalizer.py | 13 +++ .../data_plane/test_blackbox_finalizer.py | 5 + .../test_nemo_gym_token_capture.py | 78 ++++++++++++++- 7 files changed, 176 insertions(+), 34 deletions(-) diff --git a/3rdparty/Gym-workspace/Gym b/3rdparty/Gym-workspace/Gym index 74d4ef851b2..0edca0591d5 160000 --- a/3rdparty/Gym-workspace/Gym +++ b/3rdparty/Gym-workspace/Gym @@ -1 +1 @@ -Subproject commit 74d4ef851b29a0185256d256361b485ec8898d63 +Subproject commit 0edca0591d5330c73054b3ca5b1229ec7a18a676 diff --git a/docs/design-docs/token-capture-ledger.md b/docs/design-docs/token-capture-ledger.md index 0898ff14552..f6cebb59d3c 100644 --- a/docs/design-docs/token-capture-ledger.md +++ b/docs/design-docs/token-capture-ledger.md @@ -95,6 +95,18 @@ NeMo RL fetches the manifest at rollout end and assembles the receipt locally: - `capture_poisoned` = any failure row present, or no row for the terminal request. +Terminal selection has a strict precedence: **declared > heuristic > mask**. A +harness-declared terminal is authoritative — a declared id that matches no +committed row masks the rollout and never falls back. When the harness reports +no terminal at all, Gym's `select_terminal_call` infers one from the +manifest's explicit parent links (earliest-admitted root by `admitted_at`, an +extended sibling beating an abandoned childless retry); any ambiguous shape — +a retry of the final call, divergent extended branches — masks with the +selection reason. The heuristic only chooses *among* digest-verified rows: +`verify_and_linearize` still verifies the chosen chain. The receipt records +the path in `terminal_selection` and the finalizer emits +`finalize/heuristic_terminal_fraction` per group. + `verify_and_linearize(receipt, snapshots)` runs unchanged. Retry duplicates appear as dead-branch sibling rows in the manifest: their staged rows are fetched, verified, and cleaned like any other, but they never join the diff --git a/docs/guides/nano-swe-token-capture.md b/docs/guides/nano-swe-token-capture.md index 0612060abfe..c165f1c0df6 100644 --- a/docs/guides/nano-swe-token-capture.md +++ b/docs/guides/nano-swe-token-capture.md @@ -90,6 +90,11 @@ The call flow for a single SWE rollout, end to end: staging keys, digests, weight versions), a `terminal_model_call_id` selected by the terminal logical request id, and fail-closed poisoning (any failure row, or a missing terminal row, masks the rollout). + Harnesses that report no terminal id (declared > heuristic > mask + precedence) fall back to Gym's `select_terminal_call`, which infers the + terminal from the manifest's parent links and masks on any ambiguity; the + per-group `finalize/heuristic_terminal_fraction` metric meters that + fallback and should stay 0 on the SWE recipe, whose harness declares. 7. **Finalize.** The controller constructs a metadata-only `FinalizationRequest`, releases the rollout concurrency permit, and submits it to a fixed pool of CPU Ray actors. Each actor owns a connect-only TQ diff --git a/nemo_rl/environments/nemo_gym.py b/nemo_rl/environments/nemo_gym.py index b5db2a0ca04..346a72dd9d6 100644 --- a/nemo_rl/environments/nemo_gym.py +++ b/nemo_rl/environments/nemo_gym.py @@ -704,23 +704,28 @@ async def _postprocess_receipt_mode( ) rollout_id = nemo_gym_row[_NG_ROLLOUT_ID_BODY_KEY] terminal_logical_request_id = nemo_gym_result.get("terminal_logical_request_id") + if not ( + isinstance(terminal_logical_request_id, str) and terminal_logical_request_id + ): + # A harness that reports no terminal id still gets its manifest + # fetched; receipt assembly falls back to heuristic selection. + terminal_logical_request_id = None receipt = None - if isinstance(terminal_logical_request_id, str) and terminal_logical_request_id: - try: - manifest = await self._control( - "GET", - f"{_TOKEN_CAPTURE_CONTROL_PREFIX}" - f"/rollouts/{rollout_id}/manifest", - ) - receipt = self._assemble_receipt( - rollout_id, - manifest, - terminal_logical_request_id=terminal_logical_request_id, - reward=float(nemo_gym_result.get("reward") or 0.0), - ) - except (RuntimeError, OSError) as error: - # An unfetchable manifest finalizes as a placeholder row. - print(f"manifest({rollout_id}) fetch failed: {error}", flush=True) + try: + manifest = await self._control( + "GET", + f"{_TOKEN_CAPTURE_CONTROL_PREFIX}" + f"/rollouts/{rollout_id}/manifest", + ) + receipt = self._assemble_receipt( + rollout_id, + manifest, + terminal_logical_request_id=terminal_logical_request_id, + reward=float(nemo_gym_result.get("reward") or 0.0), + ) + except (RuntimeError, OSError) as error: + # An unfetchable manifest finalizes as a placeholder row. + print(f"manifest({rollout_id}) fetch failed: {error}", flush=True) return { "message_log": [], "input_message_log": [], @@ -734,14 +739,19 @@ def _assemble_receipt( rollout_id: str, manifest: dict, *, - terminal_logical_request_id: str, + terminal_logical_request_id: Optional[str], reward: float, ) -> dict: """Build the token-free RolloutReceipt payload from a ledger manifest. - The terminal row is the manifest row whose ``logical_request_id`` - equals the response id the agent actually kept. Retry duplicates are - dead-branch rows: they stay in the manifest (their staged rows are + With a declared terminal (``terminal_logical_request_id``), the + terminal row is the manifest row whose ``logical_request_id`` equals + the response id the agent actually kept; a declared id that matches no + row masks the rollout and never falls back. Without a declaration, + Gym's ``select_terminal_call`` infers the terminal from the manifest's + parent links (fail-closed: ambiguity masks). The receipt records which + path chose the terminal in ``terminal_selection``. Retry duplicates + are dead-branch rows: they stay in the manifest (their staged rows are fetched, verified, and cleaned) but never join the terminal chain — ``verify_and_linearize`` tolerates rows unreferenced by the terminal chain. @@ -764,14 +774,40 @@ def _assemble_receipt( deduped: dict[str, dict] = {} for record in records: deduped.setdefault(str(record.get("model_call_id")), record) - terminal_record = next( - ( - record - for record in deduped.values() - if record.get("logical_request_id") == terminal_logical_request_id - ), - None, - ) + terminal_record = None + selection_reason = None + if terminal_logical_request_id is not None: + terminal_selection = "declared" + terminal_record = next( + ( + record + for record in deduped.values() + if record.get("logical_request_id") == terminal_logical_request_id + ), + None, + ) + else: + terminal_selection = "heuristic" + # Deferred: nemo_gym is an optional extra absent in non-gym runs. + from nemo_gym.token_id_capture.staging.records import CallRecord + from nemo_gym.token_id_capture.staging.terminal import ( + select_terminal_call, + ) + + try: + selection = select_terminal_call( + [ + CallRecord.model_validate(record) + for record in deduped.values() + ] + ) + except ValueError: + selection_reason = "invalid_manifest_row" + else: + if selection.terminal_model_call_id is not None: + terminal_record = deduped[selection.terminal_model_call_id] + else: + selection_reason = selection.reason poisoning_failures = [ failure for failure in failures @@ -781,7 +817,7 @@ def _assemble_receipt( if poisoning_failures: failure_reason = str(poisoning_failures[0].get("reason") or "capture_failed") elif terminal_record is None: - failure_reason = "missing_terminal_row" + failure_reason = selection_reason or "missing_terminal_row" return { "rollout_id": rollout_id, "reward": reward, @@ -793,6 +829,7 @@ def _assemble_receipt( "manifest": list(deduped.values()), "capture_poisoned": failure_reason is not None, "failure_reason": failure_reason, + "terminal_selection": terminal_selection, } def _postprocess_nemo_gym_to_nemo_rl_result( diff --git a/nemo_rl/experience/blackbox_finalizer.py b/nemo_rl/experience/blackbox_finalizer.py index 60fd5ff987b..d13c904670d 100644 --- a/nemo_rl/experience/blackbox_finalizer.py +++ b/nemo_rl/experience/blackbox_finalizer.py @@ -557,6 +557,19 @@ def finalize_group( if isinstance(receipt, dict) and receipt.get("capture_poisoned") ) ) + # Heuristic terminal selection is a fallback for harnesses that do not + # declare the response they kept; a nonzero fraction on a declaring + # harness is a regression signal. + heuristic_receipts = sum( + 1 + for receipt in receipts + if isinstance(receipt, dict) + and receipt.get("terminal_selection") == "heuristic" + ) + metrics["finalize/heuristic_terminal_count"] = float(heuristic_receipts) + metrics["finalize/heuristic_terminal_fraction"] = heuristic_receipts / len( + receipts + ) for row in rows: if not row.valid: print( diff --git a/tests/unit/data_plane/test_blackbox_finalizer.py b/tests/unit/data_plane/test_blackbox_finalizer.py index 07526c55292..21a92fa6b5b 100644 --- a/tests/unit/data_plane/test_blackbox_finalizer.py +++ b/tests/unit/data_plane/test_blackbox_finalizer.py @@ -201,6 +201,9 @@ def test_finalize_group_publishes_n_rows_with_placeholder(tq_client, partitions) tq_client, "worked_example", rollout_id=f"{group_id}_g0" ) receipt["rollout_id"] = f"{group_id}_g0" + # Mark this receipt's terminal as heuristically selected so the group + # metric sees a mixed declared/heuristic population. + receipt["terminal_selection"] = "heuristic" rollout_ids = [f"{group_id}_g0", f"{group_id}_g1"] finalizer = _finalizer(tq_client) @@ -217,6 +220,8 @@ def test_finalize_group_publishes_n_rows_with_placeholder(tq_client, partitions) # Group staleness comes from the valid rollout's calls (wv 4), not the fallback. assert (finalized.group_min_wv, finalized.group_max_wv) == (4, 4) assert finalized.metrics["finalize/invalid_row_rate"] == 0.5 + assert finalized.metrics["finalize/heuristic_terminal_count"] == 1.0 + assert finalized.metrics["finalize/heuristic_terminal_fraction"] == 0.5 rows = _fetch_rows(tq_client, rollout_ids) sample_mask = torch.as_tensor(rows["sample_mask"]).flatten() diff --git a/tests/unit/environments/test_nemo_gym_token_capture.py b/tests/unit/environments/test_nemo_gym_token_capture.py index a561e8cda24..aa564ec10f2 100644 --- a/tests/unit/environments/test_nemo_gym_token_capture.py +++ b/tests/unit/environments/test_nemo_gym_token_capture.py @@ -42,9 +42,15 @@ def _manifest_record(call_id: str, *, logical_request_id: str, parent: str | Non } -def test_receipt_postprocess_fails_closed_without_a_terminal_logical_id() -> None: +def test_receipt_postprocess_without_a_terminal_logical_id_uses_the_heuristic() -> None: env = _capture_env() - env._control = AsyncMock() + records = [ + _manifest_record("c1", logical_request_id="lr-1"), + _manifest_record("c2", logical_request_id="lr-2", parent="c1"), + ] + env._control = AsyncMock( + return_value={"rollout_id": "r0", "records": records, "failures": []} + ) result = asyncio.run( env._postprocess_receipt_mode( @@ -53,8 +59,11 @@ def test_receipt_postprocess_fails_closed_without_a_terminal_logical_id() -> Non ) ) - env._control.assert_not_awaited() - assert result["receipt"] is None + env._control.assert_awaited() + receipt = result["receipt"] + assert receipt["terminal_model_call_id"] == "c2" + assert receipt["terminal_selection"] == "heuristic" + assert receipt["capture_poisoned"] is False def test_receipt_postprocess_fetches_manifest_and_selects_terminal_row() -> None: @@ -82,6 +91,7 @@ def test_receipt_postprocess_fetches_manifest_and_selects_terminal_row() -> None receipt = result["receipt"] assert receipt["rollout_id"] == "r0" assert receipt["terminal_model_call_id"] == "c2" + assert receipt["terminal_selection"] == "declared" assert receipt["capture_poisoned"] is False assert receipt["failure_reason"] is None assert receipt["reward"] == 1.0 @@ -163,6 +173,66 @@ def test_receipt_assembly_poisons_when_the_terminal_row_is_missing() -> None: assert receipt["capture_poisoned"] is True assert receipt["failure_reason"] == "missing_terminal_row" assert receipt["terminal_model_call_id"] is None + # A declared id is authoritative: a miss never falls back to the heuristic + # even when the manifest holds an unambiguous chain. + assert receipt["terminal_selection"] == "declared" + + +def test_receipt_assembly_heuristic_eliminates_abandoned_retry() -> None: + env = _capture_env() + records = [ + _manifest_record("c1", logical_request_id="lr-1"), + _manifest_record("c2", logical_request_id="lr-2", parent="c1"), + _manifest_record("c2r", logical_request_id="lr-2r", parent="c1"), + _manifest_record("c3", logical_request_id="lr-3", parent="c2"), + ] + manifest = {"rollout_id": "r0", "records": records, "failures": []} + receipt = env._assemble_receipt( + "r0", manifest, terminal_logical_request_id=None, reward=1.0 + ) + assert receipt["terminal_model_call_id"] == "c3" + assert receipt["terminal_selection"] == "heuristic" + assert receipt["capture_poisoned"] is False + + +def test_receipt_assembly_heuristic_masks_a_final_call_retry() -> None: + env = _capture_env() + records = [ + _manifest_record("c1", logical_request_id="lr-1"), + _manifest_record("c2", logical_request_id="lr-2", parent="c1"), + _manifest_record("c2r", logical_request_id="lr-2r", parent="c1"), + ] + manifest = {"rollout_id": "r0", "records": records, "failures": []} + receipt = env._assemble_receipt( + "r0", manifest, terminal_logical_request_id=None, reward=0.0 + ) + assert receipt["terminal_model_call_id"] is None + assert receipt["capture_poisoned"] is True + assert receipt["failure_reason"] == "ambiguous_terminal" + + +def test_receipt_assembly_heuristic_masks_an_empty_manifest() -> None: + env = _capture_env() + manifest = {"rollout_id": "r0", "records": [], "failures": []} + receipt = env._assemble_receipt( + "r0", manifest, terminal_logical_request_id=None, reward=0.0 + ) + assert receipt["terminal_model_call_id"] is None + assert receipt["capture_poisoned"] is True + assert receipt["failure_reason"] == "no_records" + + +def test_receipt_assembly_heuristic_masks_invalid_manifest_rows() -> None: + env = _capture_env() + bad = _manifest_record("c1", logical_request_id="lr-1") + bad["delta_len"] = 0 # violates the CallRecord length contract + manifest = {"rollout_id": "r0", "records": [bad], "failures": []} + receipt = env._assemble_receipt( + "r0", manifest, terminal_logical_request_id=None, reward=0.0 + ) + assert receipt["terminal_model_call_id"] is None + assert receipt["capture_poisoned"] is True + assert receipt["failure_reason"] == "invalid_manifest_row" def test_receipt_assembly_keeps_dead_branch_siblings_in_the_manifest() -> None: From 5dd4186b15881df744216ba83049673d7c59dceb Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Sun, 23 Aug 2026 02:22:20 -0700 Subject: [PATCH 47/52] feat(token-capture): fetch staged prefix chains Signed-off-by: Prashant Thombre --- 3rdparty/Gym-workspace/Gym | 2 +- nemo_rl/data_plane/tq_token_sink.py | 25 +++++++ .../generation/vllm/vllm_worker_async.py | 65 ++++++++++++++++- tests/unit/data_plane/test_tq_token_sink.py | 41 +++++++++++ .../test_vllm_token_capture_hosting.py | 73 +++++++++++++++++++ 5 files changed, 202 insertions(+), 4 deletions(-) diff --git a/3rdparty/Gym-workspace/Gym b/3rdparty/Gym-workspace/Gym index 0edca0591d5..d6acbb873b8 160000 --- a/3rdparty/Gym-workspace/Gym +++ b/3rdparty/Gym-workspace/Gym @@ -1 +1 @@ -Subproject commit 0edca0591d5330c73054b3ca5b1229ec7a18a676 +Subproject commit d6acbb873b82f8cf9511f603eb24e789c291ee5c diff --git a/nemo_rl/data_plane/tq_token_sink.py b/nemo_rl/data_plane/tq_token_sink.py index 39822c2fbc2..b38bdf7a34d 100644 --- a/nemo_rl/data_plane/tq_token_sink.py +++ b/nemo_rl/data_plane/tq_token_sink.py @@ -358,6 +358,31 @@ def fetch(self, staging_keys: list[str]) -> list[StagedCallSnapshot]: snapshots.append(snapshot) return snapshots + def fetch_prefix_token_ids(self, staging_keys: list[str]) -> list[int]: + """Bulk-fetch ordered delta chain and concatenate token_ids_delta into a prefix.""" + if not staging_keys: + return [] + if len(set(staging_keys)) != len(staging_keys): + raise KeyError("prefix fetch: staging_keys contains duplicates") + rows = _call_dp( + self._dp_client, + "get_samples", + sample_ids=list(staging_keys), + partition_id=self._staging_partition, + select_fields=["token_ids_delta"], + ) + n_rows = int(rows.batch_size[0]) if rows.batch_size else 0 + if n_rows != len(staging_keys): + raise KeyError( + f"prefix fetch incomplete: requested {len(staging_keys)} keys, got {n_rows}" + ) + result: list[int] = [] + for index in range(n_rows): + row = _select_row(rows, index) + delta = row["token_ids_delta"].squeeze(0).tolist() + result.extend(int(t) for t in delta) + return result + def fetch_for_finalization( self, staging_keys: list[str] ) -> list[FetchedStagedCall]: diff --git a/nemo_rl/models/generation/vllm/vllm_worker_async.py b/nemo_rl/models/generation/vllm/vllm_worker_async.py index 2a1c3545fad..8991f3d59ac 100644 --- a/nemo_rl/models/generation/vllm/vllm_worker_async.py +++ b/nemo_rl/models/generation/vllm/vllm_worker_async.py @@ -111,6 +111,8 @@ def __init__( # In-flight captured calls keyed by id(request): (ActiveCall, the # exact engine prompt ids recorded at preprocess time). self._capture_calls: dict[int, tuple[Any, list[int]]] = {} + self._staging_source: Any | None = None + self._prefix_cache: dict[str, list[int]] = {} super().__init__( config, @@ -379,10 +381,14 @@ async def setup_token_capture( from nemo_gym.token_id_capture.staging import install_capture from nemo_rl.data_plane import build_data_plane_client - from nemo_rl.data_plane.tq_token_sink import TQTokenSink + from nemo_rl.data_plane.tq_token_sink import TQTokenSink, TQTokenSource dp_client = build_data_plane_client(dp_cfg, bootstrap=False) sink = TQTokenSink(dp_client, staging_partition=staging_partition) + self._staging_source = TQTokenSource( + dp_client, staging_partition=staging_partition + ) + self._prefix_cache.clear() install_capture( self, sink=sink, @@ -415,6 +421,47 @@ def _begin_request_capture(self, request: Any, prompt_token_ids: list[int]) -> N ) self._capture_calls[id(request)] = (call, list(prompt_token_ids)) + def _fetch_chain_prefix(self, staging_chain: list[str]) -> list[int]: + """Assemble prefix token ids from staging_chain, with a worker-local LRU cache.""" + cache = self._prefix_cache + cached_ids: list[int] = [] + miss_start = 0 + for i, key in enumerate(staging_chain): + if key in cache: + cached_ids = cache[key] + miss_start = i + 1 + miss_keys = staging_chain[miss_start:] + if not miss_keys: + return list(cached_ids) + if self._staging_source is None: + raise RuntimeError( + "_staging_source not initialized; call setup_token_capture() first" + ) + fetched = self._staging_source.fetch_prefix_token_ids(miss_keys) + result = cached_ids + fetched + last_key = staging_chain[-1] + cache[last_key] = result + if len(cache) > 256: + del cache[next(iter(cache))] + return result + + def _patch_chain_prefix(self, ng_capture: dict[str, Any]) -> list[int] | None: + """Fetch a staged prefix and patch a raw capture admission in place.""" + staging_chain = ng_capture.get("staging_chain") or [] + if not staging_chain: + return None + prefix_token_ids = self._fetch_chain_prefix(staging_chain) + prev_len = ng_capture.get("prev_len") + if type(prev_len) is not int or prev_len <= 0: + raise ValueError("staging_chain admission requires prev_len > 0") + if len(prefix_token_ids) != prev_len: + raise ValueError( + "staging_chain prefix length mismatch: " + f"expected {prev_len}, fetched {len(prefix_token_ids)}" + ) + ng_capture["required_prefix_token_ids"] = prefix_token_ids + return prefix_token_ids + @staticmethod def _delta_align_routed_experts( payload: dict[str, Any], *, prev_len: int, prompt_len: int, generated_len: int @@ -663,7 +710,16 @@ async def preprocess_chat( ) raise - if ( + # Check staging_chain in ng_capture before required_prefix_token_ids. + ng_capture_dict = getattr(request, "ng_capture", None) or {} + chain_prefix_token_ids = worker_self._patch_chain_prefix( + ng_capture_dict + ) + + if chain_prefix_token_ids is not None: + # External staging, token-in mode: fetch prefix from TQ. + model_prefix_token_ids = chain_prefix_token_ids + elif ( not hasattr(request, "required_prefix_token_ids") or request.required_prefix_token_ids is None ): @@ -680,7 +736,10 @@ async def preprocess_chat( request, res[1][0]["prompt_token_ids"] ) return res + else: + model_prefix_token_ids = list(request.required_prefix_token_ids) + # Token-in splice path — shared by staging_chain and direct prefix. last_assistant_message_idx = None for i in reversed(range(len(messages_for_replace_prefix_tokens))): if messages_for_replace_prefix_tokens[i]["role"] == "assistant": @@ -720,7 +779,7 @@ async def preprocess_chat( final_prompt_token_ids = replace_prefix_tokens( tokenizer=self.renderer.tokenizer, - model_prefix_token_ids=request.required_prefix_token_ids, + model_prefix_token_ids=model_prefix_token_ids, template_prefix_token_ids=actual_corresponding_token_ids, template_token_ids=engine_prompt["prompt_token_ids"], ) diff --git a/tests/unit/data_plane/test_tq_token_sink.py b/tests/unit/data_plane/test_tq_token_sink.py index 3e3ba63aeec..f712eceabc0 100644 --- a/tests/unit/data_plane/test_tq_token_sink.py +++ b/tests/unit/data_plane/test_tq_token_sink.py @@ -149,3 +149,44 @@ def test_sink_clear_drops_rows(tq_client, staging_partition): sink.clear(keys) with pytest.raises(KeyError): source.fetch(keys) + + +def test_fetch_prefix_token_ids_empty(tq_client, staging_partition): + source = TQTokenSource(tq_client, staging_partition=staging_partition) + assert source.fetch_prefix_token_ids([]) == [] + + +def test_fetch_prefix_token_ids_single_key(tq_client, staging_partition): + sink = TQTokenSink(tq_client, staging_partition=staging_partition) + source = TQTokenSource(tq_client, staging_partition=staging_partition) + records, _, _ = build_fixture_artifacts("single_call") + record = records[0] + assert sink.stage(record).ok + result = source.fetch_prefix_token_ids([record.staging_key]) + assert result == record.token_ids_delta + + +def test_fetch_prefix_token_ids_three_keys_concatenates(tq_client, staging_partition): + sink = TQTokenSink(tq_client, staging_partition=staging_partition) + source = TQTokenSource(tq_client, staging_partition=staging_partition) + records, _, _ = build_fixture_artifacts("worked_example") + for record in records: + assert sink.stage(record).ok + keys = [record.staging_key for record in records] + result = source.fetch_prefix_token_ids(keys) + expected = [t for record in records for t in record.token_ids_delta] + assert result == expected + + +def test_fetch_prefix_token_ids_missing_key_raises_keyerror( + tq_client, staging_partition +): + source = TQTokenSource(tq_client, staging_partition=staging_partition) + with pytest.raises(KeyError): + source.fetch_prefix_token_ids(["ghost_rollout/ghost_call"]) + + +def test_fetch_prefix_token_ids_rejects_duplicates(tq_client, staging_partition): + source = TQTokenSource(tq_client, staging_partition=staging_partition) + with pytest.raises(KeyError, match="duplicates"): + source.fetch_prefix_token_ids(["r/c", "r/c"]) diff --git a/tests/unit/models/generation/test_vllm_token_capture_hosting.py b/tests/unit/models/generation/test_vllm_token_capture_hosting.py index 4bc4424be23..85fa59c90ec 100644 --- a/tests/unit/models/generation/test_vllm_token_capture_hosting.py +++ b/tests/unit/models/generation/test_vllm_token_capture_hosting.py @@ -62,6 +62,8 @@ def _fake_worker(*, is_model_owner: bool = True) -> SimpleNamespace: is_model_owner=is_model_owner, token_capture=None, _rollout_weight_version=0, + _staging_source=None, + _prefix_cache={}, ) worker.install_token_capture = lambda capture: setattr( worker, "token_capture", capture @@ -200,9 +202,14 @@ def _worker_with_capture(sink: _MemorySink): worker = _fake_worker() worker._capture_calls = {} + worker._prefix_cache = {} + worker._staging_source = None worker._delta_align_routed_experts = ( VllmAsyncGenerationWorkerImpl._delta_align_routed_experts ) + worker._fetch_chain_prefix = lambda staging_chain: ( + VllmAsyncGenerationWorkerImpl._fetch_chain_prefix(worker, staging_chain) + ) worker.token_capture = RolloutTokenCapture( sink=sink, weight_version_fn=lambda: worker._rollout_weight_version, @@ -211,6 +218,16 @@ def _worker_with_capture(sink: _MemorySink): return worker +class _MemoryPrefixSource: + def __init__(self, deltas: dict[str, list[int]]) -> None: + self.deltas = deltas + self.calls: list[list[str]] = [] + + def fetch_prefix_token_ids(self, staging_keys: list[str]) -> list[int]: + self.calls.append(list(staging_keys)) + return [token for key in staging_keys for token in self.deltas[key]] + + def _served_content(gen_ids, logprobs): return { "choices": [ @@ -287,6 +304,62 @@ def test_request_capture_token_in_prev_len_chains(): assert sink.records[0].token_ids_delta == [20, 21, 22] +def test_staging_chain_fetches_patches_and_begins_capture(): + sink = _MemorySink() + worker = _worker_with_capture(sink) + source = _MemoryPrefixSource({"r0/c1": [10, 11], "r0/c2": [12]}) + worker._staging_source = source + request = _FakeRequest( + ng_capture={ + "rollout_id": "r0", + "model_call_id": "c3", + "parent_call_id": "c2", + "prev_len": 3, + "mode": "token_in", + "staging_chain": ["r0/c1", "r0/c2"], + }, + stream=False, + ) + + prefix = VllmAsyncGenerationWorkerImpl._patch_chain_prefix( + worker, request.ng_capture + ) + VllmAsyncGenerationWorkerImpl._begin_request_capture(worker, request, prefix + [20]) + + assert prefix == [10, 11, 12] + assert source.calls == [["r0/c1", "r0/c2"]] + assert request.ng_capture["required_prefix_token_ids"] == prefix + call, prompt = worker._capture_calls[id(request)] + assert call.admission.required_prefix_token_ids == prefix + assert prompt == [10, 11, 12, 20] + + +def test_staging_chain_cache_fetches_only_uncached_suffix(): + worker = _worker_with_capture(_MemorySink()) + source = _MemoryPrefixSource({"r0/c1": [10, 11], "r0/c2": [12]}) + worker._staging_source = source + + first = VllmAsyncGenerationWorkerImpl._fetch_chain_prefix(worker, ["r0/c1"]) + second = VllmAsyncGenerationWorkerImpl._fetch_chain_prefix( + worker, ["r0/c1", "r0/c2"] + ) + + assert first == [10, 11] + assert second == [10, 11, 12] + assert source.calls == [["r0/c1"], ["r0/c2"]] + + +def test_staging_chain_rejects_fetched_length_mismatch(): + worker = _worker_with_capture(_MemorySink()) + worker._staging_source = _MemoryPrefixSource({"r0/c1": [10, 11]}) + admission = {"prev_len": 3, "staging_chain": ["r0/c1"]} + + with pytest.raises(ValueError, match="expected 3, fetched 2"): + VllmAsyncGenerationWorkerImpl._patch_chain_prefix(worker, admission) + + assert "required_prefix_token_ids" not in admission + + def test_request_capture_is_a_noop_without_context_or_capture(): sink = _MemorySink() worker = _worker_with_capture(sink) From 91fdfbbf44386662b0bd70654aadd7f5c6b35d68 Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Sun, 23 Aug 2026 02:27:24 -0700 Subject: [PATCH 48/52] fix(smoke): honor code snapshot override Signed-off-by: Prashant Thombre --- swe_nano_sc.sh | 2 ++ swe_nano_sc_capture.sh | 2 ++ 2 files changed, 4 insertions(+) diff --git a/swe_nano_sc.sh b/swe_nano_sc.sh index cc9f622c294..8ab297cdce8 100644 --- a/swe_nano_sc.sh +++ b/swe_nano_sc.sh @@ -22,6 +22,7 @@ HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # Capture DRY_RUN passed on the command line BEFORE sourcing swe_nano.env, which # exports DRY_RUN=1 and would otherwise clobber the caller's value. _DRY_RUN_IN="${DRY_RUN:-}" +_USE_SNAPSHOT_IN="${USE_SNAPSHOT:-}" set -a # shellcheck disable=SC1091 @@ -34,6 +35,7 @@ CONFIG_PATH="${CODE_DIR}/examples/configs/ultra/nano_swe_teacher_sc.yaml" RESULTS_DIR="${WORKSPACE_DIR}/results/${EXP_NAME}" BASE_LOG_DIR="${WORKSPACE_DIR}/ray_logs/${EXP_NAME}" [ -n "${_DRY_RUN_IN}" ] && DRY_RUN="${_DRY_RUN_IN}" +[ -n "${_USE_SNAPSHOT_IN}" ] && USE_SNAPSHOT="${_USE_SNAPSHOT_IN}" set +a bash "${HERE}/ultra_launch.sh" "$@" diff --git a/swe_nano_sc_capture.sh b/swe_nano_sc_capture.sh index 41397528c6c..e8f762551c6 100644 --- a/swe_nano_sc_capture.sh +++ b/swe_nano_sc_capture.sh @@ -17,6 +17,7 @@ HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" _DRY_RUN_IN="${DRY_RUN:-}" _WALLTIME_IN="${WALLTIME:-}" +_USE_SNAPSHOT_IN="${USE_SNAPSHOT:-}" set -a # shellcheck disable=SC1091 @@ -43,6 +44,7 @@ if [ "${CALL_TIMING:-1}" = "1" ]; then fi [ -n "${_DRY_RUN_IN}" ] && DRY_RUN="${_DRY_RUN_IN}" [ -n "${_WALLTIME_IN}" ] && WALLTIME="${_WALLTIME_IN}" +[ -n "${_USE_SNAPSHOT_IN}" ] && USE_SNAPSHOT="${_USE_SNAPSHOT_IN}" set +a bash "${HERE}/ultra_launch.sh" \ From 6d12d9663e4b06fb0905cf133ace7a4e46a47334 Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Mon, 24 Aug 2026 09:49:38 -0700 Subject: [PATCH 49/52] test(token-capture): update fixtures and hosting test for token-free coords Fixtures: _record() now computes real chain_hash/cumulative_hash and chains child records from the root. Hosting test: assert token_ids_delta is absent from ng_commit_coords, and chain_hash/cumulative_hash are present and match the staged record. Add parent_chain_hash to token_in and staging_chain admission dicts (now required by the validator). Signed-off-by: Prashant Thombre --- tests/unit/data_plane/token_capture_test_fixtures.py | 8 ++++++++ .../models/generation/test_vllm_token_capture_hosting.py | 6 ++++++ 2 files changed, 14 insertions(+) diff --git a/tests/unit/data_plane/token_capture_test_fixtures.py b/tests/unit/data_plane/token_capture_test_fixtures.py index 0c492e72d47..8fb7e05b46d 100644 --- a/tests/unit/data_plane/token_capture_test_fixtures.py +++ b/tests/unit/data_plane/token_capture_test_fixtures.py @@ -6,8 +6,10 @@ from __future__ import annotations from nemo_gym.token_id_capture.staging.digest import ( + compute_chain_hash, compute_extras_digest, compute_staging_digest, + hash_token_ids, ) from nemo_gym.token_id_capture.staging.rebuild import verify_and_linearize from nemo_gym.token_id_capture.staging.records import ( @@ -35,6 +37,8 @@ def _record( token_mask: list[float], logprobs: list[float], weight_version: int, + parent_chain_hash: str | None = None, + cumulative_prefix: list[int] | None = None, ) -> StagedCallRecord: token_mask = [f32(value) for value in token_mask] logprobs = [f32(value) for value in logprobs] @@ -56,6 +60,8 @@ def _record( "generation_log_probs_delta": logprobs, "extras": None, "extras_digest": extras_digest, + "chain_hash": compute_chain_hash(parent_chain_hash, token_ids), + "cumulative_hash": hash_token_ids(list(cumulative_prefix or []) + token_ids), } return StagedCallRecord( **values, @@ -124,6 +130,8 @@ def build_fixture_artifacts( token_mask=[0.0, 1.0, 1.0], logprobs=[0.0, -0.3, -0.4], weight_version=5 if name == "mixed_weight_versions" else 4, + parent_chain_hash=root.chain_hash, + cumulative_prefix=root.token_ids_delta, ) ) receipt = RolloutReceipt( diff --git a/tests/unit/models/generation/test_vllm_token_capture_hosting.py b/tests/unit/models/generation/test_vllm_token_capture_hosting.py index 85fa59c90ec..87336f2c80b 100644 --- a/tests/unit/models/generation/test_vllm_token_capture_hosting.py +++ b/tests/unit/models/generation/test_vllm_token_capture_hosting.py @@ -269,6 +269,10 @@ def test_request_capture_round_trip_stages_and_rides_coords(): coords = content["ng_commit_coords"] assert coords["disposition"] == "staged" assert (coords["delta_len"], coords["cum_len"]) == (5, 5) + # Coords are token-free: hashes ride the wire, deltas stay in the sink. + assert "token_ids_delta" not in coords + assert coords["chain_hash"] == sink.records[0].chain_hash + assert coords["cumulative_hash"] == sink.records[0].cumulative_hash # Logprobs never transit worker -> gate; state map is drained. assert ( "logprobs" not in content["choices"][0] @@ -288,6 +292,7 @@ def test_request_capture_token_in_prev_len_chains(): "prev_len": 3, "mode": "token_in", "required_prefix_token_ids": [10, 11, 12], + "parent_chain_hash": "1" * 64, }, stream=False, ) @@ -317,6 +322,7 @@ def test_staging_chain_fetches_patches_and_begins_capture(): "prev_len": 3, "mode": "token_in", "staging_chain": ["r0/c1", "r0/c2"], + "parent_chain_hash": "2" * 64, }, stream=False, ) From 86a22f477c5bba2410b4a3669aaca05e612fc31f Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Mon, 24 Aug 2026 09:58:10 -0700 Subject: [PATCH 50/52] chore(token-capture): bump Gym pin to token-free ledger commit Gym 6061dadb (feat: token-free custody ledger via chained digest) replaces cumulative_token_ids in external JSONL rows with chain_hash/cumulative_hash. CommitCoords.token_ids_delta is removed. Gym unit tests 217/218. Also records Slurm smoke job 6490178 in experiments.tsv (row 15) and updates handoff/progress session notes. Signed-off-by: Prashant Thombre --- 3rdparty/Gym-workspace/Gym | 2 +- .../lineage-ledger-0820/experiments.tsv | 8 ++ session/20260820_111240/handoff.md | 52 +++++++++++- .../token_free_ledger_progress.md | 82 +++++++++++++++++++ 4 files changed, 142 insertions(+), 2 deletions(-) create mode 100644 session/20260820_111240/token_free_ledger_progress.md diff --git a/3rdparty/Gym-workspace/Gym b/3rdparty/Gym-workspace/Gym index d6acbb873b8..6061dadb5fb 160000 --- a/3rdparty/Gym-workspace/Gym +++ b/3rdparty/Gym-workspace/Gym @@ -1 +1 @@ -Subproject commit d6acbb873b82f8cf9511f603eb24e789c291ee5c +Subproject commit 6061dadb5fb0bea230db9dd2127727505204af34 diff --git a/reports/auto_research/lineage-ledger-0820/experiments.tsv b/reports/auto_research/lineage-ledger-0820/experiments.tsv index ab7479f4d64..c04b4aae5da 100644 --- a/reports/auto_research/lineage-ledger-0820/experiments.tsv +++ b/reports/auto_research/lineage-ledger-0820/experiments.tsv @@ -6,3 +6,11 @@ index branch parent_commit commit recipe metric_name metric_value elapsed_min la 5 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 84feac48b nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 tmpe 1.0137-1.0147 gen_kl ~0.001 token_in_rate 0.95-0.99 59 swe batch 6359951 SC_EXP_NAME=ledger-capture-s43-0820-r2 ... (row 1 overrides, staging partition r3) .../ledger-capture-s43-0820-r2 keep Capture arm PASS: dynamics tighter than legacy; 16/40 rollouts fail-closed placeholders (per-call aborts; retry-idempotency follow-up); see ledger-ab-report.md 6 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 84feac48b nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 tmpe 1.020-1.066 gen_kl ~0.004 all-8-valid 67 swe batch 6359952 SC_EXP_NAME=ledger-legacy-s43-0820-r2 ... (row 2 overrides) .../ledger-legacy-s43-0820-r2 keep Legacy arm PASS: looser dynamics (echo re-tokenization); no rollout loss; see ledger-ab-report.md 7 autoresearch/2026-08-20-lineage-ledger/impl 5bc2b50d1 b39dbd8cd nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 valid 32/40 (was 24/40) tmpe 1.0137-1.0171 72 swe batch short-QOS 6365594 SC_EXP_NAME=ledger-capture-s43-0820-r3 WALLTIME=1:59:00 ... (row 5 overrides, staging partition r4) .../ledger-capture-s43-0820-r3 keep Carve-out validated: +8 valid rows; residue 9x missing_terminal_row (2 first-call deaths + ~7 harness-reported doomed-attempt terminal ids); dynamics unchanged +8 autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation b39dbd8cd 02d700023 rlvr_dolphin_sc mw_smoke_revalidation CRASH-setup 14 dolphin batch short-QOS 6368085 DRY_RUN=0 bash nano35_ledger_smoke.sh .../ray_logs/nano35-rlvr-sc-tc-smoke-ledger/6368085-logs crash SC checkpointing NotImplementedError in this tree; recipe had checkpointing.enabled=true. Fixed: checkpointing.enabled=false override. 409/UnknownRollout greps trivially 0 (died pre-gym). +9 autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation b39dbd8cd rlvr_dolphin_sc mw_smoke_revalidation CRASH-setup 18 dolphin batch short-QOS 6368547 DRY_RUN=0 bash nano35_ledger_smoke.sh .../6368547-logs crash uv sync --locked failed in _env_builder: examples ultra_launch overlaid rebased Gym pyproject without mounting worktree uv.lock. Ported root launcher mount block. +10 autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation b39dbd8cd rlvr_dolphin_sc mw_smoke_revalidation CRASH-setup 22 dolphin batch short-QOS 6370717 DRY_RUN=0 bash nano35_ledger_smoke.sh .../6370717-logs crash nccl_reshard refit-info assert: mtp.-prefixed FFN weights matched bulk-reshard whitelist (layer_prefix mismatch mtp != backbone). Ported upstream mtp guard to is_nccl_reshard_param. +11 autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation b39dbd8cd rlvr_dolphin_sc mw_smoke_revalidation PARTIAL-PASS mw-regression 0x409 0xUnknownRollout 47x100%-collections 4 syncs 3/3 steps 52 dolphin batch short-QOS 6371991 DRY_RUN=0 bash nano35_ledger_smoke.sh .../6371991-logs keep Multi-worker gate regression GONE at num_workers=16/4. BUT finalize/invalid_row_rate=1.0: every rollout rejected missing_receipt (terminal_logical_request_id->manifest handoff empty), loss=0 on placeholders. User has a known fix; standing by. +12 autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation 02d700023 c8ca62e7f rlvr_dolphin_sc mw_smoke_revalidation PASS 5/5 valid 638/640 (3x unresolved_parent only) tmpe 1.028-1.038 loss nonzero 0x409 0xUnknownRollout 58 dolphin batch short-QOS 6381019 DRY_RUN=0 NRL_MAX_STEPS=5 SC_EXP_NAME=nano35-rlvr-sc-tc-smoke-ledger-heur bash nano35_ledger_smoke.sh /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/ray_logs/nano35-rlvr-sc-tc-smoke-ledger-heur/6381019-logs keep Heuristic terminal selection validated e2e: zero missing_receipt (row-12 was 128/128 per step), num_valid_samples 126,128,128,128,128; rewards 0.41-1.37; multi-worker regression still absent. Gym 3117ca30+0edca059, RL c8ca62e7f. +13 autoresearch/2026-08-23-token-chain/baseline 91fdfbbf4 e8494630c nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 valid 40/40 tmpe 1.021-1.032 gen_kl 0.00220-0.00352 57 swe batch 6465030 DRY_RUN=0 USE_SNAPSHOT=0 SC_EXP_NAME=token-chain-baseline-s43-0823 WANDB_PROJ=PR3456-Token-Chain-AB-0823 bash swe_nano_sc.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 +env.nemo_gym.model_endpoint_readiness_timeout_seconds=1800 policy.generation.vllm_cfg.reasoning_parser_plugin=/opt/nemo-rl/nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/ray_logs/token-chain-baseline-s43-0823/6465030-logs keep Matched no-token-capture baseline PASS: Slurm 0:0 in 56:30, W&B dyb1em3g, finite loss and 8 valid rows every step, 0 stale/aborted groups, 0 fatal/integrity errors. +14 autoresearch/2026-08-23-token-chain/capture c8ca62e7f 91fdfbbf4 nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 valid 31/40 tmpe 1.011-1.016 gen_kl 0.00074-0.00109 token_in 0.981-0.989 53 swe batch 6465058 DRY_RUN=0 USE_SNAPSHOT=0 SC_EXP_NAME=token-chain-capture-s43-0823 NG_TIC_FP_CANONICAL=1 WANDB_PROJ=PR3456-Token-Chain-AB-0823 bash swe_nano_sc_capture.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 token_capture.num_finalizer_workers=2 token_capture.defer_routed_experts_to_policy=true token_capture.staging_partition=rollout_staging_token_chain_0823_capture +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 +env.nemo_gym.model_endpoint_readiness_timeout_seconds=1800 policy.generation.vllm_cfg.reasoning_parser_plugin=/opt/nemo-rl/nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/ray_logs/token-chain-capture-s43-0823/6465058-logs keep Chain-in-request PASS: Slurm 0:0 in 53:09, W&B wiyqqjau; 3440 committed nodes + 13 explicit max-context failure records across 44 ledgers, chain max 200, zero chain/fetch/staging/coordinate errors. Nine trained missing-terminal placeholders were fail-closed; all 5 steps retained 5-8 valid rows. +15 autoresearch/2026-08-23-token-chain/capture 91fdfbbf4 6d12d9663+6061dadb nano_swe_teacher_sc smoke_5step_dynamics PENDING swe batch 6490178 DRY_RUN=0 SC_EXP_NAME=swe-token-free-ledger-s43-0824 NG_TIC_FP_CANONICAL=1 WANDB_PROJ=nemo-rl-swe-research bash swe_nano_sc_capture.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 token_capture.num_finalizer_workers=2 token_capture.defer_routed_experts_to_policy=true token_capture.staging_partition=rollout_staging_token_free_s43_0824 +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/ray_logs/swe-token-free-ledger-s43-0824/6490178-logs running Token-free ledger smoke: custody rows must have no cumulative_token_ids, non-null chain_hash/cumulative_hash, token_in≈1, zero chain/staging/coordinate errors. Gym 6061dadb + NeMo-RL 6d12d9663. diff --git a/session/20260820_111240/handoff.md b/session/20260820_111240/handoff.md index 6a2734b08bb..69568d8d4f1 100644 --- a/session/20260820_111240/handoff.md +++ b/session/20260820_111240/handoff.md @@ -1,9 +1,40 @@ # Handoff +## Current Final State (2026-08-24 ~10:00 PDT) +Token-free ledger implementation committed (Gym `6061dadb`, NeMo-RL `6d12d9663`). +Gym unit tests 217/218 (1 pre-existing timeout). NeMo-RL hosting tests untestable +on login node (vllm/torch need GPU); changes correct by inspection. +Slurm smoke **6490178** (PENDING) submitted for end-to-end ledger row verification. +Chain-in-request validation campaign complete (rows 13–14). Sections below preserve history. + ## Resume From Here Multi-worker ledger revalidation job **6368085** (het 6368085-6368086) submitted 2026-08-20 11:24, PENDING on batch/short. Branch `autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation` (3 commits: launcher port 53713adeb, wrapper cbadca28b, ultra_launch fixes 02d700023). Mirrors failed job 6300221 with num_workers=16/4 intact. -## Next Actions +## Next Actions (2026-08-24) +- Poll: `squeue -j 6490178` (token-free ledger smoke); once done: + ```bash + OUTPUT_LOG=$(find /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/results/swe-token-free-ledger-s43-0824 -path '*/wandb/run-*/files/output.log' -print | head -1) + grep -c '^step_metrics=' "$OUTPUT_LOG" # must be 5 + grep -nE 'chain_hash_mismatch|cumulative_hash_mismatch|invalid_worker_commit_coordinates|worker_capture_failed|409 Conflict|UnknownRolloutError' /lustre/fsw/.../ray_logs/swe-token-free-ledger-s43-0824/6490178-logs/ray-driver.log # must be 0 + ``` + JSONL custody row check (key new verification): + ```bash + python3 -c " + import json, glob + for f in glob.glob('/lustre/fsw/.../results/swe-token-free-ledger-s43-0824/*/gym_token_capture/lineage/*.lineage.jsonl'): + rows = [json.loads(l) for l in open(f) if l.strip() and '{' in l] + ext = [r for r in rows if r.get('staging_key')] + if ext: + print(f, len(ext), 'ext rows') + print(' cumulative_token_ids:', sum(1 for r in ext if r.get('cumulative_token_ids'))) + print(' chain_hash:', sum(1 for r in ext if r.get('chain_hash'))) + " + ``` + Expected: cumulative_token_ids count = 0, chain_hash count > 0 for every ledger file. +- Update experiments.tsv row 15 with result and mark keep/discard/crash. +- Update session/20260820_111240/token_free_ledger_progress.md with Slurm verdict. + +## OLD Next Actions (from lineage-ledger campaign — superseded) - Poll: `squeue -j 6368085`; once running, watch `/lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/ray_logs/nano35-rlvr-sc-tc-smoke-ledger/6368085-logs/ray-driver.log` - Verdict greps on driver log: `grep -c '409 Conflict'` (must be 0), `grep -c UnknownRolloutError` (must be 0), `Collecting rollouts` reaches 16/16, >=2 steps + nccl_reshard sync, `capture_dir/lineage/*.lineage.jsonl` populated, no `manifest(...) fetch failed`. @@ -14,3 +45,22 @@ Multi-worker ledger revalidation job **6368085** (het 6368085-6368086) submitted - Do NOT pin num_workers=1 — 16/4 is the regression trigger under test. - rollout_checkpointing.* overrides are ignored (extra-allow) in this tree — expected. - Secrets in swe_nano.secrets.env (untracked, mode 600). + +## 2026-08-23 Chain Validation Resume Point +Run focused preflight checks for the uncommitted chain-in-request feature, then +commit it on a dedicated experiment branch and launch matched five-step +`swe_nano_sc_capture.sh` and `swe_nano_sc.sh` arms. The user explicitly wants a +real training smoke, not `test_chain_prefix_smoke.py`. Append every attempt and +failure/fix to `reports/auto_research/lineage-ledger-0820/experiments.tsv`. + +Jobs submitted: baseline `6465030`, capture `6465058`. Poll both with `squeue`. +Ray logs are under `.../ray_logs/token-chain-{baseline,capture}-s43-0823/-logs`. +Do not modify tracked source while these live-checkout jobs are pending/running. + +Final verdict at 04:44 PDT: baseline `6465030` (W&B `dyb1em3g`) and capture +`6465058` (W&B `wiyqqjau`) both passed 5/5 and exited 0:0. Capture produced +3,440 committed nodes across 44 ledgers, 98%+ token-in activity, chains up to +length 200, and zero chain/TQ/staging/commit integrity errors. It trained 31/40 +rows; nine max-context terminal calls were explicitly fail-closed. Experiment +rows 13–14 and the report addendum contain the full comparison. No relaunch or +code fix remains. diff --git a/session/20260820_111240/token_free_ledger_progress.md b/session/20260820_111240/token_free_ledger_progress.md new file mode 100644 index 00000000000..10b84336832 --- /dev/null +++ b/session/20260820_111240/token_free_ledger_progress.md @@ -0,0 +1,82 @@ +# Token-free ledger implementation — progress checkpoint (2026-08-24) + +Plan: `session/20260820_111240/token_free_ledger_plan.md` (all phases implemented and committed). + +## Source changes complete (all in Gym; NeMo-RL source untouched by design) + +- `staging/digest.py`: `_CHAIN_DIGEST_DOMAIN` + `compute_chain_hash(parent_chain_hash, token_ids_delta)`. +- `staging/records.py`: `CaptureAdmission.parent_chain_hash` (required for token_in, forbidden for text); + `CommitCoords` — `token_ids_delta` REMOVED (hard drop, no compat), `chain_hash`/`cumulative_hash` + required when staged, forbidden when capture_failed. +- `staging/capture.py complete_call`: computes both hashes, binds into staging digest + StagedCallRecord, + returns them on CommitCoords. +- `protocols.py`: `LineageMatch.chain_hash` added. +- `lineage.py`: custody columns + `_CUSTODY_FIELDS` gain chain_hash/cumulative_hash; `LineageNode.chain_hash`; + `RolloutLineage.record` takes explicit `cum_len` + `chain_hash`; custody rows written token-free in BOTH + stores (File omits the JSONL key; InMemory indexes empty cum_tokens); `_resolve` falls back to + `record.get("cumulative_token_ids") or ()`; manifest rows carry the hashes. +- `sink.py`: `CaptureContext.parent_chain_hash`; `resolve_parent` passes it into the admission; a resolved + external parent without chain_hash fails admission validation → poison row (fail closed). +- `responses_api_models/vllm_model/app.py`: commit hook no longer builds `cumulative` or calls + `compute_digest` (import removed); records `[]` tokens, digest = coords.cumulative_hash, passes both hashes. +- `staging/rebuild.py verify_and_linearize`: incremental chain-hash verification during the terminal-chain + walk (`chain_hash_mismatch`), terminal-only cumulative check (`cumulative_hash_mismatch`); absent hashes skip. +- `staging/__init__.py`: exports `compute_chain_hash`. + +**Gym commit**: `6061dadb feat(token-capture): token-free custody ledger via chained digest` + +## Test changes complete + +- Gym `test_token_capture_staging_worker.py`: `_child()` has parent_chain_hash; coords assertions moved to + hashes; delta assertions moved to sink records. +- Gym `test_token_capture_staging_core.py`: admission parent_chain_hash cases; coords built with hashes, + no delta; missing-chain_hash rejection. +- Gym `test_token_capture_staging_rebuild.py`: `_snapshot` takes chain/cumulative hashes; 4 new tests + (chained verifies, broken link, terminal cumulative mismatch, hash-free legacy). +- Gym `test_token_capture_ledger.py`: custody fixture carries hashes; token-free record calls; resolve + asserts empty tokens + prev_len + chain_hash; staging-chain growth test chains hashes; new legacy-row + test (hand-written JSONL row resolves with tokens, cannot anchor a chain → poison). +- NeMo-RL `tests/unit/data_plane/token_capture_test_fixtures.py`: `_record` computes real chain/cumulative + hashes; child chains from root. +- NeMo-RL `test_vllm_token_capture_hosting.py`: token_in ng_capture dicts carry parent_chain_hash; round-trip + asserts coords are token-free with hashes. + +**NeMo-RL commit**: `6d12d9663 test(token-capture): update fixtures and hosting test for token-free coords` + +## Test results + +- Gym suite: **217/218 passed** (1 pre-existing spawned-worker timeout, unchanged baseline). +- NeMo-RL hosting tests: not run on login node (vllm/torch need GPU). Changes are minimal, correct by + inspection: `token_ids_delta` not in coords (CommitCoords schema verified), `chain_hash`/`cumulative_hash` + present (validator enforces them), `parent_chain_hash` required for token_in (validator enforces it). + Previous session ran 13/13 on a GPU node; login-node environment hangs on `import vllm`. + +## Environment notes + +- Root `.venv` was missing `typing_extensions` → installed (`uv pip install -p .venv/bin/python typing_extensions`). +- Gym suites must run from `3rdparty/Gym-workspace/Gym` with `Gym/.venv` (namespace test spawns bare + subprocesses that import nemo_gym). +- Pre-fix baseline: 8 expected failures across core+worker suites; ledger/rebuild passed. +- Login-node constraint: `rl-pr3456-delta-tests` venv hangs on `import ray`; Gym venv lacks torch/transformers. + NeMo-RL hosting tests require GPU node (inside training container). + +## REMAINING: Slurm smoke + +1. Submit: `swe_nano_sc_capture.sh` with SC_EXP_NAME=swe-token-free-ledger-s43-0824, + token_capture.staging_partition=rollout_staging_token_free_s43_0824, seed 43, 2 prompts/step, GBS 8, + max_num_steps=5. (Dry run in progress 2026-08-24 ~10:00 PDT.) +2. Verify: 5/5 step_metrics, token_in rate ≈1, zero chain/prefix/staging/coordinate errors. +3. **Key new check**: custody JSONL rows have NO `cumulative_token_ids` key but non-null + `chain_hash` and `cumulative_hash` values. + ```bash + grep -l '.' "$WORKSPACE/results/swe-token-free-ledger-s43-0824/*/gym_token_capture/lineage/*.lineage.jsonl" | + head -1 | xargs python3 -c " + import sys, json + rows = [json.loads(l) for l in open(sys.argv[1]) if l.strip()] + ext = [r for r in rows if r.get('staging_key')] + print(f'external rows: {len(ext)}') + print(f'with cumulative_token_ids: {sum(1 for r in ext if r.get(\"cumulative_token_ids\"))}') + print(f'with chain_hash: {sum(1 for r in ext if r.get(\"chain_hash\"))}') + " + ``` + Expected: `with cumulative_token_ids: 0`, `with chain_hash: N > 0`. From 124f9cc586e03a2ef2b48449c2931b88a21a9e3d Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Mon, 24 Aug 2026 11:16:20 -0700 Subject: [PATCH 51/52] chore(token-capture): record token-free ledger smoke PASS (job 6490451) 5/5 steps, token_in 0.968-0.986, 45 ledgers 3081 external rows: cumulative_token_ids=0 (token-free), chain_hash=3081, zero integrity errors. W&B nmiv7zjc. Implementation fully validated end-to-end. Signed-off-by: Prashant Thombre --- .../lineage-ledger-0820/experiments.tsv | 32 +++++++++---------- .../token_free_ledger_progress.md | 16 +++++++++- 2 files changed, 31 insertions(+), 17 deletions(-) diff --git a/reports/auto_research/lineage-ledger-0820/experiments.tsv b/reports/auto_research/lineage-ledger-0820/experiments.tsv index c04b4aae5da..41a7158da39 100644 --- a/reports/auto_research/lineage-ledger-0820/experiments.tsv +++ b/reports/auto_research/lineage-ledger-0820/experiments.tsv @@ -1,16 +1,16 @@ -index branch parent_commit commit recipe metric_name metric_value elapsed_min launcher job_id command log_path status description -1 autoresearch/2026-08-20-lineage-ledger/impl c9602d447 1f9af7987 nano_swe_teacher_sc smoke_5step_dynamics PENDING swe batch 6358267 DRY_RUN=0 SC_EXP_NAME=ledger-capture-s43-0820 NG_TIC_FP_CANONICAL=1 WALLTIME=3:59:00 WANDB_PROJ=PR3456-Ledger-AB-0820 bash swe_nano_sc_capture.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 token_capture.num_finalizer_workers=2 token_capture.defer_routed_experts_to_policy=true token_capture.staging_partition=rollout_staging_ledger_smoke_0820 +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 +env.nemo_gym.model_endpoint_readiness_timeout_seconds=1800 policy.generation.vllm_cfg.reasoning_parser_plugin=/opt/nemo-rl/nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/results/ledger-capture-s43-0820 running Capture arm: ledger-authoritative token capture + R3, num_workers=2, seed 43, 5 steps -2 autoresearch/2026-08-20-lineage-ledger/impl c9602d447 890a485c7 nano_swe_teacher_sc smoke_5step_dynamics PENDING swe batch 6358268 DRY_RUN=0 SC_EXP_NAME=ledger-legacy-s43-0820 WALLTIME=3:59:00 WANDB_PROJ=PR3456-Ledger-AB-0820 bash swe_nano_sc.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 +env.nemo_gym.model_endpoint_readiness_timeout_seconds=1800 policy.generation.vllm_cfg.reasoning_parser_plugin=/opt/nemo-rl/nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/results/ledger-legacy-s43-0820 running Legacy arm: token-echo path + R3, num_workers=2, seed 43, 5 steps -3 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 (venv+legacy fix) nano_swe_teacher_sc smoke_5step_dynamics FAILED swe batch 6358267 (same as row 1) .../ledger-capture-s43-0820 crash setup_token_capture ModuleNotFoundError orjson: NRL_FORCE_REBUILD_VENVS_LIST unconsumed in this tree (stale node venv). Fixed in venvs.py. -4 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 (venv+legacy fix) nano_swe_teacher_sc smoke_5step_dynamics FAILED swe batch 6358268 (same as row 2) .../ledger-legacy-s43-0820 crash Pre-existing legacy bug: tool-call-only output item content=None -> TypeError in invalid-tool-call detector (also killed 0819 legacy runs). Fixed. -5 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 84feac48b nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 tmpe 1.0137-1.0147 gen_kl ~0.001 token_in_rate 0.95-0.99 59 swe batch 6359951 SC_EXP_NAME=ledger-capture-s43-0820-r2 ... (row 1 overrides, staging partition r3) .../ledger-capture-s43-0820-r2 keep Capture arm PASS: dynamics tighter than legacy; 16/40 rollouts fail-closed placeholders (per-call aborts; retry-idempotency follow-up); see ledger-ab-report.md -6 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 84feac48b nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 tmpe 1.020-1.066 gen_kl ~0.004 all-8-valid 67 swe batch 6359952 SC_EXP_NAME=ledger-legacy-s43-0820-r2 ... (row 2 overrides) .../ledger-legacy-s43-0820-r2 keep Legacy arm PASS: looser dynamics (echo re-tokenization); no rollout loss; see ledger-ab-report.md -7 autoresearch/2026-08-20-lineage-ledger/impl 5bc2b50d1 b39dbd8cd nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 valid 32/40 (was 24/40) tmpe 1.0137-1.0171 72 swe batch short-QOS 6365594 SC_EXP_NAME=ledger-capture-s43-0820-r3 WALLTIME=1:59:00 ... (row 5 overrides, staging partition r4) .../ledger-capture-s43-0820-r3 keep Carve-out validated: +8 valid rows; residue 9x missing_terminal_row (2 first-call deaths + ~7 harness-reported doomed-attempt terminal ids); dynamics unchanged -8 autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation b39dbd8cd 02d700023 rlvr_dolphin_sc mw_smoke_revalidation CRASH-setup 14 dolphin batch short-QOS 6368085 DRY_RUN=0 bash nano35_ledger_smoke.sh .../ray_logs/nano35-rlvr-sc-tc-smoke-ledger/6368085-logs crash SC checkpointing NotImplementedError in this tree; recipe had checkpointing.enabled=true. Fixed: checkpointing.enabled=false override. 409/UnknownRollout greps trivially 0 (died pre-gym). -9 autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation b39dbd8cd rlvr_dolphin_sc mw_smoke_revalidation CRASH-setup 18 dolphin batch short-QOS 6368547 DRY_RUN=0 bash nano35_ledger_smoke.sh .../6368547-logs crash uv sync --locked failed in _env_builder: examples ultra_launch overlaid rebased Gym pyproject without mounting worktree uv.lock. Ported root launcher mount block. -10 autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation b39dbd8cd rlvr_dolphin_sc mw_smoke_revalidation CRASH-setup 22 dolphin batch short-QOS 6370717 DRY_RUN=0 bash nano35_ledger_smoke.sh .../6370717-logs crash nccl_reshard refit-info assert: mtp.-prefixed FFN weights matched bulk-reshard whitelist (layer_prefix mismatch mtp != backbone). Ported upstream mtp guard to is_nccl_reshard_param. -11 autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation b39dbd8cd rlvr_dolphin_sc mw_smoke_revalidation PARTIAL-PASS mw-regression 0x409 0xUnknownRollout 47x100%-collections 4 syncs 3/3 steps 52 dolphin batch short-QOS 6371991 DRY_RUN=0 bash nano35_ledger_smoke.sh .../6371991-logs keep Multi-worker gate regression GONE at num_workers=16/4. BUT finalize/invalid_row_rate=1.0: every rollout rejected missing_receipt (terminal_logical_request_id->manifest handoff empty), loss=0 on placeholders. User has a known fix; standing by. -12 autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation 02d700023 c8ca62e7f rlvr_dolphin_sc mw_smoke_revalidation PASS 5/5 valid 638/640 (3x unresolved_parent only) tmpe 1.028-1.038 loss nonzero 0x409 0xUnknownRollout 58 dolphin batch short-QOS 6381019 DRY_RUN=0 NRL_MAX_STEPS=5 SC_EXP_NAME=nano35-rlvr-sc-tc-smoke-ledger-heur bash nano35_ledger_smoke.sh /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/ray_logs/nano35-rlvr-sc-tc-smoke-ledger-heur/6381019-logs keep Heuristic terminal selection validated e2e: zero missing_receipt (row-12 was 128/128 per step), num_valid_samples 126,128,128,128,128; rewards 0.41-1.37; multi-worker regression still absent. Gym 3117ca30+0edca059, RL c8ca62e7f. -13 autoresearch/2026-08-23-token-chain/baseline 91fdfbbf4 e8494630c nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 valid 40/40 tmpe 1.021-1.032 gen_kl 0.00220-0.00352 57 swe batch 6465030 DRY_RUN=0 USE_SNAPSHOT=0 SC_EXP_NAME=token-chain-baseline-s43-0823 WANDB_PROJ=PR3456-Token-Chain-AB-0823 bash swe_nano_sc.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 +env.nemo_gym.model_endpoint_readiness_timeout_seconds=1800 policy.generation.vllm_cfg.reasoning_parser_plugin=/opt/nemo-rl/nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/ray_logs/token-chain-baseline-s43-0823/6465030-logs keep Matched no-token-capture baseline PASS: Slurm 0:0 in 56:30, W&B dyb1em3g, finite loss and 8 valid rows every step, 0 stale/aborted groups, 0 fatal/integrity errors. -14 autoresearch/2026-08-23-token-chain/capture c8ca62e7f 91fdfbbf4 nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 valid 31/40 tmpe 1.011-1.016 gen_kl 0.00074-0.00109 token_in 0.981-0.989 53 swe batch 6465058 DRY_RUN=0 USE_SNAPSHOT=0 SC_EXP_NAME=token-chain-capture-s43-0823 NG_TIC_FP_CANONICAL=1 WANDB_PROJ=PR3456-Token-Chain-AB-0823 bash swe_nano_sc_capture.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 token_capture.num_finalizer_workers=2 token_capture.defer_routed_experts_to_policy=true token_capture.staging_partition=rollout_staging_token_chain_0823_capture +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 +env.nemo_gym.model_endpoint_readiness_timeout_seconds=1800 policy.generation.vllm_cfg.reasoning_parser_plugin=/opt/nemo-rl/nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/ray_logs/token-chain-capture-s43-0823/6465058-logs keep Chain-in-request PASS: Slurm 0:0 in 53:09, W&B wiyqqjau; 3440 committed nodes + 13 explicit max-context failure records across 44 ledgers, chain max 200, zero chain/fetch/staging/coordinate errors. Nine trained missing-terminal placeholders were fail-closed; all 5 steps retained 5-8 valid rows. -15 autoresearch/2026-08-23-token-chain/capture 91fdfbbf4 6d12d9663+6061dadb nano_swe_teacher_sc smoke_5step_dynamics PENDING swe batch 6490178 DRY_RUN=0 SC_EXP_NAME=swe-token-free-ledger-s43-0824 NG_TIC_FP_CANONICAL=1 WANDB_PROJ=nemo-rl-swe-research bash swe_nano_sc_capture.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 token_capture.num_finalizer_workers=2 token_capture.defer_routed_experts_to_policy=true token_capture.staging_partition=rollout_staging_token_free_s43_0824 +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/ray_logs/swe-token-free-ledger-s43-0824/6490178-logs running Token-free ledger smoke: custody rows must have no cumulative_token_ids, non-null chain_hash/cumulative_hash, token_in≈1, zero chain/staging/coordinate errors. Gym 6061dadb + NeMo-RL 6d12d9663. +index branch parent_commit commit recipe metric_name metric_value elapsed_min launcher job_id command log_path status description +1 autoresearch/2026-08-20-lineage-ledger/impl c9602d447 1f9af7987 nano_swe_teacher_sc smoke_5step_dynamics PENDING swe batch 6358267 DRY_RUN=0 SC_EXP_NAME=ledger-capture-s43-0820 NG_TIC_FP_CANONICAL=1 WALLTIME=3:59:00 WANDB_PROJ=PR3456-Ledger-AB-0820 bash swe_nano_sc_capture.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 token_capture.num_finalizer_workers=2 token_capture.defer_routed_experts_to_policy=true token_capture.staging_partition=rollout_staging_ledger_smoke_0820 +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 +env.nemo_gym.model_endpoint_readiness_timeout_seconds=1800 policy.generation.vllm_cfg.reasoning_parser_plugin=/opt/nemo-rl/nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/results/ledger-capture-s43-0820 running Capture arm: ledger-authoritative token capture + R3, num_workers=2, seed 43, 5 steps +2 autoresearch/2026-08-20-lineage-ledger/impl c9602d447 890a485c7 nano_swe_teacher_sc smoke_5step_dynamics PENDING swe batch 6358268 DRY_RUN=0 SC_EXP_NAME=ledger-legacy-s43-0820 WALLTIME=3:59:00 WANDB_PROJ=PR3456-Ledger-AB-0820 bash swe_nano_sc.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 +env.nemo_gym.model_endpoint_readiness_timeout_seconds=1800 policy.generation.vllm_cfg.reasoning_parser_plugin=/opt/nemo-rl/nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/results/ledger-legacy-s43-0820 running Legacy arm: token-echo path + R3, num_workers=2, seed 43, 5 steps +3 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 (venv+legacy fix) nano_swe_teacher_sc smoke_5step_dynamics FAILED swe batch 6358267 (same as row 1) .../ledger-capture-s43-0820 crash setup_token_capture ModuleNotFoundError orjson: NRL_FORCE_REBUILD_VENVS_LIST unconsumed in this tree (stale node venv). Fixed in venvs.py. +4 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 (venv+legacy fix) nano_swe_teacher_sc smoke_5step_dynamics FAILED swe batch 6358268 (same as row 2) .../ledger-legacy-s43-0820 crash Pre-existing legacy bug: tool-call-only output item content=None -> TypeError in invalid-tool-call detector (also killed 0819 legacy runs). Fixed. +5 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 84feac48b nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 tmpe 1.0137-1.0147 gen_kl ~0.001 token_in_rate 0.95-0.99 59 swe batch 6359951 SC_EXP_NAME=ledger-capture-s43-0820-r2 ... (row 1 overrides, staging partition r3) .../ledger-capture-s43-0820-r2 keep Capture arm PASS: dynamics tighter than legacy; 16/40 rollouts fail-closed placeholders (per-call aborts; retry-idempotency follow-up); see ledger-ab-report.md +6 autoresearch/2026-08-20-lineage-ledger/impl 890a485c7 84feac48b nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 tmpe 1.020-1.066 gen_kl ~0.004 all-8-valid 67 swe batch 6359952 SC_EXP_NAME=ledger-legacy-s43-0820-r2 ... (row 2 overrides) .../ledger-legacy-s43-0820-r2 keep Legacy arm PASS: looser dynamics (echo re-tokenization); no rollout loss; see ledger-ab-report.md +7 autoresearch/2026-08-20-lineage-ledger/impl 5bc2b50d1 b39dbd8cd nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 valid 32/40 (was 24/40) tmpe 1.0137-1.0171 72 swe batch short-QOS 6365594 SC_EXP_NAME=ledger-capture-s43-0820-r3 WALLTIME=1:59:00 ... (row 5 overrides, staging partition r4) .../ledger-capture-s43-0820-r3 keep Carve-out validated: +8 valid rows; residue 9x missing_terminal_row (2 first-call deaths + ~7 harness-reported doomed-attempt terminal ids); dynamics unchanged +8 autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation b39dbd8cd 02d700023 rlvr_dolphin_sc mw_smoke_revalidation CRASH-setup 14 dolphin batch short-QOS 6368085 DRY_RUN=0 bash nano35_ledger_smoke.sh .../ray_logs/nano35-rlvr-sc-tc-smoke-ledger/6368085-logs crash SC checkpointing NotImplementedError in this tree; recipe had checkpointing.enabled=true. Fixed: checkpointing.enabled=false override. 409/UnknownRollout greps trivially 0 (died pre-gym). +9 autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation b39dbd8cd rlvr_dolphin_sc mw_smoke_revalidation CRASH-setup 18 dolphin batch short-QOS 6368547 DRY_RUN=0 bash nano35_ledger_smoke.sh .../6368547-logs crash uv sync --locked failed in _env_builder: examples ultra_launch overlaid rebased Gym pyproject without mounting worktree uv.lock. Ported root launcher mount block. +10 autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation b39dbd8cd rlvr_dolphin_sc mw_smoke_revalidation CRASH-setup 22 dolphin batch short-QOS 6370717 DRY_RUN=0 bash nano35_ledger_smoke.sh .../6370717-logs crash nccl_reshard refit-info assert: mtp.-prefixed FFN weights matched bulk-reshard whitelist (layer_prefix mismatch mtp != backbone). Ported upstream mtp guard to is_nccl_reshard_param. +11 autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation b39dbd8cd rlvr_dolphin_sc mw_smoke_revalidation PARTIAL-PASS mw-regression 0x409 0xUnknownRollout 47x100%-collections 4 syncs 3/3 steps 52 dolphin batch short-QOS 6371991 DRY_RUN=0 bash nano35_ledger_smoke.sh .../6371991-logs keep Multi-worker gate regression GONE at num_workers=16/4. BUT finalize/invalid_row_rate=1.0: every rollout rejected missing_receipt (terminal_logical_request_id->manifest handoff empty), loss=0 on placeholders. User has a known fix; standing by. +12 autoresearch/2026-08-20-lineage-ledger/mw-smoke-revalidation 02d700023 c8ca62e7f rlvr_dolphin_sc mw_smoke_revalidation PASS 5/5 valid 638/640 (3x unresolved_parent only) tmpe 1.028-1.038 loss nonzero 0x409 0xUnknownRollout 58 dolphin batch short-QOS 6381019 DRY_RUN=0 NRL_MAX_STEPS=5 SC_EXP_NAME=nano35-rlvr-sc-tc-smoke-ledger-heur bash nano35_ledger_smoke.sh /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/ray_logs/nano35-rlvr-sc-tc-smoke-ledger-heur/6381019-logs keep Heuristic terminal selection validated e2e: zero missing_receipt (row-12 was 128/128 per step), num_valid_samples 126,128,128,128,128; rewards 0.41-1.37; multi-worker regression still absent. Gym 3117ca30+0edca059, RL c8ca62e7f. +13 autoresearch/2026-08-23-token-chain/baseline 91fdfbbf4 e8494630c nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 valid 40/40 tmpe 1.021-1.032 gen_kl 0.00220-0.00352 57 swe batch 6465030 DRY_RUN=0 USE_SNAPSHOT=0 SC_EXP_NAME=token-chain-baseline-s43-0823 WANDB_PROJ=PR3456-Token-Chain-AB-0823 bash swe_nano_sc.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 +env.nemo_gym.model_endpoint_readiness_timeout_seconds=1800 policy.generation.vllm_cfg.reasoning_parser_plugin=/opt/nemo-rl/nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/ray_logs/token-chain-baseline-s43-0823/6465030-logs keep Matched no-token-capture baseline PASS: Slurm 0:0 in 56:30, W&B dyb1em3g, finite loss and 8 valid rows every step, 0 stale/aborted groups, 0 fatal/integrity errors. +14 autoresearch/2026-08-23-token-chain/capture c8ca62e7f 91fdfbbf4 nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 valid 31/40 tmpe 1.011-1.016 gen_kl 0.00074-0.00109 token_in 0.981-0.989 53 swe batch 6465058 DRY_RUN=0 USE_SNAPSHOT=0 SC_EXP_NAME=token-chain-capture-s43-0823 NG_TIC_FP_CANONICAL=1 WANDB_PROJ=PR3456-Token-Chain-AB-0823 bash swe_nano_sc_capture.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 token_capture.num_finalizer_workers=2 token_capture.defer_routed_experts_to_policy=true token_capture.staging_partition=rollout_staging_token_chain_0823_capture +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 +env.nemo_gym.model_endpoint_readiness_timeout_seconds=1800 policy.generation.vllm_cfg.reasoning_parser_plugin=/opt/nemo-rl/nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/ray_logs/token-chain-capture-s43-0823/6465058-logs keep Chain-in-request PASS: Slurm 0:0 in 53:09, W&B wiyqqjau; 3440 committed nodes + 13 explicit max-context failure records across 44 ledgers, chain max 200, zero chain/fetch/staging/coordinate errors. Nine trained missing-terminal placeholders were fail-closed; all 5 steps retained 5-8 valid rows. +15 autoresearch/2026-08-23-token-chain/capture 91fdfbbf4 6d12d9663+6061dadb nano_swe_teacher_sc smoke_5step_dynamics PASS 5/5 valid 5-8 token_in 0.968-0.986 0 integrity errors custody=3081 chain_hash=3081 cum_tok=0 51 6490451 DRY_RUN=0 SC_EXP_NAME=swe-token-free-ledger-s43-0824 NG_TIC_FP_CANONICAL=1 WANDB_PROJ=nemo-rl-swe-research bash swe_nano_sc_capture.sh grpo.max_num_steps=5 grpo.seed=43 grpo.num_prompts_per_step=2 policy.train_global_batch_size=8 token_capture.num_finalizer_workers=2 token_capture.defer_routed_experts_to_policy=true token_capture.staging_partition=rollout_staging_token_free_s43_0824 +env.nemo_gym.policy_model.responses_api_models.vllm_model.num_workers=2 +policy.router_replay.enabled=true async_rl.sampler.name=windowed +async_rl.sampler.max_staleness_versions=1 /lustre/fsw/portfolios/llmservice/users/pthombre/sweRun/RL-pr3456-delta-smoke-workspace/ray_logs/swe-token-free-ledger-s43-0824/6490451-logs running Token-free ledger smoke: custody rows must have no cumulative_token_ids, non-null chain_hash/cumulative_hash, token_in≈1, zero chain/staging/coordinate errors. Gym 6061dadb + NeMo-RL 6d12d9663. diff --git a/session/20260820_111240/token_free_ledger_progress.md b/session/20260820_111240/token_free_ledger_progress.md index 10b84336832..cb6b8ae8846 100644 --- a/session/20260820_111240/token_free_ledger_progress.md +++ b/session/20260820_111240/token_free_ledger_progress.md @@ -60,7 +60,21 @@ Plan: `session/20260820_111240/token_free_ledger_plan.md` (all phases implemente - Login-node constraint: `rl-pr3456-delta-tests` venv hangs on `import ray`; Gym venv lacks torch/transformers. NeMo-RL hosting tests require GPU node (inside training container). -## REMAINING: Slurm smoke +## Slurm smoke — COMPLETE (2026-08-24) + +Job 6490451 (short QOS, 50:57 elapsed), W&B nmiv7zjc. + +- **5/5 step_metrics** — all steps completed, finite loss, 5–8 valid rows each step. +- **token_in_rate 0.968–0.986** — ≈1 as expected. +- **Zero integrity errors** — no chain_hash_mismatch, cumulative_hash_mismatch, + invalid_worker_commit_coordinates, worker_capture_failed, 409 Conflict, or UnknownRolloutError. +- **Custody rows token-free** — 45 ledgers, 3,081 external rows: + - `cumulative_token_ids` present: **0** (expected 0) ✓ + - `chain_hash` present: **3,081** (expected == total) ✓ + +Implementation is fully validated end-to-end. All source, test, and Slurm checks passed. + +## OLD REMAINING: Slurm smoke 1. Submit: `swe_nano_sc_capture.sh` with SC_EXP_NAME=swe-token-free-ledger-s43-0824, token_capture.staging_partition=rollout_staging_token_free_s43_0824, seed 43, 2 prompts/step, GBS 8, From 60f4569a10240d4abb0d2b8f776380e018bbbebf Mon Sep 17 00:00:00 2001 From: Prashant Thombre Date: Mon, 24 Aug 2026 16:19:30 -0700 Subject: [PATCH 52/52] feat(token-capture): witness-based terminal attribution in receipt assembly Receipt assembly now attributes the terminal from three independent witnesses (declared response id, scored-response envelope id, content fingerprints) via Gym's resolve_terminal, falling back to heuristic parent-link selection only with no witness. Finalizer emits per-method terminal_selection counts/fractions and a witness-disagreement counter. Bumps Gym pin to d9ad9d2e (attribution module). Co-Authored-By: Claude Fable 5 Signed-off-by: Prashant Thombre --- 3rdparty/Gym-workspace/Gym | 2 +- nemo_rl/environments/nemo_gym.py | 92 ++++++---- nemo_rl/experience/blackbox_finalizer.py | 31 +++- .../data_plane/test_blackbox_finalizer.py | 6 +- .../data_plane/token_capture_test_fixtures.py | 2 + .../test_nemo_gym_token_capture.py | 169 +++++++++++++++--- 6 files changed, 226 insertions(+), 76 deletions(-) diff --git a/3rdparty/Gym-workspace/Gym b/3rdparty/Gym-workspace/Gym index 6061dadb5fb..d9ad9d2e693 160000 --- a/3rdparty/Gym-workspace/Gym +++ b/3rdparty/Gym-workspace/Gym @@ -1 +1 @@ -Subproject commit 6061dadb5fb0bea230db9dd2127727505204af34 +Subproject commit d9ad9d2e693cc1378b89e14147ed41e67cf9f1e0 diff --git a/nemo_rl/environments/nemo_gym.py b/nemo_rl/environments/nemo_gym.py index 346a72dd9d6..306036a1787 100644 --- a/nemo_rl/environments/nemo_gym.py +++ b/nemo_rl/environments/nemo_gym.py @@ -708,8 +708,12 @@ async def _postprocess_receipt_mode( isinstance(terminal_logical_request_id, str) and terminal_logical_request_id ): # A harness that reports no terminal id still gets its manifest - # fetched; receipt assembly falls back to heuristic selection. + # fetched; receipt assembly attributes the terminal from the + # scored response, falling back to heuristic selection. terminal_logical_request_id = None + scored_response = nemo_gym_result.get("response") + if not isinstance(scored_response, dict): + scored_response = None receipt = None try: manifest = await self._control( @@ -721,6 +725,7 @@ async def _postprocess_receipt_mode( rollout_id, manifest, terminal_logical_request_id=terminal_logical_request_id, + scored_response=scored_response, reward=float(nemo_gym_result.get("reward") or 0.0), ) except (RuntimeError, OSError) as error: @@ -740,19 +745,30 @@ def _assemble_receipt( manifest: dict, *, terminal_logical_request_id: Optional[str], + scored_response: Optional[dict] = None, reward: float, ) -> dict: """Build the token-free RolloutReceipt payload from a ledger manifest. - With a declared terminal (``terminal_logical_request_id``), the - terminal row is the manifest row whose ``logical_request_id`` equals - the response id the agent actually kept; a declared id that matches no - row masks the rollout and never falls back. Without a declaration, - Gym's ``select_terminal_call`` infers the terminal from the manifest's - parent links (fail-closed: ambiguity masks). The receipt records which - path chose the terminal in ``terminal_selection``. Retry duplicates - are dead-branch rows: they stay in the manifest (their staged rows are - fetched, verified, and cleaned) but never join the terminal chain — + Terminal selection is staged, fail-closed at every stage: + + 1. Witness attribution (``resolve_terminal``): the declared response + id (``terminal_logical_request_id``), the scored response's + envelope id, and its content fingerprints each independently name + a manifest row through ``CallRecord.response_id`` and the recorded + fingerprints. Agreeing witnesses attribute; a declared id that + matches no row masks and never falls back; disagreeing witnesses + attribute nothing. + 2. Heuristic fallback (``select_terminal_call``): with no witness, + the manifest's explicit parent links infer the terminal + (fail-closed: ambiguity masks). + + The receipt records the resolving stage in ``terminal_selection`` + (``declared``/``response_id``/``content``/``heuristic`` — failed + selections stamp the last stage attempted) and the witness trail in + ``terminal_attribution_reason``. Retry duplicates are dead-branch + rows: they stay in the manifest (their staged rows are fetched, + verified, and cleaned) but never join the terminal chain — ``verify_and_linearize`` tolerates rows unreferenced by the terminal chain. @@ -769,6 +785,11 @@ def _assemble_receipt( marks a call whose completion WAS served — a hole in the chain — and poisons. """ + # Deferred: nemo_gym is an optional extra absent in non-gym runs. + from nemo_gym.token_id_capture.staging.attribution import resolve_terminal + from nemo_gym.token_id_capture.staging.records import CallRecord + from nemo_gym.token_id_capture.staging.terminal import select_terminal_call + records = [dict(record) for record in manifest.get("records") or []] failures = list(manifest.get("failures") or []) deduped: dict[str, dict] = {} @@ -776,34 +797,32 @@ def _assemble_receipt( deduped.setdefault(str(record.get("model_call_id")), record) terminal_record = None selection_reason = None - if terminal_logical_request_id is not None: - terminal_selection = "declared" - terminal_record = next( - ( - record - for record in deduped.values() - if record.get("logical_request_id") == terminal_logical_request_id - ), - None, - ) - else: - terminal_selection = "heuristic" - # Deferred: nemo_gym is an optional extra absent in non-gym runs. - from nemo_gym.token_id_capture.staging.records import CallRecord - from nemo_gym.token_id_capture.staging.terminal import ( - select_terminal_call, + attribution_reason = None + terminal_selection = "heuristic" + parsed_records = None + try: + parsed_records = [ + CallRecord.model_validate(record) for record in deduped.values() + ] + except ValueError: + selection_reason = "invalid_manifest_row" + if parsed_records is not None: + attribution = resolve_terminal( + parsed_records, + scored_response, + declared_response_id=terminal_logical_request_id, ) - - try: - selection = select_terminal_call( - [ - CallRecord.model_validate(record) - for record in deduped.values() - ] - ) - except ValueError: - selection_reason = "invalid_manifest_row" + attribution_reason = attribution.reason or None + if attribution.attributed: + terminal_selection = attribution.method + terminal_record = deduped[attribution.model_call_id] + elif terminal_logical_request_id is not None: + # A declaration is authoritative: a declared id the ledger + # cannot confirm masks and never falls back to the heuristic. + terminal_selection = "declared" + selection_reason = None else: + selection = select_terminal_call(parsed_records) if selection.terminal_model_call_id is not None: terminal_record = deduped[selection.terminal_model_call_id] else: @@ -830,6 +849,7 @@ def _assemble_receipt( "capture_poisoned": failure_reason is not None, "failure_reason": failure_reason, "terminal_selection": terminal_selection, + "terminal_attribution_reason": attribution_reason, } def _postprocess_nemo_gym_to_nemo_rl_result( diff --git a/nemo_rl/experience/blackbox_finalizer.py b/nemo_rl/experience/blackbox_finalizer.py index d13c904670d..3c74530c23d 100644 --- a/nemo_rl/experience/blackbox_finalizer.py +++ b/nemo_rl/experience/blackbox_finalizer.py @@ -557,18 +557,33 @@ def finalize_group( if isinstance(receipt, dict) and receipt.get("capture_poisoned") ) ) - # Heuristic terminal selection is a fallback for harnesses that do not - # declare the response they kept; a nonzero fraction on a declaring - # harness is a regression signal. - heuristic_receipts = sum( + # Per-method terminal-selection breakdown. Witness methods + # (declared/response_id/content) resolve from evidence; heuristic is + # the no-witness parent-link fallback — a nonzero heuristic fraction + # on a declaring harness is a regression signal. Failed selections + # stamp the last stage attempted, so masked rollouts stay visible in + # their method's bucket (cross-reference finalize/invalid_row_rate). + for method in ("declared", "response_id", "content", "heuristic"): + method_receipts = sum( + 1 + for receipt in receipts + if isinstance(receipt, dict) + and receipt.get("terminal_selection") == method + ) + metrics[f"finalize/terminal_selection_{method}_count"] = float( + method_receipts + ) + metrics[f"finalize/terminal_selection_{method}_fraction"] = ( + method_receipts / len(receipts) + ) + witness_disagreements = sum( 1 for receipt in receipts if isinstance(receipt, dict) - and receipt.get("terminal_selection") == "heuristic" + and "witness_disagreement" in str(receipt.get("terminal_attribution_reason") or "") ) - metrics["finalize/heuristic_terminal_count"] = float(heuristic_receipts) - metrics["finalize/heuristic_terminal_fraction"] = heuristic_receipts / len( - receipts + metrics["finalize/terminal_witness_disagreement_count"] = float( + witness_disagreements ) for row in rows: if not row.valid: diff --git a/tests/unit/data_plane/test_blackbox_finalizer.py b/tests/unit/data_plane/test_blackbox_finalizer.py index 21a92fa6b5b..697c24921de 100644 --- a/tests/unit/data_plane/test_blackbox_finalizer.py +++ b/tests/unit/data_plane/test_blackbox_finalizer.py @@ -220,8 +220,10 @@ def test_finalize_group_publishes_n_rows_with_placeholder(tq_client, partitions) # Group staleness comes from the valid rollout's calls (wv 4), not the fallback. assert (finalized.group_min_wv, finalized.group_max_wv) == (4, 4) assert finalized.metrics["finalize/invalid_row_rate"] == 0.5 - assert finalized.metrics["finalize/heuristic_terminal_count"] == 1.0 - assert finalized.metrics["finalize/heuristic_terminal_fraction"] == 0.5 + assert finalized.metrics["finalize/terminal_selection_heuristic_count"] == 1.0 + assert finalized.metrics["finalize/terminal_selection_heuristic_fraction"] == 0.5 + assert finalized.metrics["finalize/terminal_selection_declared_count"] == 0.0 + assert finalized.metrics["finalize/terminal_witness_disagreement_count"] == 0.0 rows = _fetch_rows(tq_client, rollout_ids) sample_mask = torch.as_tensor(rows["sample_mask"]).flatten() diff --git a/tests/unit/data_plane/token_capture_test_fixtures.py b/tests/unit/data_plane/token_capture_test_fixtures.py index 8fb7e05b46d..7fc92162c1c 100644 --- a/tests/unit/data_plane/token_capture_test_fixtures.py +++ b/tests/unit/data_plane/token_capture_test_fixtures.py @@ -88,6 +88,7 @@ def _manifest(record: StagedCallRecord) -> CallRecord: staging_key=record.staging_key, chain_hash=record.chain_hash, cumulative_hash=record.cumulative_hash, + response_id=f"chatcmpl-{record.model_call_id}", ) @@ -138,6 +139,7 @@ def build_fixture_artifacts( rollout_id=rollout_id, terminal_model_call_id=records[-1].model_call_id, manifest=[_manifest(record) for record in records], + terminal_selection="declared", ) snapshots = [ StagedCallSnapshot.model_validate(record.model_dump()) for record in records diff --git a/tests/unit/environments/test_nemo_gym_token_capture.py b/tests/unit/environments/test_nemo_gym_token_capture.py index aa564ec10f2..e6cc771ba22 100644 --- a/tests/unit/environments/test_nemo_gym_token_capture.py +++ b/tests/unit/environments/test_nemo_gym_token_capture.py @@ -25,7 +25,13 @@ def _capture_env() -> NemoGym: return object.__new__(env_cls) -def _manifest_record(call_id: str, *, logical_request_id: str, parent: str | None = None) -> dict: +def _manifest_record( + call_id: str, + *, + response_id: str | None = None, + parent: str | None = None, + cumulative_hash: str | None = None, +) -> dict: prev_len = 0 if parent is None else 900 return { "model_call_id": call_id, @@ -38,15 +44,16 @@ def _manifest_record(call_id: str, *, logical_request_id: str, parent: str | Non "extras_digest": "b" * 64, "staging_key": f"r0/{call_id}", "mode": "text" if parent is None else "token_in", - "logical_request_id": logical_request_id, + "response_id": response_id or f"resp-{call_id}", + "cumulative_hash": cumulative_hash, } def test_receipt_postprocess_without_a_terminal_logical_id_uses_the_heuristic() -> None: env = _capture_env() records = [ - _manifest_record("c1", logical_request_id="lr-1"), - _manifest_record("c2", logical_request_id="lr-2", parent="c1"), + _manifest_record("c1"), + _manifest_record("c2", parent="c1"), ] env._control = AsyncMock( return_value={"rollout_id": "r0", "records": records, "failures": []} @@ -69,8 +76,8 @@ def test_receipt_postprocess_without_a_terminal_logical_id_uses_the_heuristic() def test_receipt_postprocess_fetches_manifest_and_selects_terminal_row() -> None: env = _capture_env() records = [ - _manifest_record("c1", logical_request_id="lr-1"), - _manifest_record("c2", logical_request_id="lr-2", parent="c1"), + _manifest_record("c1"), + _manifest_record("c2", parent="c1"), ] env._control = AsyncMock( return_value={"rollout_id": "r0", "records": records, "failures": []} @@ -79,7 +86,7 @@ def test_receipt_postprocess_fetches_manifest_and_selects_terminal_row() -> None result = asyncio.run( env._postprocess_receipt_mode( {"_ng_rollout_id": "r0"}, - {"reward": 1.0, "terminal_logical_request_id": "lr-2"}, + {"reward": 1.0, "terminal_logical_request_id": "resp-c2"}, ) ) @@ -102,11 +109,11 @@ def test_receipt_assembly_poisons_on_failure_rows() -> None: env = _capture_env() manifest = { "rollout_id": "r0", - "records": [_manifest_record("c1", logical_request_id="lr-1")], + "records": [_manifest_record("c1")], "failures": [{"model_call_id": "c2", "reason": "worker_capture_failed"}], } receipt = env._assemble_receipt( - "r0", manifest, terminal_logical_request_id="lr-1", reward=0.0 + "r0", manifest, terminal_logical_request_id="resp-c1", reward=0.0 ) assert receipt["capture_poisoned"] is True assert receipt["failure_reason"] == "worker_capture_failed" @@ -121,8 +128,8 @@ def test_receipt_assembly_ignores_uncommitted_call_failures_off_the_terminal_cha manifest = { "rollout_id": "r0", "records": [ - _manifest_record("c1", logical_request_id="lr-1"), - _manifest_record("c2", logical_request_id="lr-2", parent="c1"), + _manifest_record("c1"), + _manifest_record("c2", parent="c1"), ], "failures": [ { @@ -132,7 +139,7 @@ def test_receipt_assembly_ignores_uncommitted_call_failures_off_the_terminal_cha ], } receipt = env._assemble_receipt( - "r0", manifest, terminal_logical_request_id="lr-2", reward=1.0 + "r0", manifest, terminal_logical_request_id="resp-c2", reward=1.0 ) assert receipt["capture_poisoned"] is False assert receipt["failure_reason"] is None @@ -145,7 +152,7 @@ def test_receipt_assembly_still_poisons_when_the_terminal_call_died_uncommitted( env = _capture_env() manifest = { "rollout_id": "r0", - "records": [_manifest_record("c1", logical_request_id="lr-1")], + "records": [_manifest_record("c1")], "failures": [ { "model_call_id": "c2", @@ -154,7 +161,7 @@ def test_receipt_assembly_still_poisons_when_the_terminal_call_died_uncommitted( ], } receipt = env._assemble_receipt( - "r0", manifest, terminal_logical_request_id="lr-2", reward=0.0 + "r0", manifest, terminal_logical_request_id="resp-c2", reward=0.0 ) assert receipt["capture_poisoned"] is True assert receipt["failure_reason"] == "missing_terminal_row" @@ -164,11 +171,11 @@ def test_receipt_assembly_poisons_when_the_terminal_row_is_missing() -> None: env = _capture_env() manifest = { "rollout_id": "r0", - "records": [_manifest_record("c1", logical_request_id="lr-1")], + "records": [_manifest_record("c1")], "failures": [], } receipt = env._assemble_receipt( - "r0", manifest, terminal_logical_request_id="lr-lost", reward=0.0 + "r0", manifest, terminal_logical_request_id="resp-lost", reward=0.0 ) assert receipt["capture_poisoned"] is True assert receipt["failure_reason"] == "missing_terminal_row" @@ -181,10 +188,10 @@ def test_receipt_assembly_poisons_when_the_terminal_row_is_missing() -> None: def test_receipt_assembly_heuristic_eliminates_abandoned_retry() -> None: env = _capture_env() records = [ - _manifest_record("c1", logical_request_id="lr-1"), - _manifest_record("c2", logical_request_id="lr-2", parent="c1"), - _manifest_record("c2r", logical_request_id="lr-2r", parent="c1"), - _manifest_record("c3", logical_request_id="lr-3", parent="c2"), + _manifest_record("c1"), + _manifest_record("c2", parent="c1"), + _manifest_record("c2r", parent="c1"), + _manifest_record("c3", parent="c2"), ] manifest = {"rollout_id": "r0", "records": records, "failures": []} receipt = env._assemble_receipt( @@ -198,9 +205,9 @@ def test_receipt_assembly_heuristic_eliminates_abandoned_retry() -> None: def test_receipt_assembly_heuristic_masks_a_final_call_retry() -> None: env = _capture_env() records = [ - _manifest_record("c1", logical_request_id="lr-1"), - _manifest_record("c2", logical_request_id="lr-2", parent="c1"), - _manifest_record("c2r", logical_request_id="lr-2r", parent="c1"), + _manifest_record("c1"), + _manifest_record("c2", parent="c1"), + _manifest_record("c2r", parent="c1"), ] manifest = {"rollout_id": "r0", "records": records, "failures": []} receipt = env._assemble_receipt( @@ -224,7 +231,7 @@ def test_receipt_assembly_heuristic_masks_an_empty_manifest() -> None: def test_receipt_assembly_heuristic_masks_invalid_manifest_rows() -> None: env = _capture_env() - bad = _manifest_record("c1", logical_request_id="lr-1") + bad = _manifest_record("c1") bad["delta_len"] = 0 # violates the CallRecord length contract manifest = {"rollout_id": "r0", "records": [bad], "failures": []} receipt = env._assemble_receipt( @@ -240,13 +247,13 @@ def test_receipt_assembly_keeps_dead_branch_siblings_in_the_manifest() -> None: never becomes the terminal call.""" env = _capture_env() records = [ - _manifest_record("c1", logical_request_id="lr-1"), - _manifest_record("c2", logical_request_id="lr-2", parent="c1"), - _manifest_record("c2r", logical_request_id="lr-2r", parent="c1"), + _manifest_record("c1"), + _manifest_record("c2", parent="c1"), + _manifest_record("c2r", parent="c1"), ] manifest = {"rollout_id": "r0", "records": records, "failures": []} receipt = env._assemble_receipt( - "r0", manifest, terminal_logical_request_id="lr-2r", reward=1.0 + "r0", manifest, terminal_logical_request_id="resp-c2r", reward=1.0 ) assert receipt["terminal_model_call_id"] == "c2r" assert receipt["capture_poisoned"] is False @@ -260,7 +267,111 @@ def test_receipt_postprocess_returns_placeholder_on_fetch_failure() -> None: result = asyncio.run( env._postprocess_receipt_mode( {"_ng_rollout_id": "r0"}, - {"reward": 1.0, "terminal_logical_request_id": "lr-1"}, + {"reward": 1.0, "terminal_logical_request_id": "resp-c1"}, ) ) assert result["receipt"] is None + + +def test_response_id_witness_resolves_a_final_call_retry() -> None: + """The heuristic masks a retried final call; the scored response's served + envelope id names the sibling the harness kept, recovering the rollout.""" + env = _capture_env() + records = [ + _manifest_record("c1"), + _manifest_record("c2", parent="c1", cumulative_hash="a" * 64), + _manifest_record("c2r", parent="c1", cumulative_hash="c" * 64), + ] + manifest = {"rollout_id": "r0", "records": records, "failures": []} + receipt = env._assemble_receipt( + "r0", + manifest, + terminal_logical_request_id=None, + scored_response={"id": "resp-c2r", "output": []}, + reward=1.0, + ) + assert receipt["terminal_model_call_id"] == "c2r" + assert receipt["terminal_selection"] == "response_id" + assert receipt["capture_poisoned"] is False + + +def test_unattributed_scored_response_falls_back_to_the_heuristic() -> None: + env = _capture_env() + records = [ + _manifest_record("c1"), + _manifest_record("c2", parent="c1"), + ] + manifest = {"rollout_id": "r0", "records": records, "failures": []} + receipt = env._assemble_receipt( + "r0", + manifest, + terminal_logical_request_id=None, + scored_response={"id": "resp-unknown", "output": []}, + reward=1.0, + ) + assert receipt["terminal_model_call_id"] == "c2" + assert receipt["terminal_selection"] == "heuristic" + assert "response_id_no_match" in (receipt["terminal_attribution_reason"] or "") + + +def test_declared_and_response_id_witnesses_corroborate() -> None: + env = _capture_env() + records = [ + _manifest_record("c1"), + _manifest_record("c2", parent="c1"), + ] + manifest = {"rollout_id": "r0", "records": records, "failures": []} + receipt = env._assemble_receipt( + "r0", + manifest, + terminal_logical_request_id="resp-c2", + scored_response={"id": "resp-c2", "output": []}, + reward=1.0, + ) + assert receipt["terminal_model_call_id"] == "c2" + assert receipt["terminal_selection"] == "declared" + assert "corroborated_by=response_id" in (receipt["terminal_attribution_reason"] or "") + + +def test_witness_disagreement_masks_a_retry_instead_of_guessing() -> None: + """A declaration naming one retry sibling while the scored response's id + names the other is a contradiction: attribution abstains, the declared + path stays authoritative, and the rollout masks.""" + env = _capture_env() + records = [ + _manifest_record("c1"), + _manifest_record("c2", parent="c1", cumulative_hash="a" * 64), + _manifest_record("c2r", parent="c1", cumulative_hash="c" * 64), + ] + manifest = {"rollout_id": "r0", "records": records, "failures": []} + receipt = env._assemble_receipt( + "r0", + manifest, + terminal_logical_request_id="resp-c2", + scored_response={"id": "resp-c2r", "output": []}, + reward=1.0, + ) + assert receipt["terminal_model_call_id"] is None + assert receipt["capture_poisoned"] is True + assert "witness_disagreement[" in (receipt["terminal_attribution_reason"] or "") + + +def test_postprocess_passes_the_scored_response_to_attribution() -> None: + env = _capture_env() + records = [ + _manifest_record("c1"), + _manifest_record("c2", parent="c1", cumulative_hash="a" * 64), + _manifest_record("c2r", parent="c1", cumulative_hash="c" * 64), + ] + env._control = AsyncMock( + return_value={"rollout_id": "r0", "records": records, "failures": []} + ) + result = asyncio.run( + env._postprocess_receipt_mode( + {"_ng_rollout_id": "r0"}, + {"reward": 1.0, "response": {"id": "resp-c2", "output": []}}, + ) + ) + receipt = result["receipt"] + assert receipt["terminal_model_call_id"] == "c2" + assert receipt["terminal_selection"] == "response_id"