Aniketh/arc - #573
Open
AnikethCheluva wants to merge 100 commits into
Open
Conversation
AnikethCheluva
force-pushed
the
aniketh/arc
branch
from
August 26, 2026 14:30
a412dcf to
6617e22
Compare
AnikethCheluva
changed the base branch from
bf/7-configs
to
graphite-base/573
September 3, 2026 18:31
AnikethCheluva
force-pushed
the
aniketh/arc
branch
from
September 3, 2026 18:31
6617e22 to
e3b351d
Compare
AnikethCheluva
force-pushed
the
graphite-base/573
branch
from
September 3, 2026 18:31
a4d1839 to
19b84c1
Compare
AnikethCheluva
changed the base branch from
graphite-base/573
to
aniketh/abc
September 3, 2026 18:31
Collaborator
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Sep 3, 2026
AnikethCheluva
force-pushed
the
aniketh/abc
branch
from
September 5, 2026 15:23
19b84c1 to
e009239
Compare
AnikethCheluva
force-pushed
the
aniketh/arc
branch
from
September 5, 2026 15:23
e3b351d to
4c28a74
Compare
Claude Code ReviewReview of PR #573 — "Aniketh/arc"SummaryThis PR is enormous, unfocused, and its title/description ("Aniketh/arc", no description) tell reviewers nothing. The diff mixes at least four unrelated concerns: a versioned Tsimulation package, a packed-episode dataloading pipeline, an H-Net algorithm stack with CUDA-kernel install scripts, and a "BATCHFLOW" refactor manifesto — plus doc rewrites and Key concerns
Suggestions
Reviewed by Claude · Review workflow |
AnikethCheluva
added a commit
to GaTech-RL2/EgoVerse-graph
that referenced
this pull request
Sep 8, 2026
Port of GaTech-RL2#573's arc-tokenizer work onto the graph fork, reduced to the part that belongs in this architecture: the tokenizer as a pair of pipeline nodes. The Planar arc tokenizer already existed as a loader-side transform, and that placement is invisible to the graph -- a stage list shows the model consuming `target` with no sign the target is an arc token rather than a time-indexed chunk, and tools/config_graph.py cannot lint the boundary because nothing declares it. These two stages move the boundary into the graph: `ArcTokenizeStage` takes ActionTargetBuilder's PLACE rather than following it. Two writers of `target` would be a duplicate-writer lint error and would leave the graph ambiguous about which one the denoiser models; as the sole writer, it leaves the rest of a DP or flow chain untouched. It tokenizes per sample through the same TokenizePlanarArcLength the loader transform uses, so both paths produce identical targets -- there is a test pinning that. `ArcDetokenizeStage` walks the waypoint polyline at `speed * dt * k`, the inverse of how the tokenizer lays waypoints out uniformly in arc length. It saturates at the window end rather than extrapolating past it, holds position for a zero-speed token (matching the tokenizer's degenerate branch), and interpolates heading as (cos, sin) so a chunk that wraps past +/-pi does not unwind through zero. Adds `inference_only` to core as the mirror of `train_only`. It is needed, not cosmetic: this runner treats a stage with an unsatisfied read as a configuration error and raises, so a stage that legitimately exists in only one mode has to say so. Without it the detokenizer -- whose `pred_action` input exists only at inference -- would abort every training run. config_graph.py reports the new restriction alongside the existing one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DcPCNAW3GKERANrqvfSCY1
The reconciliation layer. Every file where main and the batchflow lineage genuinely disagreed lands here, so everything above it is bulk-new code. embodiment.py takes main's collapsed enum (HUMAN_* 1-3, EVA_* 4-6) and re-adds PUSHSHAPES_SIM 15 / _STICK 16 / _SMALL_CIRCLE 17 -- pinned because trained checkpoints and collected datasets encode those IDs. zarr_dataset_multi.py is a 3-way merge against the fork point: main's SafeS3EpisodeResolver, EvenStrideDataset, _evenly_spaced_indices and intrinsics property, plus batchflow's _read_span, _annotations_for_span and LocalEpisodeResolverWithEmbodimentOverride. action_chunk_transforms.py keeps batchflow's DeltaAction alongside main's PadGripperZeros. The batchflow repo notes land in AGENTS.md rather than a second CLAUDE.md, so the repo keeps one conventions file. DESIGN.md is not carried over: it was a 2026-06-06 restructure proposal written against EgoVerse-pact-2, still marked "awaiting approval", describing a move that has since happened here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The JPEG-decodability probe at _probe_image_key calls simplejpeg.decode_jpeg(...) but the module was never imported in this file. The call sits inside a try/except Exception, so instead of crashing it made the probe report EVERY image as undecodable -- a silent false negative rather than an error. _common.py in the same package already imports it the same way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AGENTS.md still documented egomimic/models/hnet_nets/, egomimic/algo/hnet.py and egomimic/eval/eval_hnet.py. None of those paths exist: the packages are models/hnet/, algo/hnet/ and eval/core/eval_hnet.py. Anyone -- human or agent -- following the doc went looking for files that are not there, and AGENTS.md is the first thing an agent reads. Section headings renamed hnet_nets -> hnet to match. test_hnet_nets.py is left alone: that file genuinely still has that name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…oader
This PR introduces the packed subsystem -- ZarrEpisodePackedDataset and
pack_collate -- but MultiDataModuleWrapper, which is inherited unchanged from
main, hardcoded annotation_collate for every dataset. annotation_collate ends in
default_collate, which tries to torch.stack ragged packed samples, so every
packed_episode config died on its first batch with
RuntimeError: Trying to resize storage that is not resizable
pack_collate needs two call sites. The other one -- MultiDataset's norm-stat
inference in zarr_dataset_multi.py -- already had it (collate_fn = pack_collate
if is_packed else None) and is covered by test_packed_pipeline. The training
dataloader had neither the wiring nor a test, and main has no packed configs at
all, so nothing exercised it.
_collate_fn_for is ported from EgoVerse-gmm-dualstream / EgoVerse2, where this
dispatch already backs the live H-Net runs, rather than written fresh.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two standalone simulator packages, each with its own pushshapes/, collect/, examples/ and tests/. There is no v3: what was labelled v3 is the socket fix that ships AS v2 -- the intermediate all-faces-grip build was a bug. __init__.py aliases the active version's submodules to the top level so existing 'from Tsimulation.pushshapes import X' call sites keep working; TSIM_VERSION selects the version per process. Placed before eval because eval calls into it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hing on them
Adding the u_socket was done by editing the environment. Its latch, friction and
penetration guards -- 12 methods, ~470 lines -- went into env.py as
`if pusher_shape == "u_socket"` branches, and its 3-DOF action became a hardcoded
`expected_shape = (3,) if self.pusher_shape == "u_socket" else (2,)`. That is the
single largest reason the two sims diverged: sim_v1's env.py has ZERO socket
references, sim_v2's had 115, and env.py's step loop called nine socket-specific
guards in sequence.
An Agent now owns the three things the environment should not know about:
* ACTION SPACE -- action_dim (2 for a free-moving pusher, 3 when the agent
also controls orientation) and target_pose() to decode a raw action;
* BODY -- build() in the pymunk space;
* CONTACT MODEL -- pre_substep()/post_substep() hooks around each physics
substep, plus on_reset() for per-episode state.
env.step is agent-agnostic:
captured = self.agent.pre_substep(self)
self._drive_pusher_toward(tx, ty, dt_sub, target_angle)
self._space.step(dt_sub)
self._clamp_pusher_to_static()
self.agent.post_substep(self, captured)
Agent (circle, circle_small, stick, L) implements the hooks as no-ops.
USocketAgent owns all the latch/guard logic and the socket geometry constants,
and its solid_pusher / socket_inside_friction_only flags become constructor
arguments rather than environment state. A new agent with an unusual action
space is a new class plus one line in make_agent(), not another branch in the
simulator.
env.py 1292 -> 795 lines.
sim_v1 IS DELIBERATELY UNTOUCHED. It is frozen so pre-rewrite data replays
exactly; refactoring it would put that at risk for no benefit, since it has no
socket to abstract in the first place.
VERIFIED BY REPLAY EQUIVALENCE, not by inspection. Baselined the unmodified sim
with the identical harness first, then compared:
u_socket_3000_v2 100.0% -> 100.0% (p50 0.0033 -> 0.0033)
circle_3000_plus_gen_v2 89.7% -> 89.7%
circle_v2_obstonly 17.1% -> 25.7%
The gate caught two real bugs that inspection did not: the moved
_socket_contact_is_on_inner_face call site lost its env argument, and
socket_latched -- a property DERIVED from `_socket_constraints is not None` --
had become a plain attribute nothing updated, initialised to [] so it would have
read as permanently latched. Both fixed; the socket went 0% -> 100%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012V58H37tmcvgDthELMd5Xk
New subsystems with no counterpart on main: models/hnet (stages, blocks, the scan and register chunk interfaces, routing) and models/diffusion (DiT3D and spatial backbones, sampling, image VAE), plus cores and stems. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
widths=[512] in hpt_heads/hpt_stems and down_dims=[256,512,1024] in denoising_nets are evaluated once at import, so every caller that omits the argument shares one list object. None of the three currently mutates it, so nothing is broken today -- this removes the footgun before something does. Each becomes None with the original value restored inside the function, so the behaviour for an omitted argument is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
obs_encoder.py and cond_encoders.py each carried a private _mlp() of the same shape -- but ObsEncoder's used ReLU and CondEncoderModule's used GELU. Both are live: ObsEncoder backs nine bc_rnn model configs (base/tx/tx_chunk8/hnet/ tx_cotrain_*), CondEncoderModule backs the H-Net firstend/cotrain/bf_rh configs. They collapse into one build_mlp() whose 'act' argument is keyword-only with no default, so every call site states its activation. That is the point of the change: activations hold no parameters, so a checkpoint trained under one activation loads into the other with no error and merely produces different numbers -- exactly the kind of silent divergence two near-identical private helpers invite. Behaviour is unchanged: ReLU stays at both ObsEncoder call sites and GELU at both CondEncoderModule ones. Verified by building both encoders from the old and new trees -- identical activation lists, identical parameter keys, identical forward output hashes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…thms algo/hnet, algo/diffusion (with its outer-stage variants) and algo/bc. Builds on the model zoo in the previous commit. Also carries egomimic/pipeline/ -- the batchflow stage framework, its runner PipelineAlgo and the stage implementations -- plus the three H-Net lightning callbacks (random_attn_dropout, chunker_residual_scheduler, ratio_loss_scheduler) and BATCHFLOW.md. These sit here rather than in the infra commit because they import egomimic.models.hnet and egomimic.models.diffusion from the model zoo below, and pipeline/algo.py additionally imports egomimic.algo.hnet.episode_transforms from this commit. Carrying them lower left the infra commit unable to import six of its own modules when checked out on its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HNetPolicy.step referenced embodiment_id at two places in its body but never
declared it -- not a parameter, local, or attribute. The method failed two ways:
* PackedAlgoBase.inference_step already calls
policy.step(..., embodiment_id=self.domain_by_id.get(emb_id)), which raised
TypeError: unexpected keyword argument;
* called without it, the body raised NameError at the action_out lookup.
Either way the AR single-step path used for closed-loop sim rollout could not
run. The sibling policy step() in this same file already declares
embodiment_id: Optional[str] = None; this matches it, so the existing callers
work unchanged and single-embodiment models keep the None default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hpt_bc_keypoints_base sets lr=5e-5 with CosineAnnealingLR(T_max=1400). scheduler_interval defaults to "step", so T_max is 1400 STEPS ~ 14 epochs at the observed 100.8 steps/epoch -- and CosineAnnealingLR keeps evaluating cos(pi*T_cur/T_max) past T_max, so the LR climbs back up rather than stopping. Over 600 epochs that is ~22 sawtooth cycles between 1e-5 and 5e-5. Predicted 1.44e-05 at step 12299 vs 1.8e-05 observed on wandb. Every arc-cartesian run used constant lr 3e-4 with scheduler: null, so the keypoint runs as launched were neither constant nor at the same LR, and not comparable. Add const-LR variants. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
viz/keypoints.yaml and viz/keypoints_wrist.yaml both use a bare front_img_1 image_key, but the batch carries observations.images.front_img_1 (confirmed by probe) -- so neither can run. Add viz/keypoints_human.yaml with the correct key, no eva entry (these runs are single-domain), and no annotation_key, since arc_tests has no annotations. The eval sweep hardcoded annotation_key=null for four cartesian viz entries; on a keypoint evaluator those keys do not exist and hydra errors. Drive them from an ANNOT_EMBODIMENTS list instead, same fix as the dataloader overrides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Same bug as the training workflow, fixed there but not here: the sweep set data.valid_dataloader_params.eva_bimanual.*, so any human-only run died with 'Key eva_bimanual is not in struct'. Drive from EMBODIMENTS. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third instance of the same assumption in this file. Drive it from EMBODIMENTS like the others. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Walks a real frame pair through the computation: two hand poses with per-joint displacement, the 21 distances as bars with L-inf/L2/L1-mean marked, the reduction to a scalar, the rotation term, and accumulation into tokens at D. Uses frames 2731->2737 where the index fingertip moves 37.9mm while the slowest joint moves 6.9mm, so the choice of norm is visible: L1-sum is 8.3x L-inf on that single step, which is what compounds into the ~22x path inflation over an episode. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…r_training zarr_key_to_keyname returns None for any batch key that isn't a registered action/proprio zarr key (intrinsics, episode_hash, image keys). The old 'if key is not None' guard checked the wrong variable — 'key' is a string from _batch.items() and never None, so the branch always fired and wrote every unregistered key under a single None dict slot. Later writes clobbered earlier ones, and 'intrinsics' was silently dropped. Downstream _intrinsics_from_batch(batch, i) then returned None, so Human.viz / Eva.viz fell back to the hardcoded class INTRINSICS constant. For episodes whose per-episode K disagrees with the aria default (mecka fx=fy≈251, cy≈184 vs ARIA cy=240), this projected the GT trajectory ~55px vertically offset — the visible wrist-vs-palm misalignment reported against arc_tests mecka fold_clothes val-videos. Fix: fall back to the original key when zarr_key_to_keyname is None so unregistered keys survive the rekey. Also fixes downstream access to episode_hash and per-episode image side-channel keys.
Layers on top of arc-length-nv-eval only what wasn't already there:
data configs (D=0.20m / M=15 hardcoded to match target convention):
- aria_train_mecka_val{,_arctok}: cross-domain (train aria fold, valid
mecka fold_clothes)
- eva_only_fold{,_arctok}: eva-only robot-baseline runs
- mecka_folding_clothes{,_arctok}: mecka folding_clothes (note "ing"
variant of task name — distinct from arc_tests fold_clothes)
- mecka_folding_eva_fold_cotrain{,_arctok}: cotrain across mismatched
task names (mecka folding_clothes + eva fold_clothes)
hydra launchers (mirror target's submitit_pace_l40s.yaml convention):
- submitit_pace_a100 / _blackwell / _h100
viz + diagnostic scripts:
- egomimic/visualization/arc_tok_viz.py: detokenize+overlay helper for
notebooks (mirrors ArcTokEvalVideo viz path)
- scripts/pixel_check.py + .sbatch: verify projected GT dots match
val-video mp4 frame 0
- scripts/visualize_trunk_latents.py + .sbatch: t-SNE + HDBSCAN over
HPTModel.forward_features, emits arc_embedding_sweep's tabbed HTML
small UI fix in scripts/arc_embedding_sweep.py: image-panel close
button now has type=button + inline onclick fallback.
Dropped from the pre-rebase stash because target already had them
(and more evolved): eval_arctok/eval_hpt/eval_video changes,
arc_tests_cotrain* config edits, D40_M100 model/evaluator configs.
Dropped as no-longer-wanted per user note: FMPolicyWithVelDecoder /
WithVelReadout wrappers, associated hpt.py / denoising_policy.py /
hpt_nets.py / model config _veldec / _velreadout variants. Also
dropped an accidental LocalFolderEpisodeResolver -> S3EpisodeResolver
revert and a stray _LEGACY_EMBODIMENT_ALIASES removal.
…configs
Ports the D=40cm / M=100 arc-tok model configs and required class code
onto the bf/7-configs config layout so the in-flight training jobs can
launch from this branch:
- Restore hpt_cotrain_enc_dec_base.yaml with _target_ paths pointing at
bf's module layout (algo.hpt.algo.HPT, models.stems.hpt_stems.*,
models.heads.hpt_heads.MultiBlockTransformerDecoder,
models.diffusion.denoising_nets.CrossTransformer).
- Point hpt_cotrain_mecka_flow_shared_head_arc.yaml back at
hpt_cotrain_enc_dec_base and update its _target_ paths.
- New: hpt_cotrain_mecka_flow_shared_head_arc_D40_M100{,_veldec,_velreadout}.yaml.
- Add MLPVelocityDecoder to egomimic/models/heads/hpt_heads.py and
FMPolicyWithVelDecoder / FMPolicyWithVelReadout to
egomimic/models/heads/fm_policy.py.
- Fix eval_arctok.yaml viz path (cartesian → cartesian/base).
- Add ``arc_tokenizer`` config group defaults to train_zarr_cartesian.yaml
so ``arc_tokenizer.min_distance_unit=…`` CLI overrides resolve.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- pushshapes arc-length tokenizer (egomimic/rldb/zarr/pushshapes_arc_tokenizer.py)
and pushshapes.get_keymap_hpt_arc for reading (M+1, 2) arc-tok windows.
- HPT closed-loop inference: expand_arc_chunk_to_time so the (M+1, D) chunk is
played back as a time-uniform buffer against the env; replan_at tracks the
variable expanded-buffer length.
- hydra config groups for the pusht arc-tok stack:
data/pushshapes/pusht/{circle,circle_arc}.yaml, evaluator/hpt/{pusht,pusht_arc}.yaml,
model/pusht/*, model/pusht_arc/*.
- Restored hpt_cotrain_mecka_flow_shared_head.yaml (non-arc baseline) with
bf/7-configs _target_ paths so mecka baselines resume from aniketh/arc.
- logger/wandb/base.yaml default project: zarr_test -> arc so arc-tok runs land
in rl2-group/arc automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adapts robot/rollout.py for arc-tokenized policies. Subclasses PolicyRollout to reuse the loader / obs transform / safety pipeline; overrides only the predict step to detokenize the model's (M+1, 8) arc-tok output into a (H, 14) time-uniform chunk before handing it to the controller.
Rewrites TokenizeBimanualArcLengthCartesian to produce (M+1, 14) with full xyz + ypr + grip per arm (was (M+1, 8) dropping rotation). Rotation is unconditionally supervised — no opt-out. Gripper padding for human aria data preserved as an existing transform option. - arc_length_tokenizer.py: ARC_TOK_PER_ARM_DIM=7, ARC_TOK_BIMANUAL_DIM=14; SLERP for waypoint ypr resample; vel row extended to 14 dims with per-axis mean angular velocity in ypr slots. - Model configs (arc, D20_M15, D40_M25, D40_M100, M15/M25/M50/M100 + veldec + velreadout): act_dim 8 -> 14, infer_ac_dims 8 -> 14, veldec output_dim 8 -> 14. - FMPolicyWithVelDecoder / FMPolicyWithVelReadout: act_dim default 14. - eval_arctok.py, rollout-arc.py, visualization/arc_tok_viz.py: shape asserts 8 -> 14; removed zero-fill of ypr; detokenize returns 14-dim including model-predicted rotation. - Data configs (arc_sweep_*, arc_tests_cotrain_arctok*, folding_clothes _arctok, folding_eva_fold_cotrain_arctok, eva_only_fold_arctok, aria_train_mecka_val_arctok): comment updates. - Embodiment keymap/transforms: docstring updates only; routing intact.
Ports the 9-run rotation-fixed setup into a first-class doc: - What each of the 9 runs is (arc-tok, veldec, velreadout, eva_only_arctok, three non-arc baselines). - Exact common overrides + arc-tok specifics + baseline specifics. - Description strings for wandb id resolution. - Launcher script paths and how to swap partitions. - Why all 9 are fresh (8-dim ckpt vs 14-dim head shape mismatch). - Known operational issues: billing quota kills, norm-stats cold start, partition availability. - Pre-fire verification checklist.
The action-transform refactor on transform_fixes replaced the single `mode:` key with action_mode / coord_frame / rotation_mode on Eva and Human. Arc's data configs were written before that landed, so once arc is stacked on abc they were passing an unknown `mode` kwarg. 121 transform_list blocks across 37 configs, mechanically: cartesian -> cartesian / camframe / euler cartesian_padded -> cartesian_gripper_padded / camframe / euler cartesian_wristframe_ypr -> cartesian / eef_frame / euler keypoints_*frame_ypr -> keypoints / camframe|eef_frame / euler arc_tokenizer_cartesian -> arc_tokenizer_cartesian / camframe / euler pushshapes keeps its own module-level `mode` vocabulary (arc_tokenizer, pad_only) — it was never part of the refactor and is left alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnBSsxfUNUZQQ9wBwxbkaf
abc gave Yam the refactored action_mode / coord_frame / rotation_mode signature but only the plain cartesian layout. arc's abc_fstshirt_mecka_freefold_arc_cotrain_D40_M100 config feeds yam_bimanual through the arc tokenizer, so it was passing min_distance_unit / resampled_vector_length into a method that had no such parameters. Routes through the same _append_arc_tokenizer helper Eva and Human use. Yam is already 14D with a real gripper, so unlike Human it needs no padding step before tokenizing. Also ports scripts/pixel_check.py, the last caller still on the old single-`mode` API. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnBSsxfUNUZQQ9wBwxbkaf
Human's arc mode was silently doing two things at once: padding the zero
gripper AND tokenizing. That made `arc_tokenizer_cartesian` mean something
different on Human than on Eva/Yam, and it did not parallel the existing
cartesian / cartesian_gripper_padded pair.
Splits them:
arc_tokenizer_cartesian tokenize the native layout
(Eva, Yam - real gripper, 14D)
arc_tokenizer_cartesian_gripper_padded pad, then tokenize
(Human - no gripper, 12D -> 14D)
Human's bare `arc_tokenizer_cartesian` now raises and points at the padded
variant instead of quietly padding. The 48 Human blocks across 24 data
configs plus scripts/pixel_check.py move to the explicit name, so no
pipeline changes shape - it is the same transform list, honestly named.
Eva's 44 blocks are untouched.
Also guards rotation_mode at build time. The tokenizer's chunk layout is a
hard-coded 14D [xyz(3), ypr(3), grip(1)] x 2 and it SLERPs the ypr slots,
so quat (16D) and 6D (20D) were only ever going to fail - previously on
the first batch, deep inside a run, now when the transform list is built.
Verified: (M+1, 14) out, arc length 0.1998m for D=0.20, gripper slots
zero, rotation supervised; all 262 transform_lists across 101 data configs
still instantiate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PnBSsxfUNUZQQ9wBwxbkaf
One evaluator serves arc-tokenized and time-indexed runs. Arc detection is by row count (M waypoints plus a velocity token vs rollout_horizon steps), so a baseline chunk passes through instead of hitting the detokenizer. Every metric is arc-matched: clip to the first D metres of travel, resample to N points spaced uniformly in arc length, then score. Names carry no hint of the action space, so an arc run and its baseline twin plot on one axis. Scoring is EEF-frame and every metric is frame invariant. Position error survives a rigid transform, and rotation is reported as a geodesic angle, |log(R_pred R_gt^T)|, rather than a per-axis ypr difference: ypr is frame dependent and breaks at wraparound and gimbal lock, so it cannot be compared across runs. arc_matched_resample can now carry rotation through instead of resampling it and discarding it. Adds the D40/M100 experiment set: cotrain and ABC-only BC, arc and baseline, plus a 300M variant, all on the same evaluator.
Replaces the fixed-D arc-matched metric with four families, all anchored on the ground truth under normal action-chunk sampling instead of a reconstruction: - arcmatch: re-tokenize pred and GT onto a matched per-arm span (the shorter of the two travelled distances) and score the waypoints. Exactly travel invariant, so it measures path shape alone. Reported with and without the velocity row; the gap between them is the timing error. - dtw: warp the prediction against the GT chunk. Elastic in time, so unlike arcmatch it does see a travel mismatch. - detok / baseline: plain MSE against the raw GT chunk, split into xyz, ypr, gripper, final and geodesic. One name per action space; same computation. - pose_err_m: position and rotation as one number in metres, via a lever arm so radians are converted to the length they cost rather than summed with metres under an arbitrary unit choice. Drops the _timeidx_ rename. It existed because the old detok metric compared two reconstructions and could not be compared across action spaces; with a common GT and a matched chunk_length that is no longer true. The tokenizer overwrote actions_cartesian in place, so an arc run's batch never carried the time-indexed GT these metrics need. Recovering it by detokenizing would be circular (the reconstruction spans D by construction, so its distance carries no information), hence preserve_action_key. Verified offline: arcmatch reads 1e-9 to 3e-8 across predictions covering 0.5x to 2x the GT distance, a perfect prediction scores exactly 0 on every family, and pose_err_m matches its closed form. The measured caveats for ranking an arc run against its baseline are in the module docstring. Also: 40h multitask data mix with scale/ removed and selection by lab+task, sample_frac 0.2, and the skynet l40s launcher an arc experiment already referenced but which did not exist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuDAQKMEVv79cvzYiargiK
evaluator/hpt/sim_pusht_arc.yaml is tracked and targets egomimic.eval.hpt.eval_sim_arc.ArcSimEval, but the module was never committed, so the config resolves to nothing on a fresh checkout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuDAQKMEVv79cvzYiargiK
make_velocity_payload's MEAN_PER_DIM is (pos[-1] - pos[0]) / total_time, i.e. net displacement over time, so the velocity token is a CHORD rate. detokenize advanced `s` by that rate but compared it against cumulative ARC length, and arc >= chord for anything that is not a straight line, so the reconstruction replayed the motion too slowly. Measured on val episode 001948a7 (fold and stack the t-shirts), 12 windows: replay duration vs the time the arm actually took was off by 47.5% on average and up to 98% where the path doubles back and arc is nearly twice the chord. After converting to the arc rate the reconstruction needs, mean error is 2.5%. The conversion uses only what the token already carries (arc and chord are both recoverable from the waypoints), so the action space, trained checkpoints and norm stats are untouched. Guarded for chord ~ 0: a closed loop has speed ~ 0 by construction and a chord-based token carries no timing for it, so that case keeps the existing hold-first-waypoint behaviour. This affects deploy as well as val videos -- rollout-arc.py shares detokenize. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuDAQKMEVv79cvzYiargiK
compute_metrics_and_viz reverted the chunk to camera frame and only then let _visualize_preds detokenize it. The revert rotates AND translates every row, but row M of an arc token is a VELOCITY -- translating it yields a position. Measured on val episode 001948a7 frame 512, left arm: the velocity token (0.0810, 0.0574, 0.0074) m/s came out of the revert as (-0.2313, -0.0193, 0.8945), i.e. the camera-to-gripper offset. detokenize then read a speed of 0.92 instead of 0.0995 and raced through the 0.393 m path in 8 control steps instead of 61, piling the other 92 dots on the endpoint. That is what made every arc val video look like the arm teleports 40 cm in a third of a second. Adds HPTEvalVideo._viz_source (identity; a time-indexed run already predicts poses) and overrides it in ArcTokEvalVideo to detokenize, so apply_transform only ever receives rows that really are poses. This also fixes cam_paired_mse_avg / cam_final_mse_avg, which were comparing tokens whose last row was garbage. Consolidates the two copies of the arc-detection predicate onto ArcTokEvalVideo._is_arc, now also checking the last dim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuDAQKMEVv79cvzYiargiK
The arc path ran InterpolatePose(new_chunk_length=100) before tokenizing, which squeezed a 200-frame yam window and a 600-frame human window into 100 rows: a 2x and 6x decimation. Arc length measured on a decimated path chord-cuts and reads systematically short, so D, the velocity token and every arc metric were built on an understated path. get_transform_list now takes chunk_length and defaults it to the RAW window for arc modes, making the interpolation a no-op. The pipeline is what it should be: read raw frames, find the frames covering D, resample those to M. Model input is unchanged at (M+1, 14); only what the tokenizer consumes changes. This also fixes the velocity scale for free. Each row is now a real 30 Hz frame, so the hardcoded dt=1/30 is correct -- it was 2x too large on yam and 6x on human, which would have driven a deployed policy that much too fast. Baseline side: Human.get_keymap gains action_horizon so the raw window can be equalized with yam. Human defaulted to 30 frames against yam's 45, so a human baseline chunk covered 1.0 s and a yam one 1.5 s while both still emitted (100, 14) -- nothing in the shapes revealed it. The multitask baseline config now pins both to 45. Adds evaluator viz_chunk_rows: a baseline chunk is interpolated UP to 100 rows from 45 raw frames, so most of the drawn dots are interpolation rather than control setpoints. Setting it to the raw window makes the overlay show what the controller actually receives. Off by default. Adds notebooks/arc_reconstruction_error.ipynb: reference chunk cut at D vs detokenize(tokenize(chunk)), as overlaid video plus per-row error, one episode each for yam and human. Existing arc checkpoints and norm stats do not transfer -- the token values change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JuDAQKMEVv79cvzYiargiK
…umes
Four arc/baseline pairs, each pair differing only in action space:
abc_fslongshirt_{bc,arc_bc} no cotrain, robot 10.01h
abc_fstshirt_mecka_flagfold_cotrain_* 2.93h robot + 2.93h human
abc_fslongshirt_micro_foldclothes_cotrain_* 10.01h robot + 9.88h human
abc_mecka_fold_multitask_cotrain_* 38.5h robot + 39.8h human
The BC pair now uses abc "fold and stack the long sleeve shirts" at the same
10h volume and the same task as the fslongshirt cotrain pair, so the delta
between them isolates the human data rather than confounding it with a
different robot task. 10h was not reachable from "fold and stack the t-shirts"
(3.15h total), which is why the BC runs moved off it and were renamed.
Episode selection is now lab+task plus a deterministic subsample,
int(episode_hash.encode().hex()[-6:], 16) % 1000 < K
so each split lands on a target number of HOURS instead of a `debug: N` episode
cap. The trailing bytes vary in every hash format we carry (abc uuids,
mecka/microagi timestamps) and the selector measures uniform on all four pools
(p10~100, p90~890). A sum(ord) variant was tried first and discarded: it is
nearly constant across same-length timestamp hashes, so K=283 and K=290 gave
2.91h and 10.13h from the same pool. The `debug: N` caps are removed because
they capped episode counts and fought the hour targets.
Every hour figure in the configs was verified by evaluating the config's own
filter lambda against the episode table, not computed by hand.
Also brings the four smaller experiments up to what the multitask pair carries:
sample_frac 0.2, gpus_per_node 2, and the header note that
`override /hydra/launcher` inside an experiment is inert and the launcher has
to be passed on the CLI.
BC models switch to a constant LR. Both inherited lr=1e-4 with
CosineAnnealingLR(T_max=1400) from bc_flow_eva while the cotrain runs are
constant, so late-epoch BC numbers were not comparable to their cotrain twins.
scheduler: null, lr unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JuDAQKMEVv79cvzYiargiK
arcmatch compared a baseline against an arc run over different amounts of motion. chunk_length counted ROWS, but a baseline row is 45/30/100 = 0.015 s against an arc row's 1/30 s, so chunk_length: 30 scored 0.45 s of baseline motion against 1.0 s of arc motion. The same mismatch broke dt in the velocity payload and left arm_travel summing a total variation over 100 samples on one side and 45 on the other. Replaced with action_horizon, the true horizon in raw control frames, which is what 'normal action chunk sampling' means and so what basedist is measured over. The arc run's preserved window is already native spaced; a baseline chunk is de-interpolated back down from its 100 rows. De-interpolating only the baseline is not neutral, though. The round trip is a low-pass and a baseline chunk has already had half of it applied, so undoing it attenuates the baseline's error and leaves the arc side's intact. On real yam chunks that alone still flattered the baseline 1.36x at identical injected error. Both sides now go through the same H -> 100 -> H trip. Measured on 54 real episodes, baseline/arc score at identical 2/5/10 mm error: before 0.188 (baseline flattered 5.3x) de-interp baseline only 0.742 (flattered 1.36x) both round-tripped 1.0000 (p10 1.0000, p90 1.0000) Perfect models of both classes score exactly 0. Round trip costs 0.29% of measured travel and 0.6 mm, equally on both sides. Also adds arcmatch_degenerate_frac: a matched span near zero collapses every waypoint onto row 0, the origin in eef_frame, so the arm scores exactly 0 whatever was predicted. Those were silently dragging every mean down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pair the fslongshirt and mecka-fold-multitask arc/baseline runs with annotation-keyed data configs and pooled-Qwen model configs so language conditioning is the only variable at matched data volume. Co-authored-by: Cursor <cursoragent@cursor.com>
The hydra-config refactor dropped the text stem and annotation_key path. Bring both back as no-ops when annotation_key is unset so existing runs stay unchanged.
Language baseline experiments use the abc 300M pooled-Qwen model as-is. Arc-tokenizer runs inherit it and only change action_horizon/act_seq.
AnikethCheluva
force-pushed
the
aniketh/arc
branch
from
September 8, 2026 18:38
6196a85 to
ea45c70
Compare
AnikethCheluva
force-pushed
the
aniketh/abc
branch
from
September 8, 2026 18:38
e009239 to
6d23fba
Compare
AnikethCheluva
added a commit
to GaTech-RL2/EgoVerse-graph
that referenced
this pull request
Sep 8, 2026
Port of GaTech-RL2#573's arc-tokenizer work onto the graph fork, reduced to the part that belongs in this architecture: the tokenizer as a pair of pipeline nodes. The Planar arc tokenizer already existed as a loader-side transform, and that placement is invisible to the graph -- a stage list shows the model consuming `target` with no sign the target is an arc token rather than a time-indexed chunk, and tools/config_graph.py cannot lint the boundary because nothing declares it. These two stages move the boundary into the graph: `ArcTokenizeStage` takes ActionTargetBuilder's PLACE rather than following it. Two writers of `target` would be a duplicate-writer lint error and would leave the graph ambiguous about which one the denoiser models; as the sole writer, it leaves the rest of a DP or flow chain untouched. It tokenizes per sample through the same TokenizePlanarArcLength the loader transform uses, so both paths produce identical targets -- there is a test pinning that. `ArcDetokenizeStage` walks the waypoint polyline at `speed * dt * k`, the inverse of how the tokenizer lays waypoints out uniformly in arc length. It saturates at the window end rather than extrapolating past it, holds position for a zero-speed token (matching the tokenizer's degenerate branch), and interpolates heading as (cos, sin) so a chunk that wraps past +/-pi does not unwind through zero. Adds `inference_only` to core as the mirror of `train_only`. It is needed, not cosmetic: this runner treats a stage with an unsatisfied read as a configuration error and raises, so a stage that legitimately exists in only one mode has to say so. Without it the detokenizer -- whose `pred_action` input exists only at inference -- would abort every training run. config_graph.py reports the new restriction alongside the existing one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DcPCNAW3GKERANrqvfSCY1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.