Skip to content

feat(hpt): configured HPT graph for the ABC yam baseline - #46

Open
AnikethCheluva wants to merge 1 commit into
graph-abcfrom
graph-hpt-configs
Open

feat(hpt): configured HPT graph for the ABC yam baseline#46
AnikethCheluva wants to merge 1 commit into
graph-abcfrom
graph-hpt-configs

Conversation

@AnikethCheluva

Copy link
Copy Markdown

Wires the stems, trunk and flow head into a runnable experiment, so this stack
now delivers the actual HPT architecture rather than a DP model matched to it:
per-modality stems with cross-attention latent pooling, a 16-block shared
trunk, and flow matching.

The stage list is the dataflow, which is the payoff of decomposing:

observations --HPTStemStage--> hpt/tokens --HPTTrunkStage--> condition
actions --ActionTargetBuilder--> target --FlowNoisingStage--> flow/*
flow/* + condition --FlowDenoiserStage--> flow/predicted_velocity
--FlowVelocityLossStage--> loss/*

Four modalities get stems: the 14-D pose through an MLP, and the front plus two
wrist cameras through ResNet encoders. Each pools to 16 latents, so the trunk
sees 64 tokens plus its action token.

hpt.embed_dim is a single shared bus width read by every stem output, every
cross_attn modality_embed_dim, the trunk and the head's cond_dim. A test walks
all of those rather than trusting the interpolation, because a mismatch is the
most likely way this config rots and the stem stage only catches it at the
first batch.

The head uses CrossTransformer, as upstream does, which needs the token axis
the trunk's pooled vector lacks -- hence condition_as_tokens.

Also adds experiment/abc/yam_fstshirt_hpt as the architecture counterpart to
yam_fstshirt_dp: same data, same action space, same dataloader params, so a
delta between them is attributable to the model. A test asserts the two stay
distinct and keep sharing their data config.

Verified: composes, instantiates at 55.3M params, lints clean in train and
inference, and a real batch trains with gradient reaching all 564 parameter
tensors, then samples a (B, 100, 14) chunk at inference. Full suite 581 passed.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

@AnikethCheluva
AnikethCheluva marked this pull request as ready for review September 8, 2026 05:03
Wires the stems, trunk and flow head into a runnable experiment, so this stack
now delivers the actual HPT architecture rather than a DP model matched to it:
per-modality stems with cross-attention latent pooling, a 16-block shared
trunk, and flow matching.

The stage list is the dataflow, which is the payoff of decomposing:

  observations --HPTStemStage--> hpt/tokens --HPTTrunkStage--> condition
  actions --ActionTargetBuilder--> target --FlowNoisingStage--> flow/*
  flow/* + condition --FlowDenoiserStage--> flow/predicted_velocity
                                      --FlowVelocityLossStage--> loss/*

Four modalities get stems: the 14-D pose through an MLP, and the front plus two
wrist cameras through ResNet encoders. Each pools to 16 latents, so the trunk
sees 64 tokens plus its action token.

hpt.embed_dim is a single shared bus width read by every stem output, every
cross_attn modality_embed_dim, the trunk and the head's cond_dim. A test walks
all of those rather than trusting the interpolation, because a mismatch is the
most likely way this config rots and the stem stage only catches it at the
first batch.

The head uses CrossTransformer, as upstream does, which needs the token axis
the trunk's pooled vector lacks -- hence condition_as_tokens.

Also adds experiment/abc/yam_fstshirt_hpt as the architecture counterpart to
yam_fstshirt_dp: same data, same action space, same dataloader params, so a
delta between them is attributable to the model. A test asserts the two stay
distinct and keep sharing their data config.

Verified: composes, instantiates at 55.3M params, lints clean in train and
inference, and a real batch trains with gradient reaching all 564 parameter
tensors, then samples a (B, 100, 14) chunk at inference. Full suite 581 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant