feat(hpt): configured HPT graph for the ABC yam baseline - #46
Open
AnikethCheluva wants to merge 1 commit into
Open
feat(hpt): configured HPT graph for the ABC yam baseline#46AnikethCheluva wants to merge 1 commit into
AnikethCheluva wants to merge 1 commit into
Conversation
This was referenced Sep 8, 2026
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Sep 8, 2026
AnikethCheluva
marked this pull request as ready for review
September 8, 2026 05:03
AnikethCheluva
force-pushed
the
graph-hpt-configs
branch
from
September 8, 2026 21:07
118f0b2 to
c142158
Compare
AnikethCheluva
force-pushed
the
graph-abc
branch
from
September 8, 2026 21:07
39326ca to
2240843
Compare
AnikethCheluva
force-pushed
the
graph-hpt-configs
branch
from
September 9, 2026 22:21
c142158 to
61eee62
Compare
AnikethCheluva
force-pushed
the
graph-abc
branch
from
September 9, 2026 22:21
2240843 to
281053f
Compare
AnikethCheluva
force-pushed
the
graph-abc
branch
from
September 10, 2026 17:56
281053f to
7a6d5f1
Compare
AnikethCheluva
force-pushed
the
graph-hpt-configs
branch
from
September 10, 2026 17:56
61eee62 to
2a2d247
Compare
This was referenced Sep 10, 2026
Wires the stems, trunk and flow head into a runnable experiment, so this stack
now delivers the actual HPT architecture rather than a DP model matched to it:
per-modality stems with cross-attention latent pooling, a 16-block shared
trunk, and flow matching.
The stage list is the dataflow, which is the payoff of decomposing:
observations --HPTStemStage--> hpt/tokens --HPTTrunkStage--> condition
actions --ActionTargetBuilder--> target --FlowNoisingStage--> flow/*
flow/* + condition --FlowDenoiserStage--> flow/predicted_velocity
--FlowVelocityLossStage--> loss/*
Four modalities get stems: the 14-D pose through an MLP, and the front plus two
wrist cameras through ResNet encoders. Each pools to 16 latents, so the trunk
sees 64 tokens plus its action token.
hpt.embed_dim is a single shared bus width read by every stem output, every
cross_attn modality_embed_dim, the trunk and the head's cond_dim. A test walks
all of those rather than trusting the interpolation, because a mismatch is the
most likely way this config rots and the stem stage only catches it at the
first batch.
The head uses CrossTransformer, as upstream does, which needs the token axis
the trunk's pooled vector lacks -- hence condition_as_tokens.
Also adds experiment/abc/yam_fstshirt_hpt as the architecture counterpart to
yam_fstshirt_dp: same data, same action space, same dataloader params, so a
delta between them is attributable to the model. A test asserts the two stay
distinct and keep sharing their data config.
Verified: composes, instantiates at 55.3M params, lints clean in train and
inference, and a real batch trains with gradient reaching all 564 parameter
tensors, then samples a (B, 100, 14) chunk at inference. Full suite 581 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AnikethCheluva
force-pushed
the
graph-hpt-configs
branch
from
September 10, 2026 22:20
2a2d247 to
111cb46
Compare
AnikethCheluva
force-pushed
the
graph-abc
branch
from
September 10, 2026 22:20
7a6d5f1 to
037f68c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Wires the stems, trunk and flow head into a runnable experiment, so this stack
now delivers the actual HPT architecture rather than a DP model matched to it:
per-modality stems with cross-attention latent pooling, a 16-block shared
trunk, and flow matching.
The stage list is the dataflow, which is the payoff of decomposing:
observations --HPTStemStage--> hpt/tokens --HPTTrunkStage--> condition
actions --ActionTargetBuilder--> target --FlowNoisingStage--> flow/*
flow/* + condition --FlowDenoiserStage--> flow/predicted_velocity
--FlowVelocityLossStage--> loss/*
Four modalities get stems: the 14-D pose through an MLP, and the front plus two
wrist cameras through ResNet encoders. Each pools to 16 latents, so the trunk
sees 64 tokens plus its action token.
hpt.embed_dim is a single shared bus width read by every stem output, every
cross_attn modality_embed_dim, the trunk and the head's cond_dim. A test walks
all of those rather than trusting the interpolation, because a mismatch is the
most likely way this config rots and the stem stage only catches it at the
first batch.
The head uses CrossTransformer, as upstream does, which needs the token axis
the trunk's pooled vector lacks -- hence condition_as_tokens.
Also adds experiment/abc/yam_fstshirt_hpt as the architecture counterpart to
yam_fstshirt_dp: same data, same action space, same dataloader params, so a
delta between them is attributable to the model. A test asserts the two stay
distinct and keep sharing their data config.
Verified: composes, instantiates at 55.3M params, lints clean in train and
inference, and a real batch trains with gradient reaching all 564 parameter
tensors, then samples a (B, 100, 14) chunk at inference. Full suite 581 passed.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com