Skip to content

feat(hpt): trunk primitives and tensor helpers - #42

Open
AnikethCheluva wants to merge 1 commit into
graph-nested-valfrom
graph-hpt-deps
Open

feat(hpt): trunk primitives and tensor helpers#42
AnikethCheluva wants to merge 1 commit into
graph-nested-valfrom
graph-hpt-deps

Conversation

@AnikethCheluva

Copy link
Copy Markdown

First step of decomposing HPT into graph stages. Vendors the transformer trunk
the HPT architecture is built on -- SimpleTransformer, MultiheadAttention,
CrossAttention and their blocks -- from the main repo's arc branch.

CrossAttention is what the stems use to compress a variable-length modality
into a fixed latent count before the trunk sees it, so it lands here rather
than with the stems.

Also re-adds get_sinusoid_encoding_table and EinOpsRearrange, which HPT needs
and which this fork dropped along with egomimic/utils/tensor_utils.py during
the graph consolidation. They live in models/cores/hpt_utils.py, scoped to the
callers that want them, rather than resurrecting a 996-line module the fork
deliberately removed.

Two fixes on the way in: the timm import moves off the deprecated
timm.models.layers path so it stops warning on every import, and the helpers
now validate their arguments instead of failing later with a shape error.

Tests cover what the later stages depend on: the trunk preserves token shape,
returns per-block outputs, mixes across positions, matches its own
gradient-checkpointed path, honours an attention mask, and the sinusoid table
handles an odd hidden size and a nonzero start offset.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

First step of decomposing HPT into graph stages. Vendors the transformer trunk
the HPT architecture is built on -- SimpleTransformer, MultiheadAttention,
CrossAttention and their blocks -- from the main repo's arc branch.

CrossAttention is what the stems use to compress a variable-length modality
into a fixed latent count before the trunk sees it, so it lands here rather
than with the stems.

Also re-adds get_sinusoid_encoding_table and EinOpsRearrange, which HPT needs
and which this fork dropped along with egomimic/utils/tensor_utils.py during
the graph consolidation. They live in models/cores/hpt_utils.py, scoped to the
callers that want them, rather than resurrecting a 996-line module the fork
deliberately removed.

Two fixes on the way in: the timm import moves off the deprecated
timm.models.layers path so it stops warning on every import, and the helpers
now validate their arguments instead of failing later with a shape error.

Tests cover what the later stages depend on: the trunk preserves token shape,
returns per-block outputs, mixes across positions, matches its own
gradient-checkpointed path, honours an attention mask, and the sinusoid table
handles an odd hidden size and a nonzero start offset.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant