Skip to content

feat(hpt): stems and trunk as graph stages - #43

Open
AnikethCheluva wants to merge 1 commit into
graph-hpt-depsfrom
graph-hpt-stems
Open

feat(hpt): stems and trunk as graph stages#43
AnikethCheluva wants to merge 1 commit into
graph-hpt-depsfrom
graph-hpt-stems

Conversation

@AnikethCheluva

Copy link
Copy Markdown

Decomposes HPT's input side into two stages instead of one opaque node:

observations --HPTStemStage--> hpt/tokens --HPTTrunkStage--> condition

HPTStemStage runs a PolicyStem per modality, each compressing to a fixed latent
count via cross-attention, and concatenates them along the token axis. That
pooling is what lets a 14-D pose and a 512-D image feature meet in one sequence
the trunk can attend over. Stems run in sorted key order so the token layout is
deterministic -- dict ordering would otherwise let DDP ranks diverge.

HPTTrunkStage attends over those tokens and pools to one vector, written as
"condition". That is deliberately the key the DP and flow heads already read,
so an HPT representation feeds any head in the graph without changing it --
which is the whole point of decomposing rather than wrapping.

Per-domain behaviour keeps the runner ignorant of domains: optional
domain_stems and the trunk's domain embedding both resolve their branch from a
batch key via resolve_homogeneous_scalar, the same mechanism
KeyedFeatureProjection uses.

Two adaptations were needed. nn.ModuleDict rejects "." in keys and batch keys
are dotted paths, so modules are stored under sanitized names with the mapping
kept explicit for readable checkpoint keys. And the cross-attention spec is
accepted as either a DictConfig or a plain dict, since hydra gives one and a
hand-built stem the other.

Tests cover the contracts, latent pooling, sorted ordering, dotted keys, width
mismatches, all three pooling modes, position-embedding bounds, domain
selection and rejection, that the position table is a buffer rather than a
trained parameter, and that the two compose into a runnable subgraph whose
condition is differentiable back to the stems.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

Decomposes HPT's input side into two stages instead of one opaque node:

    observations --HPTStemStage--> hpt/tokens --HPTTrunkStage--> condition

HPTStemStage runs a PolicyStem per modality, each compressing to a fixed latent
count via cross-attention, and concatenates them along the token axis. That
pooling is what lets a 14-D pose and a 512-D image feature meet in one sequence
the trunk can attend over. Stems run in sorted key order so the token layout is
deterministic -- dict ordering would otherwise let DDP ranks diverge.

HPTTrunkStage attends over those tokens and pools to one vector, written as
"condition". That is deliberately the key the DP and flow heads already read,
so an HPT representation feeds any head in the graph without changing it --
which is the whole point of decomposing rather than wrapping.

Per-domain behaviour keeps the runner ignorant of domains: optional
domain_stems and the trunk's domain embedding both resolve their branch from a
batch key via resolve_homogeneous_scalar, the same mechanism
KeyedFeatureProjection uses.

Two adaptations were needed. nn.ModuleDict rejects "." in keys and batch keys
are dotted paths, so modules are stored under sanitized names with the mapping
kept explicit for readable checkpoint keys. And the cross-attention spec is
accepted as either a DictConfig or a plain dict, since hydra gives one and a
hand-built stem the other.

Tests cover the contracts, latent pooling, sorted ordering, dotted keys, width
mismatches, all three pooling modes, position-embedding bounds, domain
selection and rejection, that the position table is a buffer rather than a
trained parameter, and that the two compose into a runnable subgraph whose
condition is differentiable back to the stems.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant