Summary
Running python eval_interhuman.py with the released intermoe-interhuman checkpoint crashes during the first replication, partway through the multimodality (MM) sample generation loop inside BatchEvaluationDataset.init. The iteration index where it crashes varies between runs because of dataloader shuffling (e.g. 23it or 29it), but the failure is deterministic and the root cause is the same.
Environment
Repo: current main (commit e5edc47)
Python 3.10, PyTorch 2.1.2+cu121
Config: checkpoints/intermoe-interhuman/config.yaml (released denoiser + VAE checkpoints)
Reproduction
python eval_interhuman.py
The script completes the first BatchEvaluationDataset generation tqdm (18/18 batches), then crashes inside the second tqdm (the mm_dataloader loop) at the first iteration whose index is in mm_idxs.
Stack trace (key frames)
File "evaluators/evaluator_interhuman.py", line 99, in __init__
batch = self.model.forward_test(batch)
File "models/intermoe.py", line 72, in forward_test
batch.update(self.decode_motion(batch))
File "models/nets.py", line 263, in forward
output = self.diffusion_test.ddim_sample_loop(...)
File "models/cfg_sampler.py", line 22, in forward
out = self.model(x_combined, timesteps_combined, cond=cond, mask=mask)
File "models/nets.py", line 111, in forward
h_a, ones_a, pred_c_a = block(h_a_prev, h_b_prev, emb, key_padding_mask)
...
File ".../torch/nn/functional.py", line 5379, in multi_head_attention_forward
assert key_padding_mask.shape == (bsz, src_len), ...
AssertionError: expecting key_padding_mask shape of (60, 41), but got torch.Size([2, 41])Root cause
In evaluators/evaluator_interhuman.py, the multimodality loop expands text by mm_num_repeats but leaves motion_lens (and, for the VAE branch, motions) at batch-size 1:
# evaluators/evaluator_interhuman.py (current)
for i, data in tqdm(enumerate(mm_dataloader)):
if i not in mm_idxs:
continue
name, text, motion1, motion2, motion_lens = data
batch = {}
batch["text"] = list(text) * mm_num_repeats # 30 entries
batch["motion_lens"] = motion_lens + 3 # still size 1
if is_vae:
motion_lens = (motion_lens // 4) * 4
batch["motion_lens"] = motion_lens # still size 1
batch["motions"] = torch.cat([motion1, motion2], dim=-1) # still batch 1
else:
batch["motion_lens"] = motion_lens + 3 # still size 1What happens downstream:
InterMoE.text_process encodes 30 texts → cond.shape = (30, 768).
LatentInterDiffusion.forward sets B = cond.shape[0] = 30, but builds the mask from motion_lens of length 1: seq_mask = generate_src_mask(T, motion_lens // 4) → (1, T, 2).
ddim_sample_loop samples x with shape (30, T, 2D).
ClassifierFreeSampleModel.forward concatenates along the batch dim: x → (60, T, 2D), but mask → (2, T, 2) (and cond → (60, ...)).
Inside LatentInterDenoiser.forward, mask = mask[..., 0] → (2, T), which is passed as key_padding_mask to a MultiheadAttention whose query batch is 60. PyTorch's MHA asserts and fails.
Suggested fix
Repeat motion_lens (and motions in the VAE path) to match mm_num_repeats, so the model input is internally consistent:
# evaluators/evaluator_interhuman.py
batch["text"] = list(text) * mm_num_repeats
if is_vae:
motion_lens = (motion_lens // 4) * 4
batch["motion_lens"] = motion_lens.repeat(mm_num_repeats)
batch["motions"] = torch.cat(
[motion1, motion2], dim=-1
).repeat(mm_num_repeats, 1, 1)
else:
batch["motion_lens"] = (motion_lens + 3).repeat(mm_num_repeats)
After this change, eval_interhuman.py runs to completion (Replication 0 produces MM Distance / R-precision / FID / Diversity / Multimodality without errors).Why it didn't show up earlier (guess)
Probably an upstream copy/paste leftover from a path where the same batch dimension was either implicit in motions or already broadcast elsewhere. The non-MM generation loop right above uses batch_size=64 and supplies a length-matched motion_lens, so the bug is exclusive to the MM branch.
Summary
Running python eval_interhuman.py with the released intermoe-interhuman checkpoint crashes during the first replication, partway through the multimodality (MM) sample generation loop inside BatchEvaluationDataset.init. The iteration index where it crashes varies between runs because of dataloader shuffling (e.g. 23it or 29it), but the failure is deterministic and the root cause is the same.
Environment
Repo: current main (commit e5edc47)
Python 3.10, PyTorch 2.1.2+cu121
Config: checkpoints/intermoe-interhuman/config.yaml (released denoiser + VAE checkpoints)
Reproduction
The script completes the first BatchEvaluationDataset generation tqdm (18/18 batches), then crashes inside the second tqdm (the mm_dataloader loop) at the first iteration whose index is in mm_idxs.
Stack trace (key frames)
Root cause
In evaluators/evaluator_interhuman.py, the multimodality loop expands text by mm_num_repeats but leaves motion_lens (and, for the VAE branch, motions) at batch-size 1:
What happens downstream:
InterMoE.text_process encodes 30 texts → cond.shape = (30, 768).
LatentInterDiffusion.forward sets B = cond.shape[0] = 30, but builds the mask from motion_lens of length 1: seq_mask = generate_src_mask(T, motion_lens // 4) → (1, T, 2).
ddim_sample_loop samples x with shape (30, T, 2D).
ClassifierFreeSampleModel.forward concatenates along the batch dim: x → (60, T, 2D), but mask → (2, T, 2) (and cond → (60, ...)).
Inside LatentInterDenoiser.forward, mask = mask[..., 0] → (2, T), which is passed as key_padding_mask to a MultiheadAttention whose query batch is 60. PyTorch's MHA asserts and fails.
Suggested fix
Repeat motion_lens (and motions in the VAE path) to match mm_num_repeats, so the model input is internally consistent:
Why it didn't show up earlier (guess)
Probably an upstream copy/paste leftover from a path where the same batch dimension was either implicit in motions or already broadcast elsewhere. The non-MM generation loop right above uses batch_size=64 and supplies a length-matched motion_lens, so the bug is exclusive to the MM branch.