IEEE TPAMI
PRIMEdit: Probability Redistribution for Instance-aware Multi-object Video Editing with a New Benchmark Dataset

🚀 A zero-shot multi-instance video editing framework: one mask and one caption per object, no training, no global edit caption.

Samuel Teodoro^1*, Agus Gunawan^1*, Soo Ye Kim², Jihyong Oh^3†, Munchurl Kim^1†
1KAIST, ²Adobe Research, ³Chung-Ang University
^*Equal Contribution, ^†Co-Corresponding Authors

💿 Introduction

PRIMEdit edits multiple objects in a single video at once. You supply one mask and one caption per instance; each instance changes as asked and the rest of the frame does not.

Two modules do the work:

• IPR (Instance-centric Probability Redistribution) keeps each edit inside its own mask.
• DMS (Disentangled Multi-instance Sampling) stops one instance's edit from leaking onto another.

Neither requires training or per-video optimization, and unlike prior multi-instance methods, PRIMEdit needs no global edit caption describing the whole scene.

📑 Table of Contents

• 🔥 News
• 🔧 Quick Start
• 🗂 Repository Layout
• 📁 Data Format
• 🚀 Inference
• 📊 Evaluation
• 🚧 TODO
• 🙏 Acknowledgments
• 📜 License
• 📮 Contact
• 📄 Citation

🔥 News

• 2026.08.10: 📚 We released the MIVE Dataset! Download it here.
• 2026.08.10: 🎉 We released the PRIMEdit inference code.
• 2026.08.07: 🎉 PRIMEdit was accepted to IEEE TPAMI!

🔧 Quick Start

1. Clone the repository:

   git clone https://github.com/KAIST-VICLab/PRIMEdit.git
   cd PRIMEdit

2. Install dependencies:

   # 1. Create and activate a conda environment
   conda create -n primedit python=3.10
   conda activate primedit
   
   # 2. Install PyTorch (choose the build matching your CUDA)
   pip install torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu121
   
   # 3. Install the remaining dependencies
   pip install -r requirements.txt

3. Download models:

   python download_weights.py

   Weights land in ~/.cache/huggingface.

4. Download the MIVE dataset:

   python download_dataset.py

   This fetches MIVE-Dataset.zip (1.1 GB) and extracts it to PRIMEdit/MIVE-Dataset/, which is where inference.py and evaluation/compute_cia.py look by default. The archive holds 200 videos: 3,633 frames and 27,741 instance masks.

   To download it by hand instead, grab MIVE-Dataset.zip and unzip it in the repository root.

🗂 Repository Layout

PRIMEdit/
├─ inference.py                      # Entry point: edits every video in a directory
├─ download_weights.py               # Optional cache warm-up; single source of model ids
├─ download_dataset.py               # Fetches and unpacks MIVE-Dataset/
├─ primedit/
│  ├─ pipeline.py                    # PRIMEditPipeline: DMS sampling loop, re-inversion, latent fusion
│  ├─ attention_processor.py         # IPR: cross-attention probability redistribution
│  └─ utils.py                       # Caption tag parsing and per-instance mask loading
├─ evaluation/
│  └─ compute_cia.py                 # Cross-Instance Accuracy (CIA) Score
├─ scripts/
│  └─ run_inference.sh               # Multi-instance editing with paper defaults
├─ MIVE-Dataset/                     # Unzipped MIVE dataset (see Quick Start step 4)
└─ requirements.txt

📁 Data Format

inference.py reads the dataset from MIVE-Dataset/ by default:

MIVE-Dataset/
├─ frames/
│  └─ <video_id>/                    # 00000000.png, 00000001.png, ... (sorted order matters)
├─ instance_mask/
│  └─ <video_id>/
│     └─ <tag>-<class>-<id>/         # e.g. 13158411-person-60, one .png per frame
├─ caption/
│  ├─ source/caption.json            # source captions with instance tags
│  └─ target/caption.json            # target captions, same tags, edited nouns
└─ panoptic_gt.json                  # panoptic ground truth, used for evaluation

To use your own data, keep this layout and pass --frames_path, --masks_path, --source_caption_json and --target_caption_json.

Both caption files are JSON lists of {"video_id": ..., "text": ...}. Instances are marked inline with numeric tags, and the same tag numbers must appear in both files:

[
  {"video_id": "1014_rzVzaDhIkzo",
   "text": "A <13158411>man</13158411> plays an <13958912>electric guitar</13958912>."}
]

The target file repeats those tags with the edited nouns, e.g. <13158411>statue</13158411>. A tag is matched to its mask directory by substring, so each tag number must appear in exactly one directory name under that video. Instances whose target phrase equals their source phrase are treated as unedited and skipped.

🚀 Inference

sh scripts/run_inference.sh

or directly — with the dataset unzipped at MIVE-Dataset/, every data path is already the default:

python inference.py --output_dir outputs/primedit

Useful arguments:

Argument	Default	Meaning
--video_ids	all	Edit only the listed video ids
--ipr_steps_ratio	0.1	Fraction of denoising steps applying IPR. 0.0 disables IPR
--lambda_warmup	0.5	λ — warm-up amplitude for the SOS probability moved onto the instance phrase
--lambda_r	0.5	λ_r — share of λ_S redistributed to the text tokens vs the end token
--sns_steps_ratio	0.4	Fraction of denoising steps using Series Noise Sampling before switching to PNS
--reinversion_steps	2	l — re-inversion steps in DMS
--guidance_scale	12.5	Classifier-free guidance; must be > 1.0
--controlnet_scale	0.4	Depth ControlNet conditioning strength

📊 Evaluation

Compute the Cross-Instance Accuracy (CIA) Score on a directory of edited videos. For each frame, every edited instance is cropped by its ground-truth mask and scored with CLIP against the target captions of all edited instances in that video; a row counts as correct only when an instance's best-matching caption is its own. This is what penalizes editing leakage.

python evaluation/compute_cia.py \
  --edited_frames_root outputs/primedit \
  --output_json outputs/cia.json

The CLIP ViT-L/14@336px checkpoint is downloaded on first use into --clip_download_root. Scores are averaged over frames, then over videos, matching the paper's protocol.

🚧 TODO

• Release paper.
• Release inference code.
• Release the MIVE dataset.
• Release evaluation code for the CIA Score.

🙏 Acknowledgments

We thank the open-source community, in particular AnimateDiff, ControlNet, Diffusers and DenseDiffusion, whose work this implementation builds on.

📜 License

This project is licensed under the Apache License 2.0.

📮 Contact

For any questions, please feel free to reach out to Samuel Teodoro @sateodoro, email: sateodoro@kaist.ac.kr

📄 Citation

If you find our work useful, please consider citing:

@misc{teodoro2026primedit,
  title={PRIMEdit: Probability Redistribution for Instance-aware Multi-object Video Editing with a New Benchmark Dataset},
  author={Teodoro, Samuel and Gunawan, Agus and Kim, Soo Ye and Oh, Jihyong and Kim, Munchurl},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2026}
}