MNAD Clean-Room Reimplementation

Clean-room research implementation of Park et al., "Learning Memory-guided Normality for Anomaly Detection" (CVPR 2020), built from the paper itself — not a port of the official implementation.

The architecture is frozen (v1) — see docs/FROZEN_ARCHITECTURE.md for exactly what that means, the evidence behind it, and how to reproduce it. The commands below show both the bare paper-faithful invocation and the frozen recipe's flags; use the frozen recipe (--learned_metric at train and eval time, --smooth_window 15 --smooth_mode mean at eval time) unless you specifically need the unmodified paper baseline for comparison.

Setup

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install torch torchvision   # CUDA wheel; see requirements.txt note

Known gotcha: if source .venv/bin/activate leaves python unresolvable in your shell, don't fight it — every command below also works called as .venv/bin/python ... directly, which is what the checked-in shell scripts (run_phase3_baseline.sh, etc.) do for exactly this reason.

Commands

All commands assume you're in the repo root (cd here first). Every script takes --help for the full flag list; below is what each one is for.

Run the test suite

.venv/bin/python -m pytest tests/ -q

Phase 2 component verification (memory, losses, model shapes, anomaly-score math) — fast, no dataset needed. Run this before trusting any change.

Train a single model

Frozen recipe (recommended — see docs/FROZEN_ARCHITECTURE.md):

.venv/bin/python experiments/baseline/train.py \
  --data_root dataset/ped2/training/frames --task prediction \
  --seed 42 --out_dir runs/ped2_prediction_seed42 --learned_metric

Bare paper-faithful baseline (drop --learned_metric):

.venv/bin/python experiments/baseline/train.py \
  --data_root dataset/ped2/training/frames --task prediction \
  --seed 42 --out_dir runs/ped2_prediction_seed42

Trains one model to convergence (60 epochs by default) on one dataset/task/ seed. Useful flags: --task {reconstruction,prediction}, --epochs, --batch_size, --seed, --lambda_c/--lambda_s/--alpha (loss weights, override the paper-default per task), --num_memory_items, --write_grad_mode {detached,graph} (graph is intentionally unimplemented — see ID2a), --diag_every_n_epochs (0 disables per-epoch diagnostics), --learned_metric (frozen recipe's learned per-dimension distance metric, ID13/E10 — must match at eval time or the checkpoint won't load).

Evaluate a trained checkpoint

Frozen recipe (must match how the checkpoint was trained):

.venv/bin/python experiments/baseline/evaluate.py \
  --checkpoint runs/ped2_prediction_seed42/checkpoint.pt \
  --testing_frames_dir dataset/ped2/testing/frames \
  --gt_mat dataset/ped2/ped2.mat --task prediction --learned_metric \
  --smooth_window 15 --smooth_mode mean \
  --out runs/ped2_prediction_seed42/eval.json \
  --diagnostics_out runs/ped2_prediction_seed42/eval_diagnostics.json

Runs the trained model over every test video, computes the Eq. 14-17 abnormality score, and reports both pooled_auc (this project's default) and macro_auc. --diagnostics_out is optional — omit it to skip the extra per-video memory/representation snapshot. --disable_test_time_update turns off the Eq. 6-7 online memory adaptation for a comparison run. --learned_metric must match how the checkpoint was trained (ID13/E10). --smooth_window N --smooth_mode mean applies temporal smoothing to the score sequence before AUC (default N=1 is a no-op; docs/EXPERIMENT_LOG.md E7 confirmed window=15/mean as a real +1.5pp average AUC gain on Ped2, reproducible across seeds — window=15 is Ped2-tuned, worth re-sweeping per dataset via experiments/diagnostics/temporal_smoothing_probe.py).

Run the full 3-seed baseline sequence

bash experiments/baseline/run_phase3_baseline.sh

Trains + evaluates seeds 42, 1, 2 back-to-back on Ped2/prediction (one seed failing doesn't block the next). This is the long-running one — expect hours, not minutes. Run it in the background:

bash experiments/baseline/run_phase3_baseline.sh > runs/phase3_baseline.log 2>&1 &

B1 — causal ablation (does the decoder actually use memory / skips?)

.venv/bin/python experiments/diagnostics/causal_ablation.py \
  --checkpoints runs/ped2_prediction_seed42/checkpoint.pt runs/ped2_prediction_seed1/checkpoint.pt \
  --testing_frames_dir dataset/ped2/testing/frames --gt_mat dataset/ped2/ped2.mat \
  --task prediction --out runs/causal_ablation.json

Takes already-trained checkpoints (no retraining) and zeroes out the memory-read and/or skip-connection pathways at forward time, reporting how much the actual output pixels change and how the AUC moves under each of the 4 combinations. Minutes, not hours.

B2 — separateness-loss-weight sweep

.venv/bin/python experiments/ablations/separateness_sweep.py --stage screen
# later, once a value looks worth trusting:
.venv/bin/python experiments/ablations/separateness_sweep.py --stage confirm --confirm_values 0.3

Retrains the baseline recipe at different --lambda_s values (screen: one seed across a grid; confirm: multiple seeds on selected values only). Each run is a full training + eval, so this is as expensive as running train.py/evaluate.py that many times over.

Logs & outputs — where to look

Everything lands under runs/ (gitignored — these are large, regenerable artifacts, not source), organized by what the run is, not just its name:

runs/
  baseline/            # E1 — the established Ped2 noise floor (canonical reference)
  ablations/           # E2/E3 — causal ablation, separateness-weight sweep
  signals_cache/        # shared raw psnr/dist cache (experiments/diagnostics/collect_signals.py)
  candidates/<name>/   # an architecture/mechanism candidate currently under test
  confirmed/           # candidates kept in the frozen recipe (E7 smoothing, E10 learned_metric)
  failed/              # ruled-out candidates, kept for the record (E4/E5/E6/E8/E9)

A single training run's own directory (wherever it lives) always has the same shape:

<run_dir>/
  manifest.json           # config, git commit, torch/cuda versions, checkpoint SHA-256
  history.json            # one entry per epoch: losses, lr, and (by default) a
                           # "diagnostics" block — memory health, representation
                           # health, separate_grad_norm
  checkpoint.pt            # only written after all epochs finish
  eval.json                # pooled_auc / macro_auc / per_video_auc (after evaluate.py)
  eval_diagnostics.json    # per-video memory/representation snapshot (if requested)

experiments/baseline/run_phase3_baseline.sh additionally writes a plain text progress log next to whatever --out_dir you point it at (e.g. runs/baseline/phase3_baseline.log); experiments/ablations/separateness_sweep.py writes {screen,confirm}_manifest.json indexing the sub-runs it launched (e.g. runs/ablations/separateness_sweep/screen_manifest.json).

Watching a run live:

tail -f runs/baseline/phase3_baseline.log          # the orchestration script's own progress banner
tail -f runs/baseline/ped2_prediction_seed42/history.json    # rewritten whole-file every epoch, not
                                                                # appended — tail -f will show it jump

For the per-epoch numbers as they land, prefer polling instead of tail -f on history.json (it's a JSON array, rewritten in full each epoch):

watch -n 5 '.venv/bin/python -c "import json; d=json.load(open(\"runs/baseline/ped2_prediction_seed42/history.json\")); print(len(d), \"epochs done —\", d[-1])"'

Reading a finished result:

.venv/bin/python -c "import json; print(json.dumps(json.load(open('runs/baseline/ped2_prediction_seed42/eval.json')), indent=2))"

or just cat/open the .json files directly — they're all small, indented, human-readable JSON.