CVMC — Active Speaker Detection

Given video with one or more visible faces, determine which face is currently speaking by fusing the audio track with per-face visual crops.

Interactive Demo — Live in-browser active speaker detection →

Point your webcam at one or more faces and watch a live audio-visual correlator decide who is speaking — with the internal signals (audio energy, per-face lip motion) plotted in real time.

---

Built during the UTD Summer Research Internship in Dr. Yapeng Tian's Computer Vision Lab (2024), studying training optimization — VGG16 transfer learning and selective weight freezing — for maximizing mAP on AVA Active Speaker format data. Also showcased through Freetail Hackers. Results and experiments are in the research presentation.

The task

Audio alone cannot solve this: an off-screen narrator produces speech with no speaking face, and with several faces in frame the waveform doesn't say which one is talking. Vision alone can't either: mouths move when chewing, smiling, or lip-syncing. Active speaker detection (ASD) is inherently multimodal — the model must decide whether this particular face's motion is what is producing the audio being heard right now.

flowchart LR
    A[Audio track] -->|MFCC features| F{Fuse}
    V[Face crop sequence] -->|pixels| F
    F --> S[speaking / not speaking<br/>per face, per frame]

Loading

Architecture

Two independent CNN towers — one per modality — whose flattened outputs are concatenated and classified by a fully connected head (model.py):

AUDIO                                     VISUAL
3 s waveform, mono                        112×112 grayscale face crop
  └─ MFCC (13 coefficients)                 │
     ~299×13 "spectrogram image"            │
        │                                   │
  Conv2d 1→64  + BN + MaxPool         Conv2d 1→64  + BN + MaxPool
  Conv2d 64→128 + BN + MaxPool        Conv2d 64→128 + BN + MaxPool
  Conv2d 128→128 + BN + MaxPool       Conv2d 128→128 + BN + MaxPool
  Conv2d 128→128 → Flatten            Conv2d 128→128 → Flatten
        │                                   │
     4736 dims  ──────── concat ──────  25088 dims
                           │
                  Linear 29824 → 1024
                  Linear  1024 → 128
                  Linear   128 → 2      (cross-entropy)

Honest notes on the code as it stands:

• Fusion is concatenation of the two flattened embeddings; createFusionModel is a placeholder (pass) left open for learned fusion.
• The research direction explored replacing the visual tower with VGG16 via transfer learning, freezing selected layers to control what gets fine-tuned; that experimentation lives in the lab's training runs and the presentation, not in this file.
• evaluate_network computes frame-level accuracy; the training loop logs that value under the mAP label when selecting checkpoints.
• Runs on CPU or GPU — the model picks cuda when available and falls back to cpu.

Repository structure

CVMC/
├── model.py                    # Two-stream audio-visual CNN + train/eval loops
├── trainer.py                  # Entry point: args, data loading, epoch loop, checkpointing
├── dataLoader_Image_audio.py   # Sample-level loader: one face crop + 3 s audio window per row
├── dataLoader_track.py         # Track-level loader variant: whole face tracks, length-grouped batches
├── detect_speaker_live.py      # Flask webcam server: Haar-cascade faces + ResNet18 scoring scaffold
├── requirements.txt            # ASD pipeline dependencies
├── docs/
│   └── index.html              # Live in-browser demo (GitHub Pages)
├── tools/
│   └── subtitle_app/           # Hackathon-era Flask subtitle generator (own README + deps)
├── archive/                    # Legacy files preserved with notes
└── images/                     # Figures and demo video used by pages

Data format

The loaders consume datasets in the AVA Active Speaker style: a labels CSV plus per-video audio files and per-entity face-crop folders. As implemented in dataLoader_Image_audio.py, each CSV row is comma-separated and the loader reads:

Column index	Meaning	Used for
0	video id	locating <audioPath>/<video_id>.wav and the face folder
1	frame timestamp (s)	centering the 3 s audio window; picking <timestamp>.jpg
7	entity id	face-track folder <visualPath>/<video_id>/<entity_id>/
8	label (0/1)	not-speaking / speaking target

Per sample, the loader extracts a 3-second mono audio window centered on the frame timestamp (zero-padded at clip edges) and converts it to 13-coefficient MFCCs, and loads the face crop for that timestamp as 112×112 grayscale (random flip/crop/rotate augmentation at train time). Expected dataset layout, following the paths trainer.py constructs:

<datasetPath>/
├── csv/train_labels.csv, val_labels.csv
├── orig_audios/<video_id>.wav
└── clips_videos/{train,val}/<video_id>/<entity_id>/<timestamp>.jpg

Setup

git clone https://github.com/Fangedan/CVMC.git
cd CVMC
pip install -r requirements.txt

Works on CPU; uses CUDA automatically when available.

Train

python trainer.py --datasetPath /path/to/your/AVA-format-dataset \
                  --batchSize 32 --maxEpoch 25 --savePath exps/exp1

Checkpoints and a score.txt log are written to --savePath; the best model is kept as best.model.

Evaluate

python trainer.py --evaluation --eval_model_path exps/exp1/model/best.model \
                  --datasetPath /path/to/your/AVA-format-dataset

Live webcam scaffold

python detect_speaker_live.py    # serves http://localhost:5000/video_feed and /speaker_status

This is a demo scaffold: it detects faces with a Haar cascade and scores them with an ImageNet-pretrained ResNet18 whose 2-class head is untrained — it shows the serving pipeline (capture → detect → crop → classify → stream), not trained ASD inference. For a real sense of the task, use the browser demo.

Acknowledgments

• Dr. Yapeng Tian and the UTD Computer Vision Lab — mentorship and research direction (UTD Summer Research Internship, 2024)
• Freetail Hackers — showcase venue
• AVA Active Speaker dataset format — Roth et al., AVA Active Speaker