Bidirectional Learning of Facial Action Units and Expressions via Structured Semantic Mapping across Heterogeneous Datasets

TL;DR

SSM presents a Structured Semantic Mapping (SSM) framework for bidirectional learning between Facial Action Units (AUs) and Facial Expressions (FEs) under heterogeneous datasets. Unlike prior one-way transfer (AU → FE), SSM enables mutual enhancement (AU ↔ FE) without requiring joint annotations, addressing inconsistencies in annotation granularity and data domains.

🚧 This paper is currently under review...

🚧 This code repository will be continuously updated...

🔑 Key Ideas

• Bidirectional Learning across Tasks Establishes reciprocal knowledge transfer between fine-grained AUs and coarse-grained expressions.
• Textual Semantic Prototypes (TSP) Builds structured semantic anchors from textual descriptions with learnable prompts.
• Dynamic Prior Mapping (DPM) Learns a bidirectional, data-driven association matrix guided by FACS priors for cross-task alignment.
• Heterogeneous Joint Learning Enables training across datasets with different annotation formats (frame-level vs. clip-level).

🚀 Highlights

• First systematic study of AU ↔ FE bidirectional learning under heterogeneous supervision
• Achieves state-of-the-art performance on multiple AU and DFER benchmarks
• Demonstrates that expression semantics can improve AU detection, not just the reverse
• Strong cross-dataset generalization and zero-shot transfer ability

📊 Benchmarks for Experiments

• AU datasets: BP4D, DISFA
• DFER datasets: DFEW, FERV39K, MAFW

SSM consistently outperforms single-task and baseline models across diverse dataset combinations.

Installation

conda env create -f environment.yml
conda activate ssm

Data

Prepare the datasets following docs/DATA.md. The experiment split files are included in splits/.

Training

The six BP4D/DISFA and DFEW/FERV39K/MAFW combinations are configured in configs/.

python train.py --config configs/bp4d_dfew.json

Each run jointly trains the expression and AU branches. Use --emotion-fold and --au-fold for one fold pair, or --all-folds for all configured pairs. The release configurations reproduce the original three-GPU DataParallel setup.

Fine-tune weights

Representative fine-tuned weights are provided through Google drive, and additional weights will be uploaded continuously. Each released checkpoint contains the complete joint model and evaluates both tasks.

Evaluation

python evaluate.py \
  --config configs/bp4d_dfew.json \
  --checkpoint /path/to/best_emotion.pth \
  --emotion-fold 5 \
  --au-fold 1

One evaluation command reports both expression (uar, war) and AU (f1, auc) metrics. No second task-specific checkpoint is required. The configuration and fold pair must match the checkpoint.

📎 Citation

@article{li2026bidirectional,
  title={Bidirectional Learning of Facial Action Units and Expressions via Structured Semantic Mapping across Heterogeneous Datasets},
  author={Li, Jia and Zhang, Yu and Chen, Yin and Hu, Zhenzhen and Li, Yong and Hong, Richang and Shan, Shiguang and Wang, Meng},
  journal={arXiv preprint arXiv:2604.10541},
  year={2026}
}

Acknowledgments

This repository builds on DFER-CLIP, OpenAI CLIP, and CoOp. We sincerely thank them for their open-source contributions！