Language-Guided Driving Perception

This project explores zero-shot approaches for language-guided object selection in autonomous driving scenes.

Given a driving image and a natural language command, the goal is to identify the referred object without task-specific training.

Current implementation includes:

• Grounding DINO
• Grounding DINO + CLIP score fusion
• Qwen2.5-VL
• Evaluation on the Talk2Car dataset

---

Example Results

Example 1

Ground truth and all three methods selected the same vehicle.

---

Example 2

Grounding DINO and Grounding+CLIP selected the correct truck, while Qwen selected a different candidate.

---

Project Structure

language-guided-driving-perception
│
├── src/
├── scripts/
├── data/
├── outputs/
├── demo.py
├── requirements.txt
└── README.md

---

Installation

git clone https://github.com/rasoulameri/language-guided-driving-perception.git

cd language-guided-driving-perception

pip install -r requirements.txt

---

Evaluation

Run

python scripts/evaluate_zero_shot_baselines.py

The script evaluates

• Grounding DINO
• Grounding + CLIP
• Qwen2.5-VL

and reports IoU, candidate ranking, and qualitative visualizations.

---

Current Status

✔ Version 1

• Zero-shot Grounding DINO baseline
• Grounding + CLIP fusion
• Qwen2.5-VL baseline
• Quantitative evaluation
• Qualitative visualization

---

Next Steps

Version 2

• Trainable candidate ranking module

Version 3

• Fine-tuning vision-language models on Talk2Car