🔍 Seek-and-Solve: DailyClue Benchmark

Benchmarking MLLMs for Visual Clue-Driven Reasoning in Daily Scenarios

Xiaomin Li*, Tala Wang*, Zichen Zhong*, Ying Zhang, Zirui Zheng, Takashi Isobe, Dezhuang Li, Huchuan Lu, You He, Xu Jia†

Dalian University of Technology · WeChat, Tencent Inc. · Tsinghua University

* Equal contribution. † Corresponding author.

---

📖 Abstract

Daily scenarios are characterized by visual richness, requiring Multimodal Large Language Models (MLLMs) to filter noise and identify decisive visual clues for accurate reasoning. Yet, current benchmarks predominantly aim at evaluating MLLMs' pre-existing knowledge or perceptual understanding, often neglecting the critical capability of reasoning.

To bridge this gap, we introduce DailyClue, a benchmark designed for visual clue-driven reasoning in daily scenarios. Our construction is guided by two core principles: (1) strict grounding in authentic daily activities, and (2) challenging query design that necessitates more than surface-level perception. Instead of simple recognition, our questions compel MLLMs to actively explore suitable visual clues and leverage them for subsequent reasoning. We curate a comprehensive dataset spanning 4 major daily domains and 16 distinct subtasks.

🗂️ Dataset Overview

Statistics

Category	# Questions	Formats
Daily Commonsense Reasoning	180	Multiple Choice, Yes/No, Open-ended
Location Identification	200	Open-ended (Country, Region)
Spatial Reasoning	163	Multiple Choice, Yes/No
Scientific Commonsense	123	Multiple Choice, Yes/No, Open-ended
Total	666	—

Data Format

Each sample in the dataset (.json files under dataset/) has the following fields:

Field	Type	Description
image	list[str]	Filename(s) of the associated image(s)
question	str	The question to be answered
clues	str	Ground-truth visual clues annotated by humans (see note below)
ground_truth	str	The correct answer
format	str	Answer format: "Multiple choice", "Yes or no", or "Open-ended"
category_1	str	Primary category (one of the four domains above)
category_2	str	Subtask within the primary category
language	str	Language of the question ("English")

Note on the clues field: This field contains human-annotated ground-truth visual clues that serve two purposes:

1. Insight experiments (Table 2 in our paper): injected as textual context to study the impact of explicit clues on model performance.
2. Rigorous Evaluation Protocol (Figure 7): used to verify whether a model's predicted reasoning clues semantically align with ground-truth clues.

In standard inference (prompt_mode a/b), the clues field is not fed to the model. In prompt_mode c (Clue-guided CoT), the model is prompted to generate its own clues, which are then compared against GT clues.

Directory Structure

dailyclue/
├── dataset/
│   ├── daily_life/           # Images for Daily Commonsense Reasoning
│   │   ├── 1.PNG
│   │   └── ...
│   ├── location/             # Images for Location Identification
│   ├── science/              # Images for Scientific Commonsense
│   ├── spatial/              # Images for Spatial Reasoning
│   ├── daily_life.json       # Annotations for Daily Commonsense Reasoning
│   ├── location.json         # Annotations for Location Identification
│   ├── science.json          # Annotations for Scientific Commonsense
│   └── spatial.json          # Annotations for Spatial Reasoning
├── infer/
│   └── inference.py          # Inference script (OpenAI-compatible API)
├── eval/
│   └── eval.py               # Evaluation script
├── tools/
│   └── generate_leaderboard.py # Aggregates eval results into a markdown table
├── run_demo.sh               # One-click quick start demo script
├── requirements.txt
└── README.md

🏆 Leaderboard

Performance comparison on the DailyClue benchmark. Metrics denote accuracy (%). Bold red = best, underlined = second best.

Open-source MLLMs

Model	Overall	Location	Spatial	Daily Commonsense	Scientific
LLaVA-OneVision-7B	24.47	10.50	34.97	25.56	31.71
LLaVA-OneVision-72B	33.18	15.50	47.85	33.33	42.28
LLaVA-OV-1.5-8B-Instruct	29.43	10.50	47.85	27.78	38.21
InternVL3-8B	31.08	13.50	31.67	31.67	41.46
InternVL3-38B	36.94	17.00	47.85	47.22	39.84
InternVL3-78B	40.84	18.00	54.60	52.78	42.28
InternVL-3.5-38B	36.91	14.00	49.69	43.33	43.90
Qwen2.5-VL-7B	30.63	15.00	39.88	37.22	34.15
Qwen2.5-VL-32B	35.59	21.50	42.94	42.78	38.21
Qwen2.5-VL-72B	40.84	24.50	47.85	48.33	47.15
Qwen3-VL-235B-A22B-Thinking	44.59	23.00	49.08	56.67	56.10
Qwen3-VL-235B-A22B-Instruct	40.69	22.50	46.63	50.00	48.78

Close-source MLLMs

Model	Overall	Location	Spatial	Daily Commonsense	Scientific
Gemini-2.5-Flash 🥉	50.00	32.50	55.83	59.44	56.91
Gemini-2.5-Pro 🥇	56.90	41.50	61.35	62.77	67.48
Claude-3.7-Sonnet	41.14	18.50	57.06	47.22	47.97
Claude-sonnet-4	41.74	22.00	52.15	48.89	49.59
Claude-sonnet-4.5	41.74	21.00	53.99	49.44	47.97
o4-mini	47.00	25.50	58.28	58.33	50.41
GPT-5 🥈	50.90	38.00	57.67	51.67	61.79

Agentic Models

Model	Overall	Location	Spatial	Daily Commonsense	Scientific
DeepEyes-7B	30.93	18.50	44.17	30.00	34.96
VLM-R3	33.18	19.00	42.33	36.11	39.84
TreeVGR-7B	27.78	14.00	40.49	27.18	33.33
REVPT	25.83	6.50	38.04	32.22	31.71
Thyme	46.25	69.00	42.33	29.44	39.02
PyVision	39.48	18.50	47.23	48.33	50.40

Human Baseline

	Overall	Location	Spatial	Daily Commonsense	Scientific
Human Baseline	45.50	19.33	70.67	40.00	52.00

🚀 Quick Start

0. One-Click Demo

To quickly test the pipeline, you can run our all-in-one demo script:

bash run_demo.sh

This script will automatically check dependencies, download the dataset, run a test inference using gpt-4o-mini, evaluate the results, and generate a leaderboard.

1. Installation

git clone https://github.com/xiaominli1020/DailyClue.git
cd DailyClue
pip install -r requirements.txt

2. Download Dataset

The full dataset is hosted on Hugging Face. Please use huggingface_hub to download it:

pip install huggingface_hub
huggingface-cli download --repo-type dataset Crysun/DailyClue --local-dir dataset

3. Configure API

The inference script uses an OpenAI-compatible API interface (works with GPT, Gemini, local vLLM servers, etc.).

# Linux / macOS
export OPENAI_API_KEY="your-api-key"
export OPENAI_API_BASE="https://api.openai.com/v1"   # change for other providers

# Windows (PowerShell)
$env:OPENAI_API_KEY="your-api-key"
$env:OPENAI_API_BASE="https://api.openai.com/v1"

4. Run Inference

python infer/inference.py \
  --dataset ./dataset \
  --model_names "gpt-4o" \
  --prompt_mode "b"

Key Arguments:

Argument	Default	Description
--dataset	dataset	Path to the dataset root directory
--model_names	—	Comma-separated model names (e.g., gpt-4o,gemini-2.5-flash)
--prompt_mode	b	Prompting strategy (see below)
--workers_per_model	3	Concurrent threads per model
--max_concurrent_models	—	Max parallel model processes

Prompt Modes:

Mode	Name	Description
a	Direct	No CoT. Model directly outputs the answer.
b	Standard CoT	Model reasons step-by-step inside <think> tags, then outputs the answer.
c	Clue-guided CoT	Model is required to explicitly identify visual clues (<image clue>) before reasoning. Corresponds to our proposed Seek-and-Solve paradigm.

The inference script produces a JSONL file: response_{model_name}_mode_{mode}.jsonl

5. Run Evaluation

python eval/eval.py \
  --predicted_file response_gpt-4o_mode_b.jsonl \
  --model_name "gpt-4o-mini"

Key Arguments:

Argument	Default	Description
--predicted_file	—	JSONL file produced by inference.py
--output_file	auto	Output path. If omitted, prefixes input filename with eval_result_
--model_name	gpt-4o-mini	Judge model for open-ended question scoring
--overwrite	False	Re-run evaluation from scratch, ignoring existing progress

The evaluator supports automatic resume (checkpoint) by default.

6. Generate Leaderboard

If you have evaluated multiple models (resulting in multiple eval_result_*.jsonl files), you can automatically aggregate their scores and generate a Markdown leaderboard:

python tools/generate_leaderboard.py --dir .

Evaluation Logic by Category:

Category	Format	Metric
Location Identification	Open-ended	Levenshtein-distance region matching
Spatial / Science / Daily Commonsense	Multiple Choice	Exact letter match
Spatial / Science / Daily Commonsense	Yes or No	Exact / contains match
Science / Daily Commonsense	Open-ended	LLM-as-Judge (configurable model)

📊 Key Findings

1. Visual clue identification is the bottleneck: Providing GT clues to models yields substantial accuracy gains (+10~15%), underscoring that failures are often due to missed visual semantics, not reasoning ability.

2. Visual sycophancy is real: Misleading textual clues from other models can override a model's own visual judgment and induce hallucinations.

3. Clue-guided CoT consistently outperforms vanilla CoT: Explicitly mandating visual clue extraction acts as a critical anchor that mitigates reasoning drift.

4. Even SOTA models struggle: The best model (Gemini-2.5-Pro, 56.90%) leaves a large gap compared to the theoretical upper bound when GT clues are provided.

📝 Citation

If you find our work useful in your research, please consider citing:

@article{dailyclue2026,
  title={Seek-and-Solve: Benchmarking MLLMs for Visual Clue-Driven Reasoning in Daily Scenarios},
  author={Li, Xiaomin and Wang, Tala and Zhong, Zichen and Zhang, Ying and Zheng, Zirui and Isobe, Takashi and Li, Dezhuang and Lu, Huchuan and He, You and Jia, Xu},
  journal={arXiv preprint arXiv:2604.14041},
  year={2026}
}

📄 License

This project is licensed under the MIT License.