🌍 INSIGHT: Structured Multimodal Knowledge Modeling for Vision-Guided Task-Oriented Dialogue

[ 🤗Huggingface ]

This repository contains the official implementation of our INSIGHT framework and the accompanying multimodal VISTA dataset.

coming soon

Overall, INSIGHT enhances performance of vision-guided dialogue understanding and knowledge retrieval, while the VISTA dataset provides a more challenging evaluation benchmark for image-text dialogue modeling. Together, they advance the development of task-oriented dialogue systems toward more realistic and higher-quality image-text interaction scenarios.

🔑 Key Contributions

1. INSIGHT Framework: An end-to-end vision-guided retrieval-generation framework INSIGHT is proposed, which unifies cross-modal alignment, multimodal retrieval, and generation in an end-to-end manner, thereby improving response quality.
2. Graph-structured Entity Modeling Strategy: A graph-structured entity modeling strategy is proposed for multimodal KBs with one-to-many attribute mappings, where visual diversity and structural uncertainty are preserved through distributed latent semantic modeling.
3. VISTA Dataset: A novel multimodal dataset named VISTA is constructed, which is the first to feature image-driven user questions and one-to-many attribute-value mappings, providing a more challenging and realistic benchmark.

🧠 INSIGHT Framework

📚 VISTA Dataset

we construct VISTA, a novel multi-domain image-guided dataset covering restaurant reservation, hotel booking, and travel assistance. Our VISTA dataset introduces an image-centric setting in which user questions are mainly expressed through images, and the knowledge base contains multiple visually similar images grouped under a single image attribute to form a one-to-many mapping relationship.

🚀 Requirements

To run the evaluation and processing scripts:

pip install sacrebleu