SPARTA: Spectral Prompt Agnostic Adversarial Attack on Medical Vision-Language Models (MICCAI'25 - UNSURE)

SPARTA: Spectral Prompt Agnostic Adversarial Attack on Medical Vision-Language Models

Asif Hanif, Zaigham Zaheer, Salman Khan, Fahad Shahbaz Khan and Rao Anwer

---

Overview of SPARTA SPARTA transforms the input image into the frequency domain, where it applies learnable multiplicative noise to perturb the spectrum before reconstructing the image via inverse-transform. On the text side, it introduces learnable context within the token embedding space and prepends it to the actual text prompt. Learnable parameters are optimized within a competitive optimization framework to enhance cross-prompt transferability.

---

Abstract

Medical Vision-Language Models (Med-VLMs) are gaining popularity in different medical tasks, such as visual question-answering (VQA), captioning, and diagnosis support. However, despite their impressive performance, Med-VLMs remain vulnerable to adversarial attacks, much like their general-purpose counterparts. In this work, we investigate the cross-prompt transferability of adversarial attacks on Med-VLMs in the context of VQA. To this end, we propose a novel adversarial attack algorithm that operates in the frequency domain of images and employs a learnable text context within a max-min competitive optimization framework, enabling the generation of adversarial perturbations that are transferable across diverse prompts. Evaluation on three Med-VLMs and four Med-VQA datasets shows that our approach outperforms the baseline, achieving an average attack success rate of 67% (compared to baseline's 62%).

---

---

Updates 🚀

• July 17, 2025 : Accepted in MICCAI 2025 - UNSURE    🎊 🎉

Results 🔬

Citation ⭐

If you find our work or this repository helpful, please consider giving a star ⭐ and citation.

@inproceedings{hanif2025sparta,
  title={SPARTA: Spectral Prompt Agnostic Adversarial Attack on Medical Vision-Language Models},
  author={Hanif, Asif and Zaheer, Zaigham and Khan, Salman and Khan, Fahad Shahbaz and Anwer, Rao},
  booktitle={International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging},
  pages={69--80},
  year={2025},
  organization={Springer}
}

Contact 📫

Should you have any questions, please create an issue on this repository or contact us at asif.hanif@mbzuai.ac.ae