QuISE: Defense against Typographic Attacks on VLMs via Query-Irrelevant Semantic Editing

Typographic attacks pose a critical threat to vision-language models (VLMs) by injecting misleading text into images and causing models to rely on adversarial textual cues rather than visual evidence. Existing defenses often require model-specific modifications, additional training, or access to int...