Publikation
Evaluating Explanation-Driven Vision–Language Reasoning via Generation Order Interventions
Siting Liang; Luca Rippe; Omar Adjali; Daniel Sonntag
In: IJCAI. IJCAI-ECAI 2026 Workshop on Explainable Artificial Intelligence (XAI) (IJCAI-ECAI-2026), August 15-21, Bremen, Germany, IJCAI, 2026.
Zusammenfassung
Natural language explanation generation serves as a key mechanism for exposing and evaluating vision–language reasoning. Prior work on explanation-driven vision–language models predominantly follows a post-hoc (answer-first) paradigm, implicitly suggesting that supervised rationales can reflect underlying reasoning processes. In contrast, modern large vision–language models increasingly exhibit a rationale-first generation tendency, which more closely aligns with structured, stepwise reasoning. In this work, we systematically evaluate whether explanations are causally tied to model predictions within a single generation step under a controlled experimental setup, explicitly eliminating unnecessary chain-of-thought or other intermediate reasoning processes across knowledge-intensive QA, visual entailment, and compositional grounding benchmarks. We find that larger models emerge as a prerequisite for reliably supporting rationale-first reasoning at scale. However, answer-first generation is less prone to format-related errors in structured output. Overall, explanation ordering, model scale and pre-training knowledge, task-specific fine-tuning, and task structure jointly influence both prediction accuracy and reasoning faithfulness.