Publication
AtManRL: Towards Faithful Reasoning via Differentiable Attention Saliency
Max Henning Höth; Kristian Kersting; Björn Deiseroth; Letitia Parcalabescu
In: Computing Research Repository eprint Journal (CoRR), Vol. abs/2604.16158, Pages 1-14, arXiv, 2026.
Abstract
Large language models (LLMs) increasingly rely on chain-of-thought (CoT) reason-
ing to solve complex tasks. Yet ensuring that the reasoning trace both contributes
to and faithfully reflects the processes underlying the model’s final answer, rather
than merely accompanying it, remains challenging. We introduce ATMANRL, a
method that leverages differentiable attention manipulation to learn more faithful
reasoning through reinforcement learning. By training an additive attention mask
that identifies tokens in the CoT crucial for producing correct answers, we derive
a saliency reward signal that encourages the model to generate reasoning traces
that genuinely influence its final predictions. We integrate this saliency reward
with outcome-based rewards within the GRPO framework to jointly optimize for
correctness and interpretability. Experiments on GSM8K and MMLU with Llama-
3.2-3B-Instruct demonstrate that our approach can identify influential reasoning
tokens and enable training more transparent reasoning models.