Publication
Interpretability in Activation Space Analysis of Transformers: A Focused Survey
Soniya Vijayakumar
In: AIMLAI' 22 Workshop. ACM International Conference on Information and Knowledge Management (CIKM-2022), October 17-21, Atlanta, Georgia, USA, CEUR Workshop Proceeding, Vol. 3318, CEUR, 2022.
Abstract
The field of natural language processing has reached breakthroughs with the advent of transformers. They have remained
state-of-the-art since then, and there also has been much research in analyzing, interpreting, and evaluating the attention
layers and the underlying embedding space. In addition to the self-attention layers, the feed-forward layers in the transformer
are a prominent architectural component. From extensive research, we observe that its role is under-explored. We focus on
the latent space, known as the Activation Space, that consists of the neuron activations from these feed-forward layers. In this
survey paper, we review interpretability methods that examine the learnings that occurred in this activation space. Since
there exists only limited research in this direction, we conduct a detailed examination of each work and point out potential
future directions of research. We hope our work provides a step towards strengthening activation space analysis