Analyzing Transformers via Value Matrices

DOI Web Site 13 References Open Access

Bibliographic Information

Other Title
  • Value行列を手掛かりとした Transformerの分析

Description

<p>We propose a new method to analyze Transformer language models. In Transformer self-attention modules, attention weights are calculated from the query vectors and key vectors. Then, output vectors are obtained by taking the weighted sum of value vectors. While existing works on analysis of Transformer have focused on attention weights, this work focused on value and output matrices. We obtain joint matrices by multiplying both matrices, and show that the trace of the joint matrices are correlated with word co-occurences.</p>

Journal

References(13)*help

See more

Related Projects

See more

Keywords

Details 詳細情報について

Report a problem

Back to top