Value行列を手掛かりとした Transformerの分析

DOI Web Site 参考文献13件 オープンアクセス

書誌事項

タイトル別名
  • Analyzing Transformers via Value Matrices

説明

<p>We propose a new method to analyze Transformer language models. In Transformer self-attention modules, attention weights are calculated from the query vectors and key vectors. Then, output vectors are obtained by taking the weighted sum of value vectors. While existing works on analysis of Transformer have focused on attention weights, this work focused on value and output matrices. We obtain joint matrices by multiplying both matrices, and show that the trace of the joint matrices are correlated with word co-occurences.</p>

収録刊行物

参考文献 (13)*注記

もっと見る

関連プロジェクト

もっと見る

キーワード

詳細情報 詳細情報について

問題の指摘

ページトップへ