Hierarchical Latent Alignment for Non-Autoregressive Generation under High Compression Ratio

DOI Web Site
  • XU Wang
    School of Computer Science and Technology, Harbin Institute of Technology
  • MA Yongliang
    Beijing Langboat Technology Co., Ltd.
  • CHEN Kehai
    School of Computer Science and Technology, Harbin Institute of Technology
  • ZHOU Ming
    Beijing Langboat Technology Co., Ltd.
  • YANG Muyun
    School of Computer Science and Technology, Harbin Institute of Technology
  • ZHAO Tiejun
    School of Computer Science and Technology, Harbin Institute of Technology

抄録

<p>Non-autoregressive generation has attracted more and more attention due to its fast decoding speed. Latent alignment objectives, such as CTC, are designed to capture the monotonic alignments between the predicted and output tokens, which have been used for machine translation and sentence summarization. However, our preliminary experiments revealed that CTC performs poorly on document abstractive summarization, where a high compression ratio between the input and output is involved. To address this issue, we conduct a theoretical analysis and propose Hierarchical Latent Alignment (HLA). The basic idea is a two-step alignment process: we first align the sentences in the input and output, and subsequently derive token-level alignment using CTC based on aligned sentences. We evaluate the effectiveness of our proposed approach on two widely used datasets XSUM and CNNDM. The results indicate that our proposed method exhibits remarkable scalability even when dealing with high compression ratios.</p>

収録刊行物

キーワード

詳細情報 詳細情報について

問題の指摘

ページトップへ