Cross-Attention watermarking of Large Language Models

DOI DOI オープンアクセス

説明

A new approach to linguistic watermarking of language models is presented in which information is imperceptibly inserted into the output text while preserving its readability and original meaning. A cross-attention mechanism is used to embed watermarks in the text during inference. Two methods using cross-attention are presented that minimize the effect of watermarking on the performance of a pretrained model. Exploration of different training strategies for optimizing the watermarking and of the challenges and implications of applying this approach in real-world scenarios clarified the tradeoff between watermark robustness and text quality. Watermark selection substantially affects the generated output for high entropy sentences. This proactive watermarking approach has potential application in future model development.

5 pages, 3 figures. Accepted to ICASSP 2024

収録刊行物

キーワード

詳細情報 詳細情報について

問題の指摘

ページトップへ