EVALUATION METHOD OF NON-TASK-ORIENTED DIALOGUE SYSTEM BY HMM

機関リポジトリ 機関リポジトリ (HANDLE) オープンアクセス

説明

Recently, computerized dialogue systems have been actively investigated and used in various fields. In order to realize a practical system, the performance of the system should be evaluated quantitatively. An objective and quantitative evaluation method for task-oriented dialogue systems, such as reservation services, has already been established; however, non-task-oriented dialogue systems have been evaluated only by subjective methods like questionnaires. In this paper, we propose a new criterion that can evaluate non-task-oriented dialogue systems objectively and quantitatively. We assume that a human-human dialogue is an ideal dialogue. We design an HMM (Hidden Markov Model) by learning a sequence of human-human dialogue utterance tags that are automatically assigned. We apply n-gram for auto-tagging and evaluate the humanness of dialogues using HMM. In this simulation, the rate of correct auto-tagging is 54%. If we consider partly correct tags as completely correct tags, the correct rate becomes 82%. Furthermore, it was clarified that the proposed method based on HMM can evaluate the humanness of a dialogue.

収録刊行物

詳細情報 詳細情報について

問題の指摘

ページトップへ