JVNV: A Corpus of Japanese Emotional Speech With Verbal Content and Nonverbal Expressions
-
- Detai Xin
- Graduate School of Information Science and Technology, The University of Tokyo, Bunkyo, Tokyo, Japan
-
- Junfeng Jiang
- Graduate School of Information Science and Technology, The University of Tokyo, Bunkyo, Tokyo, Japan
-
- Shinnosuke Takamichi
- Graduate School of Information Science and Technology, The University of Tokyo, Bunkyo, Tokyo, Japan
-
- Yuki Saito
- Graduate School of Information Science and Technology, The University of Tokyo, Bunkyo, Tokyo, Japan
-
- Akiko Aizawa
- National Institute of Informatics, Chiyoda, Tokyo, Japan
-
- Hiroshi Saruwatari
- Graduate School of Information Science and Technology, The University of Tokyo, Bunkyo, Tokyo, Japan
説明
We present the JVNV, a Japanese emotional speech corpus with verbal content and nonverbal vocalizations whose scripts are generated by a large-scale language model. Existing emotional speech corpora lack not only proper emotional scripts but also nonverbal vocalizations (NVs) that are essential expressions in spoken language to express emotions. We propose an automatic script generation method to produce emotional scripts by providing seed words with sentiment polarity and phrases of nonverbal vocalizations to ChatGPT using prompt engineering. We select 514 scripts with balanced phoneme coverage from the generated candidate scripts with the assistance of emotion confidence scores and language fluency scores. We demonstrate the effectiveness of JVNV by showing that JVNV has better phoneme coverage and emotion recognizability than previous Japanese emotional speech corpora. We then benchmark JVNV on emotional text-to-speech synthesis using discrete codes to represent NVs. We show that there still exists a gap between the performance of synthesizing read-aloud speech and emotional speech, and adding NVs in the speech makes the task even harder, which brings new challenges for this task and makes JVNV a valuable resource for relevant works in the future. To our best knowledge, JVNV is the first speech corpus that generates scripts automatically using large language models.
収録刊行物
-
- IEEE Access
-
IEEE Access 12 19752-19764, 2024
Institute of Electrical and Electronics Engineers (IEEE)
- Tweet
キーワード
- FOS: Computer and information sciences
- Sound (cs.SD)
- emotional speech corpus
- Corpus design
- Japanese corpus
- large-scale language model
- Computer Science - Sound
- TK1-9971
- Audio and Speech Processing (eess.AS)
- FOS: Electrical engineering, electronic engineering, information engineering
- nonverbal vocalization
- Electrical engineering. Electronics. Nuclear engineering
- nonverbal expression
- Electrical Engineering and Systems Science - Audio and Speech Processing
詳細情報 詳細情報について
-
- CRID
- 1360584340540278912
-
- ISSN
- 21693536
-
- 資料種別
- journal article
-
- データソース種別
-
- Crossref
- KAKEN
- OpenAIRE