Pre-training Using Topic Distribution for Character Level Convolutional Neural Networks

MORIYA Shun, SHIBATA Chihiro

doi:10.14923/transinfj.2019pdp0004

Bibliographic Information

Other Title

文字レベル畳み込みニューラルネットに対するトピック分布を用いた事前学習

Abstract

Several sophisticated character-level CNNs for text classification have been proposed recently and shown to be state-of-the-art architectures. On the other hand, recent studies show that pre-training is one of the most promising approach for text classification especially when the data size is not sufficient. However, existing methods of pre-training such as ELMo can not be applied to character-level CNNs since they mainly offer word-level representation as a basic part of deep neural networks close to input layers. In this research, we propose pre-training methods using topic distribution for character-level CNN. We show that the proposed pre-training methods improve the classification accuracy of character-level CNNs. Especially, our methods improve accuracy dramatically when the size of labeled data is small.

Journal

電子情報通信学会論文誌D 情報・システム

電子情報通信学会論文誌D 情報・システム J103-D (4), 280-290, 2020-04-01

The Institute of Electronics, Information and Communication Engineers

Keywords

Details 詳細情報について

CRID: 1390283659865605632

DOI: 10.14923/transinfj.2019pdp0004

ISSN: 18810225; 18804535

Text Lang: ja

Data Source

JaLC
KAKEN

Abstract License Flag: Disallowed

Export