Supervised semi-autoencoder learning for multi-layered neural networks

The present paper proposes a new type of layer-wise learning for multi-layered neural networks. Multi-layered neural networks have the serious problem of vanishing information, where information contained in input patterns is gradually lost, and preventing neural networks from learning input patterns. In particular, the autoencoders used in the greedy layer-wise pre-training tend to lose the original information by going through multiple layers. For this problem, we propose a new approach called “supervised semi-autoencoder” to solve the problem of vanishing information. The new method is close to the ordinary autoencoder, but the outputs are the original inputs and the corresponding targets for amplifying information in input patterns. In addition, the computational procedures are simplified by using potential learning, in which the potentiality of neurons is determined before learning, and is given as initial weights. Thus, the complicated adjustment of parameters is not necessary. The method was applied to the well-known crab data set as well as the real data set of eye-tracking records. In both cases, with relatively small-sized data, neural networks could find internal representations close to those obtained by larger ones, and better generalization performance could be observed.