The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions

DOI Web Site 9 Citations
  • Sepp Hochreiter
    Institut für Informatik, Technische Universität München, München, D-80290, Germany

Description

<jats:p> Recurrent nets are in principle capable to store past inputs to produce the currently desired output. Because of this property recurrent nets are used in time series prediction and process control. Practical applications involve temporal dependencies spanning many time steps, e.g. between relevant inputs and desired outputs. In this case, however, gradient based learning methods take too much time. The extremely increased learning time arises because the error vanishes as it gets propagated back. In this article the de-caying error flow is theoretically analyzed. Then methods trying to overcome vanishing gradients are briefly discussed. Finally, experiments comparing conventional algorithms and alternative methods are presented. With advanced methods long time lag problems can be solved in reasonable time. </jats:p>

Journal

Citations (9)*help

See more

Details 詳細情報について

Report a problem

Back to top