PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

Enhanced Gradient and Adaptive Learning Rate for Training Restricted Boltzmann Machines
KyungHyun Cho, Tapani Raiko and Alexander Ilin
In: International conference on machine learning (ICML 2011), 28 Jun - 2 Jul 2011, Bellevue, Washington, USA.


Boltzmann machines are often used as building blocks in greedy learning of deep networks. However, training even a simplified model, known as restricted Boltzmann machine (RBM), can be extremely laborious: Traditional learning algorithms often converge only with the right choice of the learning rate scheduling and the scale of the initial weights. They are also sensitive to specific data representation: An equivalent RBM can be obtained by flipping some bits and changing the weights and biases accordingly, but traditional learning rules are not invariant to such transformations. Without careful tuning of these training settings, traditional algorithms can easily get stuck at plateaus or even diverge. In this work, we present an enhanced gradient which is derived such that it is invariant to bit-flipping transformations. We also propose a way to automatically adjust the learning rate by maximizing a local likelihood estimate. Our experiments confirm that the proposed improvements yield more stable training of RBMs.

EPrint Type:Conference or Workshop Item (Oral)
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Learning/Statistics & Optimisation
ID Code:8835
Deposited By:Alexander Ilin
Deposited On:21 February 2012