Enhanced Gradient and Adaptive Learning Rate for Training Restricted Boltzmann Machines
Boltzmann machines are often used as building blocks in greedy learning of deep networks. However, training even a simpliﬁed model, known as restricted Boltzmann machine (RBM), can be extremely laborious: Traditional learning algorithms often converge only with the right choice of the learning rate scheduling and the scale of the initial weights. They are also sensitive to speciﬁc data representation: An equivalent RBM can be obtained by ﬂipping some bits and changing the weights and biases accordingly, but traditional learning rules are not invariant to such transformations. Without careful tuning of these training settings, traditional algorithms can easily get stuck at plateaus or even diverge. In this work, we present an enhanced gradient which is derived such that it is invariant to bit-ﬂipping transformations. We also propose a way to automatically adjust the learning rate by maximizing a local likelihood estimate. Our experiments conﬁrm that the proposed improvements yield more stable training of RBMs.