In the good old times when we wanted to try a new algorithm, we did not have much choices. Either we uses the matrix laboratory or hack the code ourself. This changed a lot in the last years, but universal toolkits are still very rare. One of them is Theano and we love the library for many reasons. It is very fast in two ways, first the actual runtime and then the time you need to get things done, in other words the implementation, just to name a few.
Recently we dug up a paper that describes a rectified variant of auto-encoders. Since we just started to investigate ReLU units for RBMs, we thought it would be nice to test also ReLU AEs to compare the results. And thanks to Theano we had the algorithm up and running in just a few minutes. That is the beauty of Theano, you say what you want and the rest is up to the computer. Okay, we have to admit it is a little more work, but not much.
In contrast to RBMs we need one more step to make sure the reconstruction of the values is within the same range as the original data, in our case binary data. However, this is no big deal and here is a high-level description of the algorithm:
X := data
hidden := max(0, dot(X, W) + bias_hidden)
X_reconstructed := sigmoid(dot(h1, W.T) + bias_visible)
We minimized the cross entropy with an L1 weight decay to promote additional sparsity and mini-batches to speed up learning. In our initial setup we did not use any denoising capabilities. As expected, the resulting AE models are very competitive with the RBM models.
Of course more work is required to assess the new approach in detail, but in our case the AE model has some advantages. First, we do not need the generative capabilities of the RBM and second the complexity of the implementation is lower, especially with Theano as the back-end.
At the end, if both models would be equivalent, there is no difference between AEs und RBMs in the sense that there is always a weight matrix W and a bias vector h which are used to convert the data into the feature space. But as we already mentioned, even so we see light at the end of the tunnel, we are not there yet.