More on Regularizers

Very recently we played a little with alternative regularizations of our model and found DropOut very promising, since it is straightforward to implement in the case of RBMs, and it is practically no overhead. The method definitely seems to influence the outcome of our model, but before we continue our analyses, we tried some generalizations, namely DropConnect. In a nutshell: DropConnect randomly omits connections in the weight matrix rather than hidden nodes as in DropOut.

The implementation of the new method is no big deal and since we are only interested to study the discovered latent topics, represented by the weights of a hidden node to the data, we can skip the inference step. The inference is no big deal either, but in comparison to DropOut it is not trivial to implement.

As usual, we started to train a model for a single genre to study the discovered latent topics. First, we trained a model with DropOut, then with DropConnect. To compare the methods, we measured the spread of the keywords and the total number of keywords with a high repetition. The procedure was repeated several times and the outcome was quite astonishing.

In case of DropConnect the spread was almost maximal and there were not any word with a high repetition. A common problem with our data set is that high-frequent keywords usually appear very often in trained models and this also happened in the case of DropOut, not too much of them, but always a few of them were present. However, with DropConnect this never happened during training which was the reason why we repeated the procedure several times with the same, but also with other genres. For the latter case, both variants produced repetitions.

With the few experiments we conducted it is not possible to draw any real conclusions but at least the results encouraged our intuition that this kind of regularization is likely to help us to train better models.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s