Non-Negative RBMs

After we did some tests with non-negative factorizations we thought about new ways to adjust existing algorithms. As mentioned before, there is no easy way to transform new samples into the feature space. In case of pure a NMF, we have to solve an optimization problem for each new sample which is expensive. This is in stark contrast to RBMs, where a single matrix vector multiplication, plus applying a point-wise non-linearity function, is sufficient.

While we gathered literature for studying RBMs in the context of text analysis, we found an approach to learn part-based representations with non-negative RBMS. The idea is very simple: We take an ordinary binary RBM and add a regularizer that encourages W to be positive. This approach also has the advantage that an existing implementation can be modified with just a few lines of code. If we compare this extension to weight decay in general, the new method penalizes negative weights instead of larger weights but the semantic is the same otherwise. Therefore, existing code can be adjusted by replacing the weight decay with the new regularizer. A further minor difference is that the weights should be drawn from small positive values.

After we implemented the required modifications, we were of course interested to test check if this approach is suitable to extract latent topics from our test data. Just a quick reminder, our test data is very sparse and binary.

We started by considering only movies from a specific genre to be able to check the quality of the results more easily. The results indicate some topic clustering for each neuron, but not as clearly as with the other approaches. Since the non-negative RBM was mainly tested on image data and larger texts, we are not surprised that the results are not stunning. We plan to continue our tests and we hope that with some fine-tuning of the parameters, the results will further improve.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s