The Winner Takes All

The dream to simulate a full-fledged brain with a computer is probably as old as the universe. Okay, maybe we exaggerate a little here, but the point is that since the beginning of machine learning, a lot of ideas have been borrowed from neuro science. Probably the best known models are neural networks, but there are many more, like sparse coding or convolutional networks.

In the last years, a lot of research for neural networks have been done and especially the creation of new types of units has been caught some attention. For instance, rectifier units or the recently introduced ‘maxout’ units.

A couple of days ago, we stumbled about a very simple but also very powerful approach for neural networks. The idea is that the nodes in the hidden layer are grouped together. For the sake of simplicity, we consider only two neurons per group. The output of such a group is the maximum activation value of all neurons. The other values are set to zero. In other words, the neurons are competing and only the winner is propagating its activation value. That is why it is called ‘the winner takes all’. The functionality is inspired from biological neural networks and furthermore, the recently discussed “catastrophic forgetting”-problem in neural networks can be avoided.

In the literature we studied, the network was used in a supervised fashion but we decided to conduct some experiments without a teacher. An additional parameter for this auto-encoder is now the group size for the neurons. Our initial setup was very simple to learn as quickly as possible from the results, in our case the discovered latent topics in the data.

During the training of the model, we saw some oddities that needs to be investigated further, but a sample from the topic space looked very promising. This particular neuron seems to have captured important aspects of teen-movies: {teenagers, high-school-life, opposites-attract, teachers-and-students, first-love, romance, school} and there are other neurons specialized on some genre like crime or family.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s