Learning With Multiple Teachers

Inspired by curriculum learning, we thought it would be a good idea to build a model that provides some hints for the student. Furthermore, the guidance should start by focusing on an easier problem which is then turned, step-by-step, into a more sophisticated one. To be more concrete, we consider the problem of genre classification. A movie has one or more top-level genres and optionally, none or more sub-genres. For example, the movie Doom might have sci-fi/horror as top-level and {creature, zombie}-film as sub-genres.

The model is as follows. We have an input layer, two hidden layers and two soft-max layers. One soft-max follows the first hidden layer and the other is the top-layer. The intuition is that we want to guide every hidden layer with additional hints. In our case, the first hidden layer also learns to separate individual top-level genres, or in other words, high-level concepts, like ‘action’ or ‘western’ with the help of the soft-max classifier. The 2nd classifier is then used to further separate concepts like ‘western’ into finer themes: comedy, modern, spaghetti. We can think of the model as a simple hierarchy for genres. We start with very broad concepts and at each level, we add more details.

The drawback of the model is that each hidden layer is now tightly coupled with a classification problem and we need to make sure that objectives at lower hidden layers do not interfere with the top-level objective. We recently found out that a very similar approach was published as “Deeply-Supervised Nets” but instead of a taxonomy-like classification the approach uses the same labels for each classifier.

The preliminary results look very promising but as usual the hand-crafted features are a bottleneck. Furthermore, for a lot of movies, we only have very limited meta-data which means we cannot utilize the full strength of the model.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s