In this year, Santa is a little early but the gifts are nonetheless still impressive. In the last days, we hacked on a model to predict if a text sequence belongs to a certain type, or not, and we found out, that if we do not process the text from left to right but in reverse order, the model learns much better and faster. The idea is not new and was already introduced in neural translation models. Still, it’s amazing that such a little modification has such a huge impact.
Next, we heard about the new PyTorch release and that it might also bring gifts, for instance performance boosts along with other nice goodies. So, we updated, still from the source since the provided pre-compiled packages do not work for us, and then run a short test. The results were pretty stunning since now the required time for a cycle takes about 5 seconds less compared to version 0.2.0.
In total, from a machine learning perspective we are pretty happy with the gifts Santa brought for us and with the experiences we made so far, we encourage everybody to update their PyTorch version to also feel the machine learning Christmas spirit.