Sam Stites

Machine Learning Debugging, Unit Tests, and Production Torch

There are various tips when debugging machine learning algorithms, which recently came up in the Torch gitter channel. Documenting some of these words of wisdom:

Koen Dejonghe pointed out these tips from BYU’s Perception, Control and Cognition Laboratory (perma). Looking at them specifically for debugging and testing:

Closely following this discussion was a discussion of “the PyTorch-to-Tensorflow production workflow” which, thankfully, I have never needed to deal with, but I thought it would be important to touch on since it has its own idiosyncrasies. In short, a common workflow (to the channel) is that a researcher will use PyTorch for prototyping, then migrate the model into TensorFlow for production use, but tensorflow models exhibit different behaviour with the same hypeparameters so you basically need to redo your hyperparameters tuning all over again. This is reminicent of Deep Reinforcement Learning that Matters. In this particular case, TF models refused to train unless you l2-normalized the input (unlike in Torch). Another researcher who doesn’t need to train tensorflow models on the field simply trains models in PyTorch, then copies the weights to a mirror tensorflow model. A few things need to be rewritten in TF to make them Torch friendly (Bi-directional LSTM had a number of subtle differences), but said development in PyTorch was worth the hassle. This still might have nuances (like the data preprocessing requirements) and should always be backed with benchmarks.