Building an efficient neural language model over a billion words

This post introduces adaptive softmax, a novel softmax approximation technique tailored for GPUs to efficiently train neural network based language models over very large vocabularies. It also details the open-sourcing of torch-rnnlib, a library that allows researchers to design new recurrent models and test prototypes on GPUs with minimal effort, including seamless access to fast baselines through torch.cudnn bindings. The post demonstrates achieving a perplexity of 45 on the 1-billion word dataset on a single GPU in a couple of days using these tools.