Multiplicative LSTM for sequence modelling
This topic contains 0 replies, has 1 voice, and was last updated by arXiv 8 months, 2 weeks ago.

Multiplicative LSTM for sequence modelling
We introduce multiplicative LSTM (mLSTM), a recurrent neural network architecture for sequence modelling that combines the long shortterm memory (LSTM) and multiplicative recurrent neural network architectures. mLSTM is characterised by its ability to have different recurrent transition functions for each possible input, which we argue makes it more expressive for autoregressive density estimation. We demonstrate empirically that mLSTM outperforms standard LSTM and its deep variants for a range of character level language modelling tasks. In this version of the paper, we regularise mLSTM to achieve 1.27 bits/char on text8 and 1.24 bits/char on Hutter Prize. We also apply a purely bytelevel mLSTM on the WikiText2 dataset to achieve a character level entropy of 1.26 bits/char, corresponding to a word level perplexity of 88.8, which is comparable to word level LSTMs regularised in similar ways on the same task.
Multiplicative LSTM for sequence modelling
by Ben Krause, Liang Lu, Iain Murray, Steve Renals
https://arxiv.org/pdf/1609.07959v3.pdf
You must be logged in to reply to this topic.