Topic Compositional Neural Language Model
This topic contains 0 replies, has 1 voice, and was last updated by arXiv 1 year, 1 month ago.

Topic Compositional Neural Language Model
We propose a Topic Compositional Neural Language Model (TCNLM), a novel method designed to simultaneously capture both the global semantic meaning and the local word ordering structure in a document. The TCNLM learns the global semantic coherence of a document via a neural topic model, and the probability of each learned latent topic is further used to build a MixtureofExperts (MoE) language model, where each expert (corresponding to one topic) is a recurrent neural network (RNN) that accounts for learning the local structure of a word sequence. In order to train the MoE model efficiently, a matrix factorization method is applied, by extending each weight matrix of the RNN to be an ensemble of topicdependent weight matrices. The degree to which each member of the ensemble is used is tied to the documentdependent probability of the corresponding topics. Experimental results on several corpora show that the proposed approach outperforms both a pure RNNbased model and other topicguided language models. Further, our model yields sensible topics, and also has the capacity to generate meaningful sentences conditioned on given topics.
Topic Compositional Neural Language Model
by Wenlin Wang, Zhe Gan, Wenqi Wang, Dinghan Shen, Jiaji Huang, Wei Ping, Sanjeev Satheesh, Lawrence Carin
https://arxiv.org/pdf/1712.09783v1.pdf
You must be logged in to reply to this topic.