Explainable and Discourse Topic-aware Neural Language Understanding
Table 2: Language Modeling Perplexity scores on three datasets under two different settings of NLM i.e., S → small-NLM and L → large-NLM. Here, (+) → augment this feature to the model in previous row, NLM → LSTM-LM, (∗) → scores taken from Wang et al. (2018). Here, bold values indicate best performing proposed model in comparison to LSTM-LM baseline, and GAIN(%) indicates the improvement in performance of the same.
['[EMPTY]', 'Model', 'APNEWS S', 'APNEWS L', 'IMDB S', 'IMDB L', 'BNC S', 'BNC L']
All topic composition models outperform LSTM-LM baseline which demonstrate the advantage of composing document topical semantics in NLM. Based on the results, here are three key observations: (i) ETA-NLM always performs better than LTA-NLM resulting in 11% (49.34 vs 55.48), 13.2% (59.20 vs 68.21) and 2.7% (95.62 vs 98.31) improvement for APNEWS, IMDB and BNC datasets respectively under small-NLM configuration. This behaviour asserts that ETR vector effectively captures fine-grained document topical semantics compared to LTR vector; (ii) LETA-NLM outperforms both LTA-NLM & ETA-NLM by exploiting complementary semantics of ETR and LTR vectors; (iii) composing sentence-level topic representations i.e., +SDT, further boost the performance by maintaining sentence-level topical discourse and hence LETA-NLM +SDT improve upon LTA-NLM model by 22% (42.98 vs 55.48), 17% (56.65 vs 68.21) and 10% (88.30 vs 98.31) for APNEWS, IMDB and BNC datasets respectively under small-NLM configuration. We evaluate the quality of representations learned by our proposed models via document classification. We use three labeled datasets: 20Newsgroups (20NS), Reuters (R21578) and IMDB movie reviews (IMDB) (See supplementary for data statistics). For each document d, we extract: (1) output state om for each input word xm via NLM component ; and (2) document topic representation vector via NTM component based on the model configuration. For IMDB movie reviews dataset, we use our best models trained on unlabeled IMDB dataset as feature extractor. However, as 20NS and R21578 are news-domain datasets, we employ best models trained on APNEWS because of its bigger corpus size than BNC. Baselines: Document retrieval is used to evaluate applicability of topic models.