Investigating the Working of Text Classifiers
Table 3: Classification accuracy of various models on random and lexicon based version of each dataset. We also show the accuracy difference (Δ) between these two results. Naïve Bayes: n-gram feature extraction using tf-idf weighting followed by naïve Bayes classifier. Logistic Regression: n-gram feature extraction using tf-idf weighting followed by Logistic Regression classifier. DocVec: Document vectors trained using DBOW model [Le and Mikolov2014]. FastText: Average of the word and subword embeddings [Joulin et al.2017]. Deep Sets [Zaheer et al.2017]: two layer MLP on top of word embeddings with max pooling layer. LSTM: Word level LSTM as a document encoder in which hidden state of the last time-step is used for classification. BiLSTM: Word level bidirectional LSTM as a document encoder in which forward LSTM and backward LSTM hidden states are concatenated followed by max pooling layer [Conneau et al.2017]. CNN-MaxPool: CNN with max pooling layer [Kim2014]. CNN-DynMaxPool: CNN with dynamic max pooling layer. For details on dynamic max pooling, we request the reader to refer to Johnson2015EffectiveUO. Adv-Training: Adversarial training of encoder is done to fool the discriminator and make the representation of training and test instances domain invariant. Multi-task Learning: A shared BiLSTM encoder is used to perform joint training of text classifier, denoising autoencoder, and adversarial training. LSTM-Anon: LSTM encoder applied to the anonymized training data. BiLSTM-Anon: BiLSTM encoder applied to the anonymized training data. Adaptive Dropout: BiLSTM encoder applied after embedding gradient-based adaptive word dropout.
['[BOLD] Model', '[BOLD] Arxiv abstracts random', '[BOLD] Arxiv abstracts lexicon', '[BOLD] Arxiv abstracts Δ', '[BOLD] IMDB reviews random', '[BOLD] IMDB reviews lexicon', '[BOLD] IMDB reviews Δ', '[BOLD] ACL IMDB random', '[BOLD] ACL IMDB lexicon', '[BOLD] ACL IMDB Δ']
It can be observed from the results that for each dataset, there is a big performance gap between the random split and lexicon split for all the models. In this section, we will analyze these performance results. During our analysis, we will mostly focus on classifier’s performance on the lexicon version of a dataset. For a detailed analysis of the performance of a method on the random version, we encourage the reader to read the associated paper.