An empirical study on large scale text classification with skip-gram embeddings
Table 3: Classification performance of the different representations. The upper part of the table presents one-hot-encoding methods, while the bottom part methods that depend on the dimensionality of the distributed representations. We report the best performance obtained when the size N of the training data N∈{1,50,100,150,200}×103. The best achieved performance per classification problem is shown in bold. We have performed statistical significance tests to test if improvements obtained compared to tf-idf representations are statistically important, which is noted by putting a dagger (†) to the accuracy scores.
['[ITALIC] hash', 'PubMed1,000 0.63', 'PubMed1,000 0.63', 'PubMed1,000 0.63', 'PubMed5,000 0.427', 'PubMed5,000 0.427', 'PubMed5,000 0.427', 'PubMed10,000 0.456', 'PubMed10,000 0.456', 'PubMed10,000 0.456']
We first discuss the performance when single representations are used: x_conc, tf-idf and hash. Notice that tf-idf performs better than both x_conc and hash, with the latter achieving the lowest performance. The performance of the three representations on PubMed1,000 is comparable, but in the bigger classification problems tf-idf and performs considerably better. We, thus, consider the tf-idf representations as our baseline model, and we examine how the models with concatenated representations behave compared to it. We investigate now whether the fusion of distributed document representations with the one-hot-encoding representations benefits the classification performance. In the experiments, both hash and tf-idf consistently achieve better performance when combined with the distributed representations. For instance, for PubMed10,000 and D=400 the tf-idf (resp. hash) representations improve in absolute numbers by 2 (resp. 3.5) F1 points. We have performed two-sided student’s t-tests (p<0.01) to compare whether the improvements obtained for each classification problem are statistically significant compared to using tf-idf representations. In addition to the important improvements, note than in the fused representations, the effect of D diminishes. For instance in PubMed1,000, one can obtain the optimal performance using embedding dimensions with D<400 and similar observations can be made in the rest of the datasets.