Encouraging Paragraph Embeddings to RememberSentence Identity Improves Classification
Table 4: CNN-SC outperforms other baseline models that do not use external data, including CNN-R. All baseline models are taken from Zhang et al. (2017).
['Model', 'Yelp', 'DBPedia', 'Yahoo']
We switch gears now to our fine-tuning experiments. Specifically, we take the CNN encoder pre-trained using our sentence content objective and then fine-tune it on downstream classification tasks with supervised labels. While our previous version of CNN-SC created just a single positive/negative pair of examples from a single paragraph, for our fine-tuning experiments we create a pair of examples from every sentence in the paragraph to maximize the training data. One interesting result is that CNN-SC relies on very few training examples to achieve comparable accuracy to the purely supervised CNN model. For instance, fine-tuning CNN-SC using just 500 labeled training examples surpasses the accuracy of training from scratch on 100,000 labeled examples, indicating that the sentence content encoder generalizes well. CNN-SC also outperforms CNN-R by large margins when only small amounts of labeled training data are available.