Harvey Mudd College at SemEval-2019 Task 4: The Clint Buchanan Hyperpartisan News Detector
Table 1: Validation accuracy for BERT-base with and without Unsupervised Pre-training (UP).
['Max Seq Len', 'BERT-base', 'BERT-base + UP']
Our first experiment was checking the importance of pre-training. We then trained the model on sequence lengths of 100, 250 and 500. The other column shows how well BERT-base trained without pre-training. We found improvements for lower sequence lengths, but not at 500 word pieces. Since the longer chunk should have been more informative, and since our hand-labeled training set only contained 516 articles, this likely indicates that BERT experiences training difficulty when dealing with long sequences on such a small dataset. As the cost to do pre-training was only a one time cost all of our remaining experiments use a pre-trained model.