Harvey Mudd College at SemEval-2019 Task 4: The Clint Buchanan Hyperpartisan News Detector
Table 2: Validation Accuracy on BERT-LARGE across sequence length and learning rate.
['Max Seq LenLearning Rate', '5e-7', '1e-6', '1.5e-6', '2e-6', '2.5e-6', '3e-6']
This is a discrepancy from BERT-BASE indicating that the larger model struggled more with training on a small amount of long sequences. For our best trained BERT-LARGE, we submitted the model for evaluation on TIRA. Surprisingly, the test performance (75.1%) of the larger model was worse than the base model. The experiments in Devlin et al. consistently found improvements when using the large model. The main distinction here is a smaller training dataset than in their tasks. The experiments in the remaining sections use the same hyperparameters as the optimal BERT-LARGE.