Structure-Tags Improve Text Classification for Scholarly Document Quality Prediction
Table 2: Hyperparameters used in the experiments.
['[EMPTY]', 'PeerRead classification', 'S2ORC regression']
In our experiments we tried to stay close to the experimental setup used by Shen et al. We used Adam Kingma and Ba We use a considerably larger learning rate of 0.005, compared to 0.0001 used by Shen et al. We use a small batch size of 4. This is necessary for \acHAN as it uses relatively much memory, because it builds rich hierarchical BiLSTM-based representations directly from the word embeddings. In comparison, the BiLSTM model of Shen et al. We furthermore use re-sampling on the computational language (cs. CL) subset, as we find that without re-sampling, due to the imbalance in the labels learning fails. The re-sampling is done for each epoch, by keeping the full subset of examples with the less frequent label, but sub-sampling an equal number of random examples from the more frequent label subset. However, we found that this does not fix the problem that the model is not learning beyond predicting always the majority class, whereas re-sampling does. In our experiments the training of all our models proceeds slower than the number of epochs (60) used by Shen et al. This observation holds in spite of the fact that we are using a higher learning rate. We therefore used a higher number of 360 training epochs. In each experiment, we used the highest accuracy score on the validation set to select the best model, using the last epoch that achieves that score in case of ties. In addition to accuracy we also report the AUC (area under the ROC curve) scores for the PeerRead dataset.