Large-Scale Multi-Label Text Classification on EU Legislation
Table 5: Best hyper parameters for neural methods. Nl: number of layers, HU: hidden units size, Dd: dropout rate across dimensions, Dwe: dropout rate of word embeddings, BS: batch size. * Hidden units size is fixed to word embedding dimensionality, + Nl, HU are fixed from the pre-trained model. Dropout rate fixed as suggested by Devlin et al. (2018).
['Hyper parameters', 'bigru-att', 'han', 'cnn-lwan', 'bigru-lwan', 'zacnn *', 'zagru *', 'bert-base +']
Concerning bert, we set the dropout rate and learning rate to 0.1 and 5e-5, respectively, as suggested by Devlin et al. Finally, we noticed that the model did not converge in the fourth epoch, as suggested by Devlin et al. Thus we used early-stopping with no patience and trained the model for eight to nine epochs on average among the five runs.