Optimizing Differentiable Relaxations of Coreference Evaluation Metrics
Table 1: Results (F1) on CoNLL 2012 test set. CoNLL is the average of MUC, B3, and CEAFe.
['[EMPTY]', 'MUC', 'B3', 'CEAF [ITALIC] m', 'CEAF [ITALIC] e', 'BLANC', 'LEA', 'CoNLL']
We firstly compare our resolvers against \newciteP15-1137 and \newciteN16-1114. Our baseline surpasses \newciteP15-1137. It is likely due to using features from \newciteN16-1114. Using the entity-centric heuristic cross entropy loss and the relaxations are clearly beneficial: Lec is slightly better than our baseline and on par with the global model of \newciteN16-1114. Lβ=1,B3,Lβ=1,LEA outperform the baseline, the global model of \newciteN16-1114, and Lec. However, the best values of β are √1.4, √1.8 respectively for Lβ,B3, and Lβ,LEA. Among these resolvers, Lβ=√1.8,LEA achieves the highest F1 scores across all the metrics except BLANC. It is worth noting that our absolute scores are weaker than these of \newciteclark-manning:2016:EMNLP2016, as they build on top of a similar but stronger mention-ranking baseline, which employs deeper neural networks and requires a much larger number of epochs to train (300 epochs, including pretraining). For the purpose of illustrating the proposed losses, we started with a simpler model by \newciteP15-1137 which requires a much smaller number of epochs, thus faster, to train (20 epochs, including pretraining). Because the resolvers are evaluated on F1 score metrics, it should be that Lβ,B3 and Lβ,LEA perform the best with β=1. There are two hypotheses. First, the statistical difference between the training set and the development set leads to the case that the optimal β on one set can be sub-optimal on the other set. Second, in our experiments we fix T=1, meaning that the relaxations might not be close to the true evaluation metrics enough. Our future work, to confirm/reject this, is to use annealing, i.e., gradually decreasing T down to (but larger than) 0. However, one should expect that Lβ,B3 would outperform Lβ,LEA on B3 metric while it would be the other way around on LEA metric. It turns out that, B3 and LEA behave quite similarly in non-extreme cases. We can see that in Figure 2, 4, 5, 6, 7 in \newcitemoosavi-strube:2016:P16-1.