SP-10K: A Large-scale Evaluation Set for Selectional Preference Acquisition
Table 9: Result of different models on the subset of Winograd Schema Challenge. NA means that the model cannot give a prediction, Ap means the accuracy of predict examples without NA examples, and Ao means the overall accuracy.
['Model', 'Correct', 'Wrong', '[EMPTY]', '[ITALIC] Ap', '[ITALIC] Ao']
From the result, we can see that ‘SP-10K’ can solve that problem with very high precision. But as we only label 4,000 multi-hop pairs, the overall coverage is limited. On the other hand, automatic SP acquisition method PP can cover more questions, but the precision also drops due to the noise of the collected SP knowledge. The experimental result shows that if we can automatically build a good multi-hop SP model, we could make some steps towards solving the hard pronoun coreference task, which is viewed a vital task of natural language understanding.