Joint Neural Entity Disambiguation with Output Space Search
Table 4: Confusion Matrices for (a) using heuristic h1 with τ=25%n, (b) heuristic h1 with τ=50%n and (c) heuristic h2 with flexible τ.
['[ITALIC] τ=25% [ITALIC] n', 'r ≤ 25%', 'r > 25%']
For heuristic h1, we use a fixed depth strategy. In particular, given a document with n mentions, we consider two different depth limits: τ=25%n and τ=50%n, which lead to an average depth of 5 and 10 per document respectively. For heuristic h2, the depth is flexible and determined by the number of mentions that are predicted to be incorrect by h2. This strategy leads to an average depth of 4. In these tables correct/incorrect mean whether the local prediction is correct or not (comparing to the ground truth). Therefore the first cell with value 1134 indicates that there are 1134 mentions in test-b that are correctly predicted by the local model, but deemed as among the top 25% least confident mentions (aka the hard queries) by h1. The confusion matrix for a good heuristic will have small diagonal values and large anti-diagonal values. These results show that heuristic h2 gives the best precision as well as recall of the real mistakes made by the local model.