Who did What: A Large-Scale Person-Centered Cloze Dataset
Table 2: Performance of suppressed baselines. ∗Random performance is computed as a deterministic function of the number of times each choice set size appears. Many questions have only two choices and there are about three choices on average.
['Baseline', 'Accuracy Before', 'Accuracy After']
In order to focus our dataset on the most interesting problems, we remove some problems to suppress the performance of the following simple baselines: First person in passage: Select the person that appears first in the passage. Most frequent person: Select the most frequent person in the passage. n-gram: Select the most likely answer to fill the blank under a 5-gram language model trained on Gigaword minus articles which are too similar to one of the questions in word overlap and phrase matching. Unigram: Select the most frequent last name using the unigram counts from the 5-gram model. To minimize the number of questions removed we solve an optimization problem defined by limiting the performance of each baseline to a specified target value while removing as few problems as possible, i.e., maxα(C)∑C∈{0,1}|b|α(C)|T(C)| (1) subject to ∀i ∑C:Ci=1α(C)|T(C)|N≤k (2) N = ∑C∈{0,1}|b|α(C)|T(C)| where T(C) is the subset of the questions solved by the subset C of the suppressed baselines, and α(C) is a keeping rate for question set T(C). Ci=1 indicates i-th baseline is in the subset and |b| is a number of baselines. Then N is a total number of questions and k is an upper bound for the baselins after suppression. k is set to the random performance. The suppression removed 49.9% of the questions.