Humor in Collective Discourse: Unsupervised Funniness Detection in the New Yorker Cartoon Caption Contest
Table 1: Comparison between the methods. Score s4 corresponds to pairs for which the seven judges agreed more significantly (a difference of 4+ votes). Score s3 requires a difference of 3+ votes. Score s includes all pairs (about 850 per method, minus a small number of errors). The best methods (CU2R, CU3, OR2, and CU2) are in bold.
['[BOLD] Category', '[BOLD] Code', '[BOLD] Method', '[ITALIC] n4', '[ITALIC] s4', '[ITALIC] n3', '[ITALIC] s3', '[ITALIC] n', '[ITALIC] s']
Each evaluation (ni, si pair) corresponds to the number of votes in favor of the given method minus the number of votes against. So the first set corresponds to pairs in which, out of seven judges, there was a difference of at least 4 votes in favor of one or the other caption. This level of significant agreement happened in 5,594/15,154 cases (36.9% of the time). A difference of at least 3 votes happened in 8,131/15,154 pairs (53.6%). The third evaluation corresponds to all pairwise comparisons, including ties. ni refers to the number of times the above constraint for i is met and score si is calculated by averaging the number of votes in favor minus the number of votes against for each ni. The probability that a random process will generate a difference of at least 4 votes (excluding ties) is 12.5%.