Efficient Online Scalar Annotation with Bounded Support
Table 1: Example propositions and the scalar political spectrum ranged between 0 (very conservative) and 100 (very liberal) by each approach: direct assessment, online ranking aggregation, and EASL. The dashed lines indicate a split by 5-ary ordinal scale.
['Propositions', 'Gold', 'DA', 'RA', 'EASL']
The distribution of the histograms shows that DA and EASL successfully fit to the distribution in the oracle, whereas RA converges to a rather narrow range. We note that renormalizing the distribution in RA will not address the issue. For instance, when the dataset has only liberal propositions, RA still fails to capture the latent distribution because it looks only at relative distances between instances but not the distance from bounds. Again, we see that RA approach has a narrower range than the oracle, DA, and EASL.