Is it worth it? Budget-related evaluation metrics for model selection
Table 3: Two hypothetical prediction scenarios (S1 and S2) that illustrate the advantage provided by considering the distribution of classes in the ordered list. Lighter-shaded cells represent data that cannot be annotated due to budget constraints. The bottom row shows total number of true positive (TP) instances included in the final resource.
['TestSet  [BOLD] ID', '[BOLD] S1  [BOLD] M1', '[BOLD] S1  [BOLD] M2', '[BOLD] S2  [BOLD] M1', '[BOLD] S2  [BOLD] M2']
However, as is often the case when building a linguistic resource, budgetary constraints mean that we cannot manually review all of the examples that a model returns as positive. Simulating the effect of these constraints on the test set requires a cut off point in the data. Thus, once budget comes into play, our list of candidates is shortened: if we can only afford to annotate ∼30% of our positively labeled data, six candidates drop out of each list, and we drop two thirds of the dataset (shaded light green and light red). Hence, once we know what the gain is for each decile of our ordered list, with our budget dictating how many deciles we can annotate, we can easily identify at which point checking more candidates stops being profitable, or how much data we need to pass to our annotator to obtain the desired number of entries for our dictionary. Comparing these measures across several systems then proves to be a suitable evaluation measure. We are assuming that each row/data point is also one decile. Hence, we know that in S1 the first decile (D1) of both M1 and M2 has a gain of 0.5. However, the gain of M1 in D2 is 0, while the gain of M2 is, again, 0.5. Given that we can afford to only annotate 2 deciles, this makes M2 the clear winner. the cumulative gain is equal to that same decile’s gain - 0.5. But once we reach D4 (which also has a gain of 0.5), its cumulative gain will be 1, as by that point we will have encountered all the positive instances in the list.