DQI: Measuring Data Quality in NLP
Table 24: T5 for DQIc6
['[BOLD] Granularity/Split', '[BOLD] Good', '[BOLD] Bad']
The fifth term first computes the frequency of each unit in a granularity, to form vectors of length three for each unit. The standard deviation of this vector is calculated for each unit, if the unit is repeated. If the unit is not repeated, then it is not considered in our calculation. The sum of these standard deviations is calculated across the given granularity. This sum is normalized by division by the size of the set of units for that granularity, across all labels. The expectation is that the good split will show lower values of this term compared to the bad split, as lesser variance within labels is desirable. So, the term has been reversed to have direct proportionality with DQI.