1 Introduction
Table 2: Dataset label distribution (in %)
['[EMPTY]', '[BOLD] Bad', '[BOLD] Good', '[BOLD] Loose']
^y=⎧⎪⎨⎪⎩Badif sBOW≤δ1∧sRFC≤δ1,Looseif sRFC≤δ3∧sRFC% ≥δ2,Goodif sBOW≥δ4∧sRFC≥δ4 (3) where the thresholds are manually set to (δ1,δ2,δ3,δ4)=(0.25,0.4,0.7,0.8). This labels about 69% of all samples into three classes. All others samples with scores that do not fall in the specified ranges are filtered out due to disagreement among the models. We refer to this set as Statistical Classification. We further augment this data with NMT on short sentences with frequent phrases like greetings with the Good label. More samples for the Loose category are generated by adding captions to the source (like whispers, sighs, etc) referred as Added Captions or changing word order in the target to degrade fluency called Scrambled Text samples. This constitutes all the positives (Good and Loose) while we synthesize negatives (Bad) in two ways; we randomly choose a target for each source for easy negatives (called Random Aligned) and to choose a target from a temporally close block for hard negatives (called Drifted alignment).