Identifying Semantic Divergences in Parallel Text without Annotations
Table 2: Intrinsic evaluation on crowdsourced semantic equivalence vs. divergence testsets. We report overall F-score, as well as precision (P), recall (R) and F-score (F) for the equivalent (+) and divergent (-) classes separately. Semantic similarity yields better results across the board, with larger improvements on the divergent class.
['[BOLD] Divergence Detection  [BOLD] Approach', '[BOLD] OpenSubtitles +P', '[BOLD] OpenSubtitles +R', '[BOLD] OpenSubtitles +F', '[BOLD] OpenSubtitles -P', '[BOLD] OpenSubtitles -R', '[BOLD] OpenSubtitles -F', '[BOLD] OpenSubtitles Overall F', '[BOLD] Common Crawl +P', '[BOLD] Common Crawl +R', '[BOLD] Common Crawl +F', '[BOLD] Common Crawl -P', '[BOLD] Common Crawl -R', '[BOLD] Common Crawl -F', '[BOLD] Common Crawl Overall F']
The break down per class shows that both equivalent and divergent examples are better detected. The improvement is larger for divergent examples with gains of about 10 points for F-score for the divergent class, when compared to the next-best scores.