Imitation Attacks and Defenses for Black-box Machine Translation Systems
Table 2: English\toGerman imitation results. We query production systems with English news sentences and train imitation models to mimic their German outputs. The imitation models closely imitate the production systems for both in-domain (WMT newstest2014) and out-of-domain test data (IWSLT TED talks).
['[BOLD] Test', '[BOLD] Model', '[BOLD] Google', '[BOLD] Bing', '[BOLD] Systran']
Test BLEU Scores Our imitation models closely match the performance of the production systems. For English\toGerman, we evaluate models on the WMT14 test set (newstest2014) and report standard tokenized case-sensitive BLEU scores. Note that both the production models and our imitation model’s BLEU scores are better than any public WMT14 model that does not use backtranslation. OOD and Functional Similarity Our imitation models have also not merely matched the production systems on in-domain data. Finally, there is also a high inter-system BLEU between the imitation models and the production systems. In particular, on the English\toGerman WMT14 test set the inter-system BLEU is 65.6, 67.7, and 69.0 for Google, Bing, and Systran, respectively.