Imitation Attacks and Defenses for Black-box Machine Translation Systems
Table 1: Imitation models are highly similar to their victims. We train imitation models that are different from their victims in input data and/or architecture. We test the models on IWSLT (Test) and news data from WMT (OOD). We also measure functionality similarity by reporting the BLEU score between the outputs of the imitation and the victim models (Inter).
['[BOLD] Mismatch', '[BOLD] Data', '[BOLD] Test', '[BOLD] Inter', '[BOLD] OOD']
Test BLEU Score We first compare the imitation models to their victims using in-domain test BLEU. We also evaluate the imitation models on OOD data to test how well they generalize compared to their victims. We use the WMT14 test set (newstest 2014). We suspect that imitation models can sometimes outperform their victims because distillation can act as a regularizer furlanello2018born; mobahi2020self. Functional Similarity Finally, we measure the BLEU score between the outputs of the victim and the imitation models to measure their functional similarity (henceforth inter-system BLEU). As a reference for inter-system BLEU, two Transformer models trained with different random seeds achieve 62.1 inter-system BLEU. The inter-system BLEU for the imitation models and their victims is as high as 70.5