Linguistic evaluation of German-English Machine Translation using a Test Suite
Tabelle 3: Accuracies of successful translations for 16 systems and 14 categories. Boldface indicates significantly best systems in each row
['[EMPTY]', '#', 'DFKI', 'FB', 'JHU', 'MMLP', 'MSRA', 'NEU', 'onlA', 'onlB', 'onlG', 'onlX', 'onlY', 'PROMT', 'RWTH', 'Tartu', 'UCAM', 'UEDIN', 'avg']
The average accuracy per system is calculated either based on all test items (with the assumption that all items have equal importance) or based on the categories (with the assumption that all categories have equal importance). In any case, since the averages are calculated on an artificial test suite and not on a sample test set, one must be careful with their interpretation. if all categories counted equally), followed by DFKI and RWTH which are in the best cluster for 10 categories. When it comes to averaging all test items, the best systems are RWTH and online-A. On specific categories, the most clear results come in punctuation where NEU has the best performance with 100% accuracy, whereas Online-X has the worst with 31.7%. Concerning ambiguity, Facebook has the highest performance with 92.6% accuracy. In verb tense/aspect/mood, RWTH Aachen and Online-A have the highest performance with 84% accuracy, whereas in this category, MSRA.MADL has the lowest performance with 60.4%. For the rest of the categories there are small differences between the systems, since more than five systems fall into the same significance cluster of the best performance.