What Level of Quality can Neural Machine Translation Attain on Literary Text?
Table 5: BLEU scores obtained by PBSMT and NMT for each of the books that make up the test set. NMT outperforms PBSMT by a statistically significant margin (p<0.01) on all the books.
['[BOLD] Novel', '[BOLD] PBSMT', '[BOLD] NMT', '[BOLD] Relative improvement']
The results across the different novels show a very high degree of variability, indicating that fiction is far from being a monolithic domain. In fact scores go from a low of 0.1611 (PBSMT for Ulysses) to the highest of 0.3892 (NMT for Harry Potter #7), which more than doubles the first figure.