Non-autoregressive Transformer by Position Learning
Table 3: Results on validation set of WMT14 DE-EN with different position strategy. “HSP” means the reference position sequence derived from the heuristic position searching.
['Model', 'Position Accuracy(%) permutation-acc', 'Position Accuracy(%) relative-acc(r=4)', 'WMT14 DE-EN BLEU', 'Speed Up']
First, we analyze whether the position derived from the heuristic search is suitable for use as supervision to the position predictor. We evaluate the effectiveness of the searched position by training a PNAT as before and testing with the heuristic searched position instead of the predicted position. The non-autoregressive position model is less effective than the current autoregressive position model, both in the accuracy of the permutation and the relative position. Even though the current PNAT with a simple AR-Predictor has surpassed the previous NAT model, the position accuracy is still less desirable (say, less than 30%) and has a great exploration space. There is also a trade-off between the effectiveness and efficiency, the choice of the non-autoregressive means the efficiency and the choice of autoregressive means the effectiveness.