Fast Neural Machine Translation Implementation
Table 3: Time taken (sec) breakdown
['[EMPTY]', 'Baseline', 'Fused']
The fused operation only has to loop through the large cost matrix once, therefore, for low beam sizes its is comparable in speed to the simple kernel to add the bias. For higher beam sizes, the cost of maintaining the n-best list is begins to impact on speed.