fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Table 1: Translation speed measured on a V100 GPU on the test set of the standard WMT’14 English-German benchmark using a big Transformer model.
['[EMPTY]', 'Sentences/sec']
During inference we build batches with a variable number of examples up to a user-specified number of tokens, similar to training. fairseq also supports inference in FP16 which increases decoding speed by 54% compared to FP32 with no loss in accuracy