Encode, Tag, Realize: High-Precision Text Editing
Table 6: Inference time (in ms) across various batch sizes on GPU (Nvidia Tesla P100) averaged across 100 runs with random inputs.
['batch size', 'LaserTaggerFF', 'LaserTaggerAR', 'seq2seqBERT']
Getting state-of-the-art results often requires using larger and more complex models. When running a model in production, one cares not only about the accuracy but also the inference time. As one can see, the seq2seqBERT baseline is impractical to run in production even for the smallest batch size. On the other hand, for a batch size 8, LaserTaggerAR is already 10x faster than comparable-in-accuracy seq2seqBERT baseline. This difference is due to the former model using a 1-layer decoder (instead of 12 layers) and no encoder-decoder cross attention. We also tried training seq2seqBERT with a 1-layer decoder but it performed very poorly in terms of accuracy. Finally, LaserTaggerFF is more than 100x faster while being only a few accuracy points below our best reported results.