A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency
Table 6: LAS rescoring computation latency (ms).
['Exp ID', '50% latency', '90% latency']
In order to show the improvement in LAS computation latency by batch inference, we benchmark the wall time for the second-pass rescoring part when we run the recognition system on 100 search utterances on a Google Pixel4 phone. Inference is run on the phone’s CPU. In this section, we compare the proposed RNN-T+LAS model (0.18G in model size) to a state-of-the-art conventional model. Similar to how the E2E model incurs cost with a 2nd-pass LAS rescorer, the conventional model also incurs cost with the MaxEnt rescorer. We found that for voice-search traffic, the 50% computation latency for the MaxEnt rescorer is around 2.3ms and the 90% computation latency is around 28ms. The figure shows that for an EP90 operating point of 550ms or above, the E2E model has a better WER and EP latency tradeoff compared to the conventional model. At the operating point of matching 90% total latency ( EP90 latency + 90% 2nd-pass rescoring computation latency) of E2E and server models