Building a Production Model for Retrieval-Based Chatbots
Table 8: Inference time (milliseconds) of our model to encode a context using an SRU or an LSTM encoder on a single CPU core. The last row shows the extra time needed to compare the response encoding to 10,000 cached candidate response encodings in order to find the best response.
['[BOLD] Encoder', '[BOLD] Layer', '[BOLD] Params', '[BOLD] Time']
The model makes use of a fast recurrent network implementation (Lei et al., 2018) and multiheaded attention (Lin et al., 2017) and achieves over a 4.1x inference speedup over traditional encoders such as LSTM (Hochreiter and Schmidhuber, 1997). [CONTINUE] SRU also exhibits a significant speedup in inference time compared to LSTM (by a factor of 4.1x in our experiments), [CONTINUE] As seen in Table 8, an SRU encoder is over 4x faster than an LSTM encoder with a similar number of parameters, making it more suitable for production use. [CONTINUE] Table 8 also highlights the scalability of using a dual encoder architecture. [CONTINUE] Retrieving the best candidate once the context is encoded takes a negligible amount of time compared to the time to encode the context. [CONTINUE] Since the SRU is more than 4x faster at inference time with the same level of performance,