Building a Production Model for Retrieval-Based Chatbots
Table 9: An ablation study showing the effect of different model architectures and training regimes on performance on the proprietary help desk dataset.
['[BOLD] Model', '[BOLD] Parameters', '[BOLD] Validation AUC@0.05', '[BOLD] Test AUC@0.05']
The results are shown in Table 9, [CONTINUE] As Table 9 shows, the training set size and the number of negative responses for each positive response are the most important factors in model performance. The model performs significantly worse when trained with hinge loss instead of cross-entropy loss, indicating the importance of the loss function. [CONTINUE] We observed no advantage to using a hierachical encoder, [CONTINUE] Finally, we see that a 2 layer LSTM performs similarly to either a 4 layer or a 2 layer SRU with a comparable number of parameters. [CONTINUE] Table 9 shows the results of an ablation study we performed to identify the most important components of our model architecture and training regime.