CFO: A Framework for Building Production NLP Systems
Table 3: F1 and latencies for BERT base and large models running on GPU and CPU for a subset of the NQ dev set. TDK is the K-th percentile query latency in seconds when running on device D (GPU or CPU).
['Model', '[ITALIC] F1', '[ITALIC] TG50', '[ITALIC] TG95', '[ITALIC] TC50', '[ITALIC] TC95']
Most model settings are taken from Alberti et al. In addition, we experiment with models trained on BERT base and BERT large to understand trade-offs between latency and accuracy. In order to decrease latency, we simulate passage retrieval to send GAAMA the most relevant passage by selecting the first correct top level candidate if there is one and the first (incorrect) top level candidate if there is not. When running GAAMA on a GPU these result in manageable 95th percentile query latencies of less than a second; whereas on the CPU the 95th percentile times are in excess of two and five seconds for base and large respectively. For large, even the median latency is greater than one and a half seconds, effectively cementing GPUs as a requirement for deploying to production environments. In future work we intend to explore network pruning or knowledge distillation techniques for potential speedups with the large model.