IntelliCode Compose: Code Generation using Transformer
Table 4: Inference speed comparison for scenarios with different beam widths and sequence lengths, with different beam search setup. The inference speed is measured using the small model as described in Tab. 3.
['[ITALIC] L', '[ITALIC] k', '[BOLD] Sequential Search (ms)', '[BOLD] Parallel Search (ms)', '[BOLD] Cached Search (ms)']
A naive beam search implementation would iterate over the top k candidates at every step to produce the output vector. However, for a sequence of length L, this would require L×k inference calls to the model, significantly increasing the inference time and degrading the overall real-time user experience. Instead, we aggregate top k candidates and perform batched inference calls at every decoding step, which reduces the number of inference calls to L. Tab.