KERMIT: Generative Insertion-Based Modeling for Sequences
Table 3: GLUE benchmark scores (as computed by the GLUE evaluation server). Of these models, only GPT and KERMIT admit a straightforward generation process.
['[BOLD] Model', '[BOLD] Generative?', '[BOLD] CoLA', '[BOLD] SST-2', '[BOLD] MRPC', '[BOLD] STS-B', '[BOLD] QQP', '[BOLD] MNLI-(m/mm)', '[BOLD] QNLI', '[BOLD] RTE', '[BOLD] WNLI', '[BOLD] AX', '[BOLD] Score']
BERT outperforms GPT in large part because of its deeply bi-directional architecture, but in the process BERT sacrifices the ability to perform straightforward generation. While we find KERMIT to perform slightly behind BERT, KERMIT maintains the ability to generate text while obtaining results that are much closer to BERT rather than GPT.