Large-scale Cloze Test Dataset Created by Teachers
Table 6: Humans’ performance compared with 1-billion-language-model. In the short context part, both 1B-LM and humans only use information of one sentence. In the long context part, humans have the whole passage as the context, while 1B-LM uses contexts of three sentences.
['[EMPTY]', 'Model', 'CLOTH', 'CLOTH-M', 'CLOTH-H']
Hence we conclude that the LM can almost perfectly solve all short-term cloze questions. However, the performance of LM is not improved significantly when a long-term context is given, indicating that the performance gap is due to the inability of long-term reasoning.