Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
Table 2: Properties of the QA Datasets.
['[BOLD] Dataset', '[BOLD] Train', '[BOLD] Valid', '[BOLD] Test', '[BOLD] Example Questions']
The questions are collected using Google Suggest API and are all asking about simple relational facts of Freebase entities. Apart from a small portion of questions to which the answers are numbers and free texts, 92.85% of the answers are Wikipedia entities. Most of the answers in this dataset are none phrases. and we also find that almost all of the answers are real-world entities. Questions in all three datasets are created without the context of a paragraph, which resembles the scenario of practical question answering applications. All the questions except WebQuestions are written by humans. This indicates that humans are generally interested to ask questions to seek information about entities. We split the training data (created by distant supervision) of WebQuestions with a ratio (9:1) for training and development. Since our model is based on our own BERT implementations, in addition to the aforementioned entity-related datasets, we first use the standard SQuAD