Story Ending Prediction by Transferable BERT
Table 2: Experimental results with all the publicly released pre-trained BERT models on SCT_v1.0 test dataset. ‘Uncased’ means all words in the training corpus will be transformed into lower case form. ‘Cased’ means keeping all the words in their original form.
['[BOLD] Methods', '[BOLD] Accuracy (%)']
They differ in how many layers and parameters are used in the model (the basic version has 12-layer transformer blocks, 768 hidden-size, and 12 self-attention heads, totally 110M parameters; the large version has 24-layer transformer blocks, 1024 hidden-size, and 16 self-attention heads, totally 340M parameters), and what kind of datasets are used to pre-train the model (multilingual or monolingual). We first conduct several comparative experiments on SCT_v1.0 dataset to study the effectiveness of different BERT versions. We find that the two multilingual models perform much worse than the monolingual models. An uncased BERTBASE performs better than the cased BERTBASE, but a cased BERTLARGE is better than the uncased BERTLARGE. The reasons are that the multilingual BERT model doesn’t improve the performance on the monolingual SCT english dataset; the BERTLARGE model can handle a larger cased vocabulary with much more parameters but the BERTBASE model cannot.