Mono vs Multilingual Transformer-based Models: a Comparison across Several Language Tasks
Table 2: RTE and STS evaluated on Test Sets.
['Data', 'Model', 'RTE Acc', 'RTE F1-M', 'STS Pearson', 'STS MSE']
Next, we applied our fine-tuned model to the entire training set and ran the evaluation over the test set. The first observation is that the BP setting is difficult for all models. The EP+BP * * in the last rows indicates that the model was trained using both EP and BP training sets, but they were evaluated only using the EP test set. The results of this evaluation can be compared to the first rows, in which the models were trained and evaluated only on EP test set. Although it makes sense to think that more training data would help the model generalize, we reached the same conclusion as FialhoMarquesMartinsCoheurQuaresma2016 that the improvement, if any, in this dataset is meaningless. We could not find a published baseline for the case where the model was trained using EP+BP and evaluated using both test sets. Again, as expected, BERT models were superior to previous baselines for all combinations. Analysing 10-fold evaluation and using only the test set, we can conclude that BERT-based models were superior to previous baselines. The goal of Semantic Textual Similarity is to quantify the degree of similarity between two sentences. In the ASSIN dataset, similarity is a number ranging from 1 (no similarity) to 5 (high similarity). This is a linear regression problem as the output is a similarity score. The evaluation measures how far the predicted score is from the ground truth using two metrics: ( i) Pearson Correlation which measures the correlation with the ground truth, so the higher, the better, and (ii) and the Mean Squared Error which measures the square of the difference between the prediction and the ground truth, so the lower the better. Following the same procedure of Textual Entailment, we used 3,000 pairs for training. Since STS is related to RTE, again, all models obtained better results than the baselines. Also, the multilingual model was again superior to our pre-trained Portuguese models.