Interpreting Transformed-based Models by Concept Classification
Table 5: Performance of the considered models on various downstream tasks, as reported in the reference papers. Not all models report performance on all tasks.
['Model', 'BERT-B', 'BERT-L', 'Distilbert', 'RoBERTa-L', 'XLM']
Finding an optimal γ⋆ value The choice of γ is important to compute the concept expertise Xγ in Eq. The goal is to obtain an optimal γ⋆ that produces an expertise representative of the generalization power of TMs. As a measure of generalization, we use the average performance of each model on typical downstream tasks: the 10 datasets composing GLUE with their different reported metrics [Wang:ICLR:2019] and SQuAD v1.1/2.0 [Rajpurkar:EMNLP:2016, Rajpurkar:ACL:2018].