BERTScore: Evaluating Text Generation with BERT
Table 8: Recommended layer of representation to use for BERTScore. The layers are chosen based on a held-out validation set (WMT16).
['Model', 'Total Number of Layers', 'Best Layer']
In designing BERTScore, we use WMT16 segment-level human judgment data as a development set to facilitate our representation choice. For Chinese models, we tune with the WMT17 ‘ ‘en-zh’’ data because the language pair ‘‘en-zh’’ is not available in WMT16. Based on results from different models, we identify a common trend that F\textscBERT computed with the intermediate representations tends to work better. We tune the number of layer to use for a range of publicly available models.