CLUE: A Chinese Language Understanding Evaluation Benchmark
Table 4: The CLUE diagnostics: Example test items in 9 linguistic categories, with their gold labels and model predictions, as well as model accuracy. E = entailment, N = neutral, C = contradiction. BE = BERT-base, RO = RoBERTa-wwm-ext-large, XL = XLNet-mid.
['[EMPTY]', '#', '[BOLD] Premise', '[BOLD] Hypothesis', 'gold', '[BOLD] Predictions BE', '[BOLD] Predictions RO', '[BOLD] Predictions XL', '[BOLD] Accuracy BE', '[BOLD] Accuracy RO', '[BOLD] Accuracy XL']
In order to examine whether the trained models can master linguistically important and meaningful phenomena, we follow GLUE (wang2018glue) to provide a diagnostic dataset, setting up as a natural language inference task and predicting whether a hypothesis is entailed by, contradicts to or is neutral to a given premise. Crucially, we did not translate the English diagnostics into Chinese, as the items in their dataset may be specific to English language or American/Western culture. Instead, we have several Chinese linguists hand-crafting 514 sentence pairs in idiomatic Chinese from scratch. These pairs cover 9 linguistic phenomena and are manually labeled by the same group of linguists. We ensured that the labels are balanced (majority baseline is 35.1%). Some of the categories directly address the unique linguistic properties of Chinese. For instance, items in the “Time of event” category test models on their ability to handle aspect markers such as 着 (imperfective marker), 了 (perfective marker), 过 (experiential marker), which convey information about the time of event, whether it is happening now or has already happened in the past. We believe that for a model to make robust inferences, it needs to understand such unique Chinese phenomena, and also has other important linguistic abilities, such as handling anaphora resolution (webster2018mind) and monotonicity reasoning (yanaka2019can; probe2020). We evaluate three representative models on the diagnostic dataset: BERT-base, XLNet-mid, RoBERTa-wwm-ext-large. Each model is fine-tuned on CMNLI training set, translated from English MNLI dataset into Chinese, then tested on our diagnostic dataset. We believe that both models and inference datasets suggest room for improvement. Monotonicity is the hardest, similar to GLUE diagnostics (wang2018glue). It seems that XLNet-mid also has a hard time dealing with double negation, highlighting the difficulty of monotonicity for some models. Thus the premise and hypothesis only differ in the last character, which all three models have decided to ignore. One possible explanation is that the current state-of-the-art models in Chinese are also using the simple lexical overlap heuristic, as illustrated in hans for English.