NEZHA: Neural Contextualized Representation for Chinese Language Understanding
Table 5: Ablation studies. (PAPE: parametric absolute positional encoding; PRPE: parametric relative positional encoding; FRPE: functional relative positional encoding; WWM: whole word masking; SL: sequence length.)
['Model', 'CMRC EM', 'CMRC F1', 'XNLI Dev', 'XNLI Test', 'LCQMC Dev', 'LCQMC Test', 'PD-NER Dev', 'PD-NER Test', 'ChnSenti Dev', 'ChnSenti Test']
The results demonstrate that the techniques mentioned above generally have positive contributions to the downstream tasks, where functional relative positional encoding shows a notable advantage compared with other positional encoding methods. For instance, we can see that when trained with a maximum of 128 tokens, the model using the absolute positional encodings performs significantly worse than those using relative positional encodings on the CMRC task, where the input passages can be much longer.