SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Check
Table 5: The ablation results for graph combination method (%). The averaged character-level C-F scores of 4 runs on the SIGHAN 2013 are reported. The models were trained with 10K training samples. Mean pooling denotes that the output representation C^{l} of each layer is the average of f_{k\in\{P,S\}}(\mathbf{A}_{k},\mathbf{H}^{l}), while sum pooling summarizes f_{k\in\{P,S\}}(\mathbf{A}_{k},\mathbf{H}^{l}).
['combination method', 'C-F']
We investigate how to better combine the graphs in the SpellGCN layer. Here, we compare the attention mechanism against sum-pooling and mean-pooling, with different hyper-parameter \beta mentioned in Section The experiments are conducted based on the 2-layer SpellGCN on SIGHAN 2013 test set. We suggest that the sum pooling is inconsistent with the normalization of GCN and fails to combine the information from different channels (i.e., graphs). The mean pooling is feasible but is surpassed by the attention mechanism. This indicates that the adaptive combination for each character node is beneficial. We incorporate a hyper-parameter \beta into the attention operation since the dot products may grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients. With these results, we chose the attention mechanism with a \beta of 3 in SpellGCN.