Investigating Self-Attention Network for Chinese Word Segmentation
Table 3: Effect of numbers of heads and layers
['[BOLD] #Layer', '1', '2', '3', '4', '5', '6']
Width and Depth Vaswani et al. We investigate the effect of number of layers on CWS, by increasing the number of layers from 1 to 6 while fixing the number of heads to 8. The model achieves the best F1-value 0.956 within 3 layers, after which the performance decreases with the increasing of layers. The F1-value decreases to 0.931 when the number of layers is set to 6. We fix the number of layers to 2 for the remaining experiments. We vary the number of heads in multi-head self-attention to investigate its effect on CWS performance. The number of layers and dimension of head is fix to 2 and 64, respectively. We fix the number of heads to 8 for the remaining experiments.