Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Document Matching
Table 5. The document matching performance with different choices of the number of layers in the sentence level Transformers on the validation data. L1 denotes the number of layers in the sentence level Transformers.
['Data', '[ITALIC] L1', 'Accuracy', 'Precision', 'Recall', 'F1']
Next we analyze the impact of the number of layers in the sentence level Transformers on the final document matching performances. We set the document level Transformer layers as 3, the maximum sentence block length as 32, the number of sentence blocks per document as 48. So the maximum document length is 1536. Then we vary the number of layers in the sentence level Transformers and observe the change of the performances of SMITH-WP+SP. We can find that the setting with 4 or 6 layers in the sentence level Transformer layers is slightly better than the setting with only 2 layers in the sentence level Transformers. Increasing the layers in the sentence level Transformers can help the model to learn sentence block semantic representation with more high level interactions. However, it also leads to larger memory cost such as the intermediate activation in each layer. Thus in practice this hyper-parameter has to be tuned with the validation data set for the trade-off between the model capacity and memory cost.