Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Document Matching
Table 6. The document matching performance with different choices of the document representation combining methods presented in Section 4.1.4 on the validation data.
['Data', 'Combine', 'Accuracy', 'Precision', 'Recall', 'F1']
We evaluate the performances of SMITH-WP+SP with different methods to combine representations from different levels including “normal”, “sum-concat”, “mean-concat” and “attention”. We can see that the “normal” combing method where we only use the output of the document level Transformers as the final document representation works best. For the other three methods, the “attention” method is better than “sum-concat” and “mean-conat”. One possible reason is that the weighted combination of sentence blocks can be helpful for generating better document representations as the attention weights can encode the relative importance of different sentence blocks on representing the document content, which is why attention based combining methods work better. The document level Transformers already learn a weighted combination based on the input sentence block representation sequences, which already provide enough signals on the importance scores of different sentence blocks in a document.