Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Document Matching
Table 4. Comparison of different models over Wiki65K and AAN104K data sets. The best performance is highlighted in boldface. SMITH-WP+SP shows significant improvements over all baseline methods with p ¡ 0.05 measured by Student’s t-test. Note that SMITH-Short with input documents with maximum length larger than 512 and MatchBERT with input documents with maximum length larger than 256 will trigger the out-of-memory (OOM) issues on TPU V3. There is ×2 in the DocLen, which denotes the maximum document length, since all the compared models are “Dual-Encoder”/ “Siamese” models. Note that all the compared models are for the document/document matching task. Methods designed for the query/document matching task are not comparable.
['Method', 'Data DocLen', 'Wiki65K Accuracy', 'Wiki65K Precision', 'Wiki65K Recall', 'Wiki65K F1', 'AAN104K Accuracy', 'AAN104K Precision', 'AAN104K Recall', 'AAN104K F1']
We summarize our observations as follows: 1) Both SMITH-WP and SMITH-WP+SP models outperform all the baseline methods including the stage-of-the-art long text matching method SMASH and MatchBERT based on pre-trained BERT models on both Wiki65K and AAN104K consistently. The comparison between SMITH-WP/ SMITH-Short and MatchBERT shows the effectiveness of introducing hierarchical document structure modeling with sentence level and document level Transformers for long document representation learning and matching. 2) If we compare the SMITH model settings with the pre-training stage (SMITH-Short, SMITH-WP, SMITH-WP+SP) with the SMITH model settings without the pre-training stage (SMITH-NP), we can find that language modeling pre-training can help increase the performance of the downstream document matching task by a large margin. Thus better language understanding via large scale language modeling pre-training will lead to better downstream task performance, which is consistent with the findings by Devlin et al. (devlin2018bert). 3) Both SMITH-WP and SMITH-WP+SP outperform SMITH-Short which is initialized by the pre-trained open source BERT model. We think the main reason for these results is that currently SMITH-Short can only process at most 512 tokens due to TPU memory issues, which will hurt the document matching performance. On the other hand, Both SMITH-WP and SMITH-WP+SP can process as long as 2048 tokens, which is a better setting for long document representation learning and matching. 4) The masked word prediction task proposed by Devlin et al. (devlin2018bert) can capture the word relations and dependencies in the pre-training corpus, whereas the masked sentence block prediction task can additionally force the model to learn the sentence-level relations and dependencies. Thus combining the masked word prediction task and the masked sentence block prediction task can contribute to a better pre-training language model for long document content understanding and better downstream document matching task performance.