SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models
TABLE V: Experimental results on 10 probing tasks, where the best results are shown in bold face.
['[BOLD] Model', '[BOLD] Dim', 'Surface  [BOLD] SentLen', 'Surface  [BOLD] WC', 'Syntactic  [BOLD] TreeDepth', 'Syntactic  [BOLD] TopConst', 'Syntactic  [BOLD] BShift', 'Semantic  [BOLD] Tense', 'Semantic  [BOLD] SubjNum', 'Semantic  [BOLD] ObjNum', 'Semantic  [BOLD] SOMO', 'Semantic  [BOLD] CoordInv']
We see that SBERT-WK yields better results than SBERT in all tasks. Furthermore, SBERT-WK offers the best performance in four of the ten tasks. By merging information from various layers, SBERT-WK can take care of these different aspects.