What you can cram into a single $&!#⁢* vector:Probing sentence embeddings for linguistic properties
Table 4: Probing task accuracies with Logistic Regression. Taking pre-learned sentence embeddings as input.
['[BOLD] Task', '[BOLD] SentLen', '[BOLD] WC', '[BOLD] TreeDepth', '[BOLD] TopConst', '[BOLD] BShift', '[BOLD] Tense', '[BOLD] SubjNum', '[BOLD] ObjNum', '[BOLD] SOMO', '[BOLD] CoordInv']
This suggests that most linguistic properties can be extracted with a linear readout of the embeddings. Interestingly, if we focus on a good model-training combination, such as BiLSTM-max trained on French NMT, the tasks where the improvement from logistic regression to MLP is relatively large (>3%) are those arguably requiring the most nuanced linguistic knowledge (TreeDepth, SOMO, CoordInv).