Natural Language Multitasking Analyzing and Improving Syntactic Saliency of Latent Representations
Table 2: Clustering errors by model
['Model', 'REP', 'REP-FR', 'REP-FR-DE', 'REP-DE', 'REP-POS', 'REP-DE-POS']
Each sentence prototype is randomly populated by common English words 100 times. The syntax of each sentence in such a category is very similar or identical to all others in the same category, but different from sentences in other categories. These sentences are then fed into our models. We record every resulting representation and pair it with its input sentence. Using K-means clustering with K=14 we cluster the representation-sentence pairs in the representation space. For each resulting cluster, we count how many sentences of each prototype it contains. This yields a list such as this: [30,3,1,7,88,0,0,0,0,0,0,0,0,0], which shows the content of one of 14 clusters: 30 sentences of type one, 3 of type two and so on. Since most sentences in this cluster are of type five, this cluster is assigned to be the cluster of sentence category five. However, 41 sentences of type other than five were “falsely” assigned to this cluster. Therefore, the error of this cluster is 41. The sum of errors of all 14 clusters is the clustering error, which is our quality measure for this experiment. Since K-means clustering is nondeterministic, we run the algorithm 100 times.