Attentive Representation Learning with Adversarial Training for Short Text Clustering
TABLE IV: Ablation study of the proposed model.
['Dataset Metrics', 'TREC NMI', 'TREC ARI', 'TREC ACC', 'GoogleNews NMI', 'GoogleNews ARI', 'GoogleNews ACC', 'Event NMI', 'Event ARI', 'Event ACC']
Since typical adversarial perturbations take continuous values and are added to real-valued vectors like images, they are not directly applied to discrete tokens like words. It is worth noting that we do not add adversarial perturbations to the word embedding matrix. The reason is that the number of words extends far beyond that of clusters, so that adding perturbations to words instead of clusters entails more parameters. Formally, we define the cluster-level adversarial perturbations in ARL-Adv as ΔC∈RK×M. Consequently, we propose our new objective function as formulated below: J2(\emph{E},\emph{C}+ΔC)=|D|∑i=1(L1(i;\emph{E},\emph{C% }+ΔC)+L2(i;\emph{E},\emph{C}+ΔC)). (9) The optimization of the above objective function can be viewed from two aspects. For learning ΔC, the optimal condition is as follows: ΔC∗=argmaxΔCJ2(\emph{E},\emph{C}+ΔC). (10) In contrary, the optimization of E and C is to minimize J2. Typically, a norm-based constraint should be adopted to restrict the scale of ΔC. By first comparing ARL-Adv(no train w) and ARL-Adv(no train c) with ARL-Adv, we can see their performance drops significantly, showing that the learning of word representation and cluster representation, accompanied with the automatic selection of cluster, is indeed indispensable. Besides, ARL-Adv(no train c) behaves better than ARL-Adv(no train w), indicating the training of word embedding is more critical in this sense. On TREC, the two losses play similar roles and removing either of them does not obviously damage the performance. In contrary, the pairwise ranking loss is more crucial for the clustering performance in the other two datasets, and adding pointwise loss could strengthen the clustering of short text. We can infer from the results that: (1) ARL-Random obtains slightly worse results than ARL. This reveals that simply adding random perturbations without learning may not bring more useful information to the model. (2) ARL-Adv(word) performs slightly better than ARL, but not as well as ARL-Adv, especially in GoogleNews and Event. For example, ARL-Adv improves ARL-Adv(word) by 2.3% and 3% in terms of ACC in the two datasets, respectively.