Multi-task Learning for Universal Sentence Embeddings: A Thorough Evaluation using Transfer and Auxiliary Tasks
Table 1: Best β and γ values for adversarial shared private model on different set of tasks.
['Tasks', '[ITALIC] β', '[ITALIC] γ']
Adversarial Training. Ideally, we want the private encoders to learn only task-specific features, and the shared encoder to learn generic features. To achieve this goal, we adopt the adversarial training strategy proposed by Liu et al. The goal of the discriminator, D is to identify which task an encoded sentence sk comes from, and the adversarial training requires the shared sentence encoder to generate representations that can “fool” the discriminator. In this way, the shared encoder is forced not to carry task-related information. The discriminator is defined as, D(sk)=\it softmax(Wsk+b), where W∈Rd×d and b∈Rd are model parameters. Optimizing the adversarial loss, Ladv=minθE(maxθD(K∑k=1Nk∑i=1dkilog[D(E(w))])) has two competing goals: the discriminator tries to maximize the classification accuracy (inside the parentheses), and the sentence encoder tries to confuse it (and thus minimize the classification accuracy). E and D represents the shared sentence encoder and the discriminator respectively and θE and θD are the model parameters of E and D. dki denotes the ground-truth label indicating the type of the current task. To encourage the shared and private encoders to capture different aspects of the sentences, the following term is added. Ldiff=∑Kk=1∥∥Hk⊤sHkp∥∥2F. where ∥⋅∥2F is the squared Frobenius norm. Here, Hks and Hkp are matrices where rows are the hidden vectors (see Eq. (1)) generated by the shared and private encoders given an input sentence of task k. The final loss function is a weighted combination of three parts: L =Lmulti−task+βLadv+γLdiff, where β and γ are hyper-parameters, Lmulti−task refers to a simple summation over the cross entropy loss for each task. We tune β and γ in the range [0.001,0.005,0.01,0.05,0.1,0.5] and found different combinations perform best for different models.