Differentiable Scheduled Sampling for Credit Assignment
Table 1: Result on NER and MT. We compare our approach (α-soft argmax with fixed and annealed temperature) with standard cross entropy training (Baseline CE) and discontinuous scheduled sampling (bengioss). ‘Greedy’ and ‘Sample’ refer to Section 3.1 and Section 3.2.
['Training procedure', 'NER (F1) Dev', 'NER (F1) Dev', 'NER (F1) Test', 'NER (F1) Test', 'MT (BLEU) Dev', 'MT (BLEU) Dev', 'MT (BLEU) Test', 'MT (BLEU) Test']
We perform experiments with machine translation (MT) and named entity recognition (NER). Data: For MT, we use the same dataset (the German-English portion of the IWSLT 2014 machine translation evaluation campaign Cettolo et al. For named entity recognition, we use the CONLL 2003 shared task data Tjong Kim Sang and De Meulder We perform no preprocessing on the data. The output vocabulary length for MT is 32000 and 10 for NER. Implementation details: For MT, we use a seq2seq model with a simple attention mechanism Bahdanau et al. For NER, we use a seq2seq model with an LSTM encoder (1 layer, 64 units) and an LSTM decoder (1 layer, 64 units) with a fixed attention mechanism that deterministically attends to the ith input token when decoding the ith output, and hence does not involve learning of attention parameters. We start by training with actual ground truth sequences for the first epoch and decay the probability of selecting the ground truth token as an inverse sigmoid Bengio et al. a decay strength parameter k. We also tuned for different values of α and explore the effect of varying α exponentially (annealing) with the epochs. To account for variance across randomly started runs, we ran multiple random restarts (RR) for all the systems evaluated and always used the RR with the best validation set score to calculate test performance. ‘Greedy’ in the table refers to scheduled sampling with soft argmax decisions (either soft or hard) and ‘Sample’ refers the corresponding reparametrized sample-based decoding scenario. We compare our approach with two baselines: standard cross-entropy loss minimization for seq2seq models (‘Baseline CE’) and the standard scheduled sampling procedure (\newcitebengioss). We report results for two variants of our approach: one with a fixed α parameter throughout the training procedure (α-soft fixed), and the other in which we vary α exponentially with the number of epochs (α-soft annealed).