Differentiable Scheduled Sampling for Credit Assignment
Table 2: Effect of different schedules for scheduled sampling on NER. k is the decay strength parameter. Higher k corresponds to gentler decay schedules. Always refers to the case when predictions at the previous predictions are always passed on as inputs to the next step.
['k', '100', '10', '1', '[ITALIC] Always']
The computational complexity of our approach is comparable to that of standard seq2seq training. However, instead of a vocabulary-sized max and lookup, our approach requires a matrix multiplication. Practically, we observed that on GPU hardware, all the models for both the tasks had similar speeds which suggests that our approach leads to accuracy gains without compromising run-time. We also observed that the ‘always sample’ case of relaxed greedy decoding, in which we never mix in ground truth inputs (see \newcitebengioss), worked well for NER but resulted in unstable training for MT. We reckon that this is an effect of large difference between the search space associated with NER and MT.