Learning Intrinsic Sparse Structures within Long Short-Term Memory
Table 3: Remaining ISS components in BiDAF by fine-tuning.
['EM', 'F1', 'ModFwd1', 'ModBwd1', 'ModFwd2', 'ModBwd2', 'OutFwd', 'OutBwd', 'weight #', 'Total time*']
The first row is the baseline BiDAF. Other rows are obtained by fine-tuning baseline using ISS regularization. In the second row by learning ISS, with small EM and F1 loss, we can reduce ISS in all LSTMs except ModFwd1. For example, almost half of the ISS components are removed in OutBwd. By increasing the strength of group Lasso regularization (λ), we can increase the ISS sparsity by losing some EM/F1 scores. With 2.63 F1 score loss, the sizes of OutFwd and OutBwd can be reduced from original 100 to 15 and 12, respectively. At last, we find it hard to reduce ISS sizes without losing any EM/F1 score. This implies that BiDAF is compact enough and its scale is suitable for both computation and accuracy. However, our method can still significantly compress this compact model under acceptable performance loss.