F10-SGD: Fast Training of Elastic-net Linear Models for Text Classification and Named-entity Recognition
Table 2: DC results for OWL-QN vs F10-SGD and F10-SGD AB. Size is in megabytes. Time is the mean of 5 training runs in minutes. %Acc is percentage accuracy relative change compared to OWL-QN (we are unable to disclose absolute accuracy). For F10-SGD AB we performed 7 normal epochs and 3 active bias epochs.
['Optimizer', 'CPU', 'Epoch', 'Size', 'Time', '%Acc']
MaxEnt DC. The accuracy difference between F10-SGD and OWL-QN is not statistically significant, although F10-SGD accuracy is slightly higher. For DC, F10-SGD AB does not bring additional accuracy improvements. The reason is that the model learns to assign high confidence to most of the sentences in the first 7 epochs of the training, and there are not many medium confidence sentences for the active bias sampling to prioritize. The training speeds of F10-SGD and F10-SGD AB are approximately 4x faster with half the number of CPU threads compared to OWL-QN. It takes only 10 epochs for SGD to reach the same accuracy level of OWL-QN with 250 epochs. Also, we get an additional 20% speed up if we use 28 CPU threads.