Decision support from financial disclosureswith deep neural networks and transfer learning
Table 2: Out-of-sample results from classifying the direction of the nominal return. Values in bold indicate approaches that outperform the naïve baseline and all models from traditional machine learning.
['[BOLD] Method', '[BOLD] Training set  [BOLD] Accuracy', '[BOLD] Test set  [BOLD] Accuracy', '[BOLD] Test set  [BOLD] Balanced accuracy', '[BOLD] Test set  [BOLD] AUC', '[BOLD] Absolute improvement on test set over baseline  [BOLD] Accuracy', '[BOLD] Absolute improvement on test set over baseline  [BOLD] Balanced accuracy', '[BOLD] Absolute improvement on test set over baseline  [BOLD] AUC']
We now proceed to evaluate the classifiers for predicting the direction of nominal returns. The first row reflects the performance of our naïve benchmark when using no predictor (i. e. voting the majority class). It results in an accuracy above average due to the severe class imbalance. This also explains why we compare merely the balanced accuracy in the following analysis. Among the baseline models from traditional machine learning, we find the highest balanced accuracy on the test set when using the random forest, which yields an improvement of 4.7 percentage points compared to the naïve benchmark. Its results stem from a random forest with 500 trees, where 3 variables are sampled at each split. This highlights once again the strength of the random forest as an out-of-the-box classifier.