Heuristic Feature Selection for Clickbait Detection
Table 1: Mean squared error (MSE) of selected models on a 2:1 training-validation-split of the Clickbait Challenge 2017 training dataset (left), and means squared error of selected submitted approaches on the challenge’s test dataset (right).
['[BOLD] Participant', '[BOLD] Rank', '[BOLD] MSE']
In total, we processed one million remove-train-predict iterations for our feature assessment (i.e., a feature was removed from an average of 26 feature subsets). After the outputs of the iterations were accumulated, the leave-many-out scores per feature was computed to rank the features. From this ranking, we then choose feature subsets of increasing size to examine the effect of removing the lowest ranking features and training a model on the remaining ones. We constructed 49 subsets including the top-ranked 100%, …, 2% of the features in steps of 2%, and three other subsets containing only the top-ranked 1.5%, 1.0%, and 0.5% of the features. For every subset, we trained a ridge regression model on 2/3 of the training data and validated it on the remaining 1/3 of the training data. In this case the split was kept constant for all the 52 feature subsets. The best-performing model in this experiment series achieves a mean squared error of 0.0297 with 12,008 features (32.0% of the original features). This corresponds to about 10% performance improvement over the full feature set (see The smallest subset size that does not perform worse than the full feature set uses 375 features (1.0%). This means that Icarfish, based only on its heuristic feature selection, achieves a performance gain of about 20% over the baseline and would have been placed second in the challenge.