A Study of Question Effectiveness Using Reddit “Ask Me Anything” Threads
Table 4: Results on 10k valid set. Baseline CNN performance with different window sizes and varying numbers of filters per window.
['[BOLD] Filters', '[BOLD] 1-gram', '[BOLD] 2-gram', '[BOLD] 3-gram', '[BOLD] 4-gram', '[BOLD] 5-gram']
Because the baseline CNN considers an unoptimized combination of context length (window size) and context variety (number of filters), we are curious to know how the two factors individually affect its efficacy in predicting question effectiveness. A set of experiments are conducted with varing window sizes of {1, 2, 3, 4, 5}-gram and number of filters of {5, 100, 300}. Results are reported on the 10k validation set. When there are only 5 filters (first row), the best performance was achieved using a context window of 4 words. It appears that the neural model may attempt to memorize a few cue phrases that are particularly good for making predictions. When there are 300 filters, the model yields the best performance using a window of 1-word. It seems the network relies on a combination of unigrams to make the prediction. The context-aware CNN model in this study combines five sub-optimal baseline CNN settings, including (1-gram, 5), (2-gram, 100), (3-gram, 300), (4-gram, 300), and (5-gram, 300) and yields a better performance of 0.537 on the same dataset.