Generating Titles for Web Tables
Table 2. Comparison of relevance and readability (from human evaluation) and ROUGE metrics for 200 titles. Results are grouped by category (baselines, neural networks, crowdsourced) and shown graphically in Figure 7. The crowdsourced titles have no ROUGE scores because they are the ground truth.
['[BOLD] Model', '[BOLD] Relevance', '[BOLD] Readability', '[BOLD] ROUGE-1', '[BOLD] ROUGE-2', '[BOLD] ROUGE-L']
Evaluators assessed the readability and relevance of each title on a 3-point scale with the following interpretations: for readability, 1 = Poor, 2 = Medium, 3 = Well. For relevance, 1 = Needs Not Met , 2 = Needs Somewhat Met, 3 = Needs Fully Met. In each case, we found that the heuristic we used biased the training data enough to significantly hinder its ability to generalize. Consequently, the results reported in this paper all correspond to models trained with the crowdsourced data. Interestingly, while Copy + Generate had the strongest performance, Copy Only and Generate Since both readability and relevance are required in any real-world use case, it makes sense that historically, simpler selection-based approaches have been preferred to generation-based ones. Combining the two mechanisms, however, results in a model that performs better than both single-mechanism models in all metrics.