Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction
Table 1: Table extraction results of our table parser on 50 tables from 10 NLP papers in PDF format.
['[EMPTY]', 'Macro P', 'Macro R', 'Macro F1']
We evaluated our table parser on a set of 10 papers from different venues (e.g., EMNLP, Computational Linguistics journal). In total, these papers contain 50 tables with 1,063 numeric content cells. Our table parser achieves a macro F1 score of 82.6 for identifying table captions, and 74.0 macro F1 for extracting tuples of  . In general, it obtains higher recall than precision in all evaluation dimensions. However, when we add the score extraction (TDMS), even if TDMS-IE outperforms the baselines, the overall performances are still unsatisfactory, underlining the challenging nature of the task. A qualitative analysis showed that many of the errors were triggered by the noise from the table parser, e.g., failing to identify bolded numeric scores or column headers Sometimes a few papers bold the numeric scores for methods from the previous work when comparing to the state-of-the-art results, and our model wrongly predicts these bolded scores for the targeting TDM triples.