Empirical Analysis of Foundational Distinctions in Linked Open Data
Table 2: POC dataset: crowd-based annotated dataset of physical objects. The table provides an insight of the dataset per level of agreement. Agreement values computed according to Formula 1.
['[BOLD] Agreement', '[BOLD] # Physical Object', '[BOLD] # ¬ Physical Object', '[BOLD] Total']
POC Dataset: Physical Object Annotation Performed by Crowd. We also crowdsourced the annotations of physical objects vs. not a physical object: the workers were asked to perform this task by using the entity’s name, its abstract, and Wikipedia page as reference descriptions. The quality of the workers has been assessed with 49 test questions, used to exclude contributors that scored an accuracy lower than 80% We collected 25,776 judgments from 173 workers. Each entity has been annotated by at least 5 different English speakers. The average agreement of the crowd’s annotations is 85.48% . The agreement between the crowd and the experts is 85,69%, suggesting that the PO vs NPO foundational distinction also matches common sense (cf. RQ1 applied to this distinction).