Empirical Analysis of Foundational Distinctions in Linked Open Data
Table 1: CIC dataset crowd-based annotated dataset of classes and instances. The table provides an insight of the dataset per level of agreement. Agreement values computed according to Formula 1.
['[BOLD] Agreement', '[BOLD] # Class', '[BOLD] # Instance', '[BOLD] Total']
CIC Dataset: Class vs. Instance Annotation Performed by Crowd. The same dataset was then crowdsourced: each worker was asked to indicate whether an entity is a class or an instance based on its name, abstract, and its corresponding Wikipedia article. We want to assess the agreement between the experts and the crowd, which indicates whether foundational distinctions match common sense or not (cf. RQ1). The quality of the contributors has been assessed with 51 test questions with a tolerance of only 20% of errors. We collected 22,510 judgments from 117 contributors: each entity was annotated by at least 5 different workers. For each entity e, we computed the level of agreement on each class c, weighted by the trustworthiness scores agreement(e,c)=SumOfTrust(e,c)SumOfTrustOfWorkers(e) (1) where SumOfTrust(e,c) is the sum of the trustworthiness scores of the workers that annotated entity e with class c; and SumOfTrustOfWorkers(e) reports the results of the task indicating the distribution of classes and instances per level of agreement. The average agreement of the crowd is 95.76% .