Semantically-Enriched Search Engine for Geoportals: A Case Study with ArcGIS Online
Table 1: Evaluation results of the semantic query expansion model and baseline
['[EMPTY]', 'Query', 'Lucene Baseline  [BOLD] DCG@3', 'Lucene Baseline  [BOLD] DCG@5', 'Lucene Baseline  [BOLD] DCG@10', 'Semantic Search  [BOLD] DCG@3', 'Semantic Search  [BOLD] DCG@5', 'Semantic Search  [BOLD] DCG@10']
A collection of user search logs is an ideal benchmark dataset to evaluate the presented framework as well as the Lucene baseline as Jiang et al. As the search logs are not available for the current project, we decide to build our own evaluation dataset. The benchmark dataset construction process can be summarized as follows: We collect a query set which consists of 20 queries. The first 10 queries are obtained from Hu et al. For each query, we get the top 10 search results from our semantic query expansion model as well as the Lucene baseline. We create a survey form for each query and each model. Each survey form consists of one query and 10 random ordered ArcGIS Online items. Users are then asked to judge the relevance between the query and each item on an ordinal scale, with labels such as“Perfect” (4), “Good” (3), “Some Relevance” (2),“Fair” (1), and “Bad” (0). The numbers in () are used as the corresponding relevance score. To host these surveys, a crowd-facing Web interface is developed and deployed on Amazon Mechanical Turk sandbox environment. Eight users completed these surveys who are from different departments of a US university.