AidMe
Table 4. Model evaluation on UserGrammar corpus
['[BOLD] Model', '[BOLD] F1_score mean', '[BOLD] F1_score std', '[BOLD] Precision', '[BOLD] Recall']
Model evaluation on UserGrammar To get an idea of how our model would perform on a task close to the user interaction simulation, we have evaluated our semantic similarity model on corpora generated with UserGrammar. We describe below the evaluation protocol. We generate a corpus from UserGrammar by generating: A train set of 5000 pairs from 150 different sentences and 100 different patterns; A test set of 1000 pairs from 100 different sentences and 40 different patterns which were not used to generate the train set. We report the F1_score, Precision and Recall of the models trained on the train set and evaluated on the test set. These metrics are more relevant than the Pearson correlation or a classic accuracy metric as the task is like a binary classification task with a highly unbalanced dataset (proportion of pairs with positive label between 2% and 4%). It is important to note that none of the sentences nor the patterns of the test set were seen during training.