Interactive Classification by Asking Informative Questions
Table A.2: Comparison of text encoders trained on different textual inputs and evaluated on three different prediction tasks. The model uses (a) the initial queries, (b) all attribute tags, or (c) both initial queries and tags as text inputs to predict the target. Each model is evaluated using Accuracy@{1, 3}.
['[BOLD] Text Input init query', '[BOLD] Text Input tags', '[BOLD] Init Query Acc@1', '[BOLD] Init Query Acc@3', '[BOLD] Init Query + Tags Acc@1', '[BOLD] Init Query + Tags Acc@3', '[BOLD] Init + Paraphrase Query Acc@1', '[BOLD] Init + Paraphrase Query Acc@3', '[BOLD] Full Data Acc@1', '[BOLD] Full Data Acc@3']
We use initial queries as well as paraphrase queries to train the encoder, which has around 16K target-query examples. To analyze the effectiveness of tags in addition to initial queries, we generate pseudo-queries by combining existing queries with sampled subset of tags from the targets. This augmentation strategy is useful to improve the classification performance. We also observe that using the set of tags instead of initial queries as text inputs for a specific target label improves classification performance, indicating that the designed tags can capture the target label well. Finally, when we concatenate user initial queries and tags and use that as text input to the classifier, we achieve Accuracy@1 of 76%. In our full model, we achieve 79% with only querying about five tags.