Edina: Building an Open Domain Socialbot with Self-dialogues
Table 3: Average human evaluation of matching score and real human responses, plus or minus standard error. Confidence threshold of 0.5 corresponded to confidence threshold for matching score component to response. Confidence threshold of 0.7 corresponded to confidence threshold for matching score component to move ahead in the queue of the likes and dislikes component.
['Response type', 'Mean human evaluation score']
We collected 181 trials of human evaluations of top-4 matching score responses and 33 trials of human evaluations of real responses from AMT workers. We found that the confidence scores of our top-4 matching score responses were positively correlated with human evaluation scores (r=0.259, p<0.0005). Edina demonstrated the ability to engage in a wide range of subtopics during conversations with Alexa customers. This was achieved by drawing responses from a conversation dataset created with the novel technique of self-dialogues. This technique plays a significant role in enabling our agent to generate relevant, topical output closely resembling human conversations. Our internal system evaluation further demonstrated the value of using a thresholded matching score to select a response.