LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
Table 5: Results of human evaluation metrics on the test set (higher is better). All these models are trained and tested give both videos and surrounding comments.
['[BOLD] Model', '[BOLD] Fluency', '[BOLD] Relevance', '[BOLD] Correctness']
We compare our Fusional RNN model and Unified Transformer model with the strong baseline model S2S-IC. All these models are trained and tested give both videos and surrounding comments. We also evaluate the reference comments in the test set, which are generated by the human. It shows that the comments from human achieve high fluency and correctness scores. However, the relevance score is lower than the fluency and correctness, mainly because the comments are not always relevant to the videos, but with the surrounding comments. According to the table, it also concludes that the comments from unified transformer are almost near to those of real-world live comments. We use Spearman’s Rank correlation coefficients to evaluate the agreement among the raters. The coefficients between any two raters are all near 0.6 and at an average of 0.63. These high coefficients show that our human evaluation scores are consistent and credible.