User Memory Reasoning for Conversational Recommendation
Table 5: Results of both offline and online evaluation: EMR stands for entity matching rate, which compares all types of predicted entities against annotated ones when the dialog act is predicted correctly; IMR stands for item matching rate, which evaluates predicted items against the ground-truth item across all turns in a dialog.
['[BOLD] Methods', '[BOLD] Offline Evaluation  [BOLD] Act Acc.', '[BOLD] Offline Evaluation  [BOLD] Act F1', '[BOLD] Offline Evaluation  [BOLD] EMR', '[BOLD] Offline Evaluation  [BOLD] EMR', '[BOLD] Offline Evaluation  [BOLD] EMR', '[BOLD] Offline Evaluation  [BOLD] IMR', '[BOLD] Online Evaluation  [BOLD] Success Rate']
Overall, it can be seen that the proposed UMGR architecture outperforms other baselines in both offline and online evaluation. Ablations: Specifically, we notice that dynamically updating the user memory graph with users’ new preference is crucial for a recommendation, as indicated by UMGR - static G that forbids updating user memory graph. It can also be seen that removing the previous dialog context does degrade the performance as expected (UMGR - Prev. User Act Only), although the UMGR architecture still maintains a competitive performance. Similarly, while UMGR -No Dialog Acts does not take past dialog acts as input, its results on non-act prediction metrics are relatively competitive. Both of these ablation studies indicate the user memory graph contains enough information for the model to perform dialog reasoning.