Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition
Table 5: Human preference on dialog session pairs that MADPL wins (W), draws with (D) or loses to (L) baselines with regard to quality (Q) and success by majority voting.
['VS.', 'System Q W', 'System Q D', 'System Q L', 'User Q W', 'User Q D', 'User Q L', 'Success W', 'Success D', 'Success L']
We can observe that the high win rate of MADPL on the task success is consistent with the results of automatic evaluation, and MADPL outperforms three baselines significantly in all aspects (sign test, p-value < 0.01) except for the system quality against RL/RL policies.