Natural Language Generation at Scale:A Case Study for Open Domain Question Answering
Table 3: Objective metrics results on three QA NLG datasets with increasingly larger ontologies. The models under comparison are a baseline with two encoders, for slot types and slot values, and its extensions with a delexicalised or lexicalised previous utterance. While for BLEU score the higher the better, for all types of Slot Error Rate (SER) the lower the better.
['[EMPTY]', '[BOLD] QA.1  [BOLD] BLEU', '[BOLD] QA.1  [BOLD] SERmr', '[BOLD] QA.1  [BOLD] SERtrg', '[BOLD] QA.1  [BOLD] SERmtrg', '[BOLD] QA.2  [BOLD] BLEU', '[BOLD] QA.2  [BOLD] SERmr', '[BOLD] QA.2  [BOLD] SERtrg', '[BOLD] QA.2  [BOLD] SERmtrg', '[BOLD] QA.3  [BOLD] BLEU', '[BOLD] QA.3  [BOLD] SERmr', '[BOLD] QA.3  [BOLD] SERtrg', '[BOLD] QA.3  [BOLD] SERmtrg']
For the NLG models for QA, experiments on QA.1 (not reported due to space limitations) with different encoders combinations showed that the best performances were achieved using all input types (slot type, value, and previous context) with lexicalized (+ Enc Utterance lex) or delexicalized (+ Enc Utterance delex) previous context in terms of all metrics, except SERtrg. On this metric, the architecture with slot types and values, but without the previous context (Enc MR (slot types, values)) achieved the best performance (cf. For this reason, we chose to report the performances of these architectures in our QA experiments. However, SERmr results show the opposite picture, where the baseline with only slot types and values performs better (except for QA.2 where the score is close to the model with the delexicalized input) and the model with the lexicalized previous utterance is the worst. SERmtrg shows, on the other hand, that the context might slightly degrade performances with bigger ontologies in terms of all text references.