Generating Descriptions from Structured Data Using a Bifocal Attention Mechanism and Gated Orthogonalization
Table 6: Out of domain results(BLEU-4)
['[BOLD] Training data', '[BOLD] Target (test) data  [BOLD] Arts', '[BOLD] Target (test) data  [BOLD] Sports']
What if the model sees a different type of person at test time? For example, what if the training data does not contain any sportspersons but at test time we encounter the infobox of a sportsperson. This is the same as seeing out-of-domain data at test time. Such a situation is quite expected in the products domain where new products with new features (fields) get frequently added to the catalog. We were interested in three questions here. First, we wanted to see if testing the model on out-of-domain data indeed leads to a drop in the performance. For this, we compared the performance of our best model in two scenarios (i) trained on data from all domains (including the target domain) and tested on the target domain (sports, arts) and (ii) trained on data from all domains except the target domain and tested on the target domain. Note that the numbers for sports domain in row 1 are much better than the Arts domain because roughly 40% of the WikiBio training data contains sportspersons. Next, we wanted to see if we can use a small amount of data from the target domain to fine tune a model trained on the out of domain data. Note that if we train a model from scratch with only limited data from the target domain instead of fine-tuning a model trained on a different source domain then the performance is very poor. In particular, training a model from scratch with 10K training instances we get a BLEU score of 16.2 and 28.4 for arts and sports respectively. Finally, even though the actual words used for describing a sportsperson (footballer, cricketer, etc.) would be very different from the words used to describe an artist (actor, musician, etc.) they might share many fields (for example, date of birth, occupation, etc.). However, once we fine tune the model with limited data from the target domain we see that it picks up the new vocabulary and produces a correct description of the occupation.