Embodied Question Answering
(a) Segmentation
['[EMPTY]', 'Pixel Accuracy', 'Mean Pixel Accuracy', 'Mean IOU']
Entropy+Frequency-based Filtering: It is important to ensure that the distribution of answers for a question is not too ‘peaky’, otherwise the mode guess from this distribution will do unreasonably well as a baseline. Thus, we compute the normalized entropy of the distribution of answers for a question. We drop questions where the normalized entropy is below 0.5. Further, we also drop questions that occur in less than 4 environments because the entropy counts for low-frequency questions are not reliable. Object Instance Count Threshold: We do not ask counting questions (room_count and count) when the answer is greater than 5, as they are tedious for humans. Object Distance Threshold: We consider triplets of objects within a room consisting of an anchor object, such that the difference of distances between two object-anchor pairs is at least 2 metres. This is to avoid ambiguity in ‘closer’/‘farther’ object distance comparison questions. Collapsing Object Labels: Object types that are visually very similar (e.g. ‘teapot’ and ‘coffee_kettle’) or semantically hierarchical in relation (e.g. ‘bread’ and ‘food’) introduce unwanted ambiguity. In these cases we collapse the object labels to manually selected labels (e.g. (‘teapot’, ‘coffee_kettle’) → ‘kettle’ and (‘bread’, ‘food’) → ‘food’).