Fingerspelling recognition in the wild with iterative visual attention
Table 7: Comparison of IoU between an off-the-shelf signing hand detector and a detector produced by iterative attention.
['[EMPTY]', 'Off-the-shelf\xa0', 'Iterative-Attn']
Iterative attention serves as an implicitly learned “detector” of signing hands. We compare the performance of this detector with a separately trained signing hand detector here. We convert the iterative attention ROI to an explicit detector through the following steps: take the input image of the last iteration, backtrack to the original image frame to get its coordinates, and use these coordinates as the bounding box. We take a model trained in the face ROI setting and compare it with an off-the-shelf detector. For quantitative evaluation, we take the dev set of hand annotation data in ChicagoFSWild, which includes 233 image frames from 19 sequences, and remove all frames with two signing hands. That amounts to 200 image frames in total. We compute average IoU and miss rate between the target bounding box and ground truth. The miss rate is defined as 1-intersection/ground-truth area. As the two detectors have different IoU’s and miss rates, for ease of comparison we resize the bounding box of the off-the-shelf detector to keep its miss rate consistent with that of the iterative-attention detector. Though numerical differences between IoU’s may be exaggerated due to the small amount of evaluation data, the effectiveness of our approach for localization of signing hands can also be inferred from improvements in recognition accuracy.