Fingerspelling recognition in the wild with iterative visual attention
Table 2: Results on ChicagoFSWild/dev; training on ChicagoFSWild/train. Ours+X: iterative attention (proposed method) applied to input obtained with X. +LM: add language model trained on ChicagoFSWild/train.
['[BOLD] Method', '[BOLD] Letter accuracy (%)']
In both face ROI and whole frame setups, the hand corresponds to only a small portion of the input image. Though no supervision regarding the hand is used for training, the location of the signing hand is implicitly learned through the attention mechanism. Higher recognition accuracy suggests that the learned attention locates the hand more precisely than a separately trained detector. To test this hypothesis, we measure hand detection performance on the dev set from the hand annotation data in ChicagoFSWild. At the same miss rate of 0.158, the average IoU’s of the attention-based detector and of the separately trained hand detector are 0.413 and 0.223, respectively. Qualitatively, we also compare the sequences of signing hands output by the two detectors. See supplementary material for more details.