  We address the problem of American Sign Language fingerspelling recognition
in the wild, using videos collected from websites. We introduce the largest
data set available so far for the problem of fingerspelling recognition, and
the first using naturally occurring video data. Using this data set, we present
the first attempt to recognize fingerspelling sequences in this challenging
setting. Unlike prior work, our video data is extremely challenging due to low
frame rates and visual variability. To tackle the visual challenges, we train a
special-purpose signing hand detector using a small subset of our data. Given
the hand detector output, a sequence model decodes the hypothesized
fingerspelled letter sequence. For the sequence model, we explore
attention-based recurrent encoder-decoders and CTC-based approaches. As the
first attempt at fingerspelling recognition in the wild, this work is intended
to serve as a baseline for future work on sign language recognition in
realistic conditions. We find that, as expected, letter error rates are much
higher than in previous work on more controlled data, and we analyze the
sources of error and effects of model variants.
