  Physical computing infrastructure, data gathering, and algorithms have
recently had significant advances to extract information from images and
videos. The growth has been especially outstanding in image captioning and
video captioning. However, most of the advancements in video captioning still
take place in short videos. In this research, we caption longer videos only by
using the keyframes, which are a small subset of the total video frames.
Instead of processing thousands of frames, only a few frames are processed
depending on the number of keyframes. There is a trade-off between the
computation of many frames and the speed of the captioning process. The
approach in this research is to allow the user to specify the trade-off between
execution time and accuracy. In addition, we argue that linking images, videos,
and natural language offers many practical benefits and immediate practical
applications. From the modeling perspective, instead of designing and staging
explicit algorithms to process videos and generate captions in complex
processing pipelines, our contribution lies in designing hybrid deep learning
architectures to apply in long videos by captioning video keyframes. We
consider the technology and the methodology that we have developed as steps
toward the applications discussed in this research.
