  In this paper, we introduce the FOCAL (Ford-OLIVES Collaboration on Active
Learning) dataset which enables the study of the impact of annotation-cost
within a video active learning setting. Annotation-cost refers to the time it
takes an annotator to label and quality-assure a given video sequence. A
practical motivation for active learning research is to minimize
annotation-cost by selectively labeling informative samples that will maximize
performance within a given budget constraint. However, previous work in video
active learning lacks real-time annotation labels for accurately assessing cost
minimization and instead operates under the assumption that annotation-cost
scales linearly with the amount of data to annotate. This assumption does not
take into account a variety of real-world confounding factors that contribute
to a nonlinear cost such as the effect of an assistive labeling tool and the
variety of interactions within a scene such as occluded objects, weather, and
motion of objects. FOCAL addresses this discrepancy by providing real
annotation-cost labels for 126 video sequences across 69 unique city scenes
with a variety of weather, lighting, and seasonal conditions. We also introduce
a set of conformal active learning algorithms that take advantage of the
sequential structure of video data in order to achieve a better trade-off
between annotation-cost and performance while also reducing floating point
operations (FLOPS) overhead by at least 77.67%. We show how these approaches
better reflect how annotations on videos are done in practice through a
sequence selection framework. We further demonstrate the advantage of these
approaches by introducing two performance-cost metrics and show that the best
conformal active learning method is cheaper than the best traditional active
learning method by 113 hours.
