  Automating video-based data and machine learning pipelines poses several
challenges including metadata generation for efficient storage and retrieval
and isolation of key-frames for scene understanding tasks. In this work, we
present two semi-supervised approaches that automate this process of manual
frame sifting in video streams by automatically classifying scenes for content
and filtering frames for fine-tuning scene understanding tasks. The first
rule-based method starts from a pre-trained object detector and it assigns
scene type, uncertainty and lighting categories to each frame based on
probability distributions of foreground objects. Next, frames with the highest
uncertainty and structural dissimilarity are isolated as key-frames. The second
method relies on the simCLR model for frame encoding followed by
label-spreading from 20% of frame samples to label the remaining frames for
scene and lighting categories. Also, clustering the video frames in the encoded
feature space further isolates key-frames at cluster boundaries. The proposed
methods achieve 64-93% accuracy for automated scene categorization for outdoor
image videos from public domain datasets of JAAD and KITTI. Also, less than 10%
of all input frames can be filtered as key-frames that can then be sent for
annotation and fine tuning of machine vision algorithms. Thus, the proposed
framework can be scaled to additional video data streams for automated training
of perception-driven systems with minimal training images.
