  In this paper, we propose a novel image set representation and classification
method by maximizing the margin of image sets. The margin of an image set is
defined as the difference of the distance to its nearest image set from
different classes and the distance to its nearest image set of the same class.
By modeling the image sets by using both their image samples and their affine
hull models, and maximizing the margins of the images sets, the image set
representation parameter learning problem is formulated as an minimization
problem, which is further optimized by an expectation -maximization (EM)
strategy with accelerated proximal gradient (APG) optimization in an iterative
algorithm. To classify a given test image set, we assign it to the class which
could provide the largest margin. Experiments on two applications of
video-sequence-based face recognition demonstrate that the proposed method
significantly outperforms state-of-the-art image set classification methods in
terms of both effectiveness and efficiency.
