  We present a supervised learning to rank algorithm that effectively orders
images by exploiting the structure in image sequences. Most often in the
supervised learning to rank literature, ranking is approached either by
analyzing pairs of images or by optimizing a list-wise surrogate loss function
on full sequences. In this work we propose MidRank, which learns from
moderately sized sub-sequences instead. These sub-sequences contain useful
structural ranking information that leads to better learnability during
training and better generalization during testing. By exploiting sub-sequences,
the proposed MidRank improves ranking accuracy considerably on an extensive
array of image ranking applications and datasets.
