  In this paper we present a dense ground truth dataset of nonrigidly deforming
real-world scenes. Our dataset contains both long and short video sequences,
and enables the quantitatively evaluation for RGB based tracking and
registration methods. To construct ground truth for the RGB sequences, we
simultaneously capture Near-Infrared (NIR) image sequences where dense markers
- visible only in NIR - represent ground truth positions. This allows for
comparison with automatically tracked RGB positions and the formation of error
metrics. Most previous datasets containing nonrigidly deforming sequences are
based on synthetic data. Our capture protocol enables us to acquire real-world
deforming objects with realistic photometric effects - such as blur and
illumination change - as well as occlusion and complex deformations. A public
evaluation website is constructed to allow for ranking of RGB image based
optical flow and other dense tracking algorithms, with various statistical
measures. Furthermore, we present an RGB-NIR multispectral optical flow model
allowing for energy optimization by adoptively combining featured information
from both the RGB and the complementary NIR channels. In our experiments we
evaluate eight existing RGB based optical flow methods on our new dataset. We
also evaluate our hybrid optical flow algorithm by comparing to two existing
multispectral approaches, as well as varying our input channels across RGB, NIR
and RGB-NIR.
