  A reliable stereo algorithm is critical for many robotics applications. But
textureless and specular regions can easily cause failure by making feature
matching difficult. Understanding whether an algorithm is robust to these
hazardous regions is important. Although many stereo benchmarks have been
developed to evaluate performance, it is hard to quantify the effect of
hazardous regions in real images because the location and severity of these
regions are unknown. In this paper, we develop a synthetic image generation
tool enabling to control hazardous factors, such as making objects more
specular or transparent, to produce hazardous regions at different degrees. The
densely controlled sampling strategy in virtual worlds enables to effectively
stress test stereo algorithms by varying the types and degrees of the hazard.
We generate a large synthetic image dataset with automatically computed
hazardous regions and analyze algorithms on these regions. The observations
from synthetic images are further validated by annotating hazardous regions in
real-world datasets Middlebury and KITTI (which gives a sparse sampling of the
hazards). Our synthetic image generation tool is based on a game engine Unreal
Engine 4 and will be open-source along with the virtual scenes in our
experiments. Many publicly available realistic game contents can be used by our
tool to provide an enormous resource for development and evaluation of
algorithms.
