  Datasets drive vision progress, yet existing driving datasets are
impoverished in terms of visual content and supported tasks to study multitask
learning for autonomous driving. Researchers are usually constrained to study a
small set of problems on one dataset, while real-world computer vision
applications require performing tasks of various complexities. We construct
BDD100K, the largest driving video dataset with 100K videos and 10 tasks to
evaluate the exciting progress of image recognition algorithms on autonomous
driving. The dataset possesses geographic, environmental, and weather
diversity, which is useful for training models that are less likely to be
surprised by new conditions. Based on this diverse dataset, we build a
benchmark for heterogeneous multitask learning and study how to solve the tasks
together. Our experiments show that special training strategies are needed for
existing models to perform such heterogeneous tasks. BDD100K opens the door for
future studies in this important venue.
