  Behavioral cloning (BC) can recover a good policy from abundant expert data,
but may fail when expert data is insufficient. This paper considers a situation
where, besides the small amount of expert data, a supplementary dataset is
available, which can be collected cheaply from sub-optimal policies. Imitation
learning with a supplementary dataset is an emergent practical framework, but
its theoretical foundation remains under-developed. To advance understanding,
we first investigate a direct extension of BC, called NBCU, that learns from
the union of all available data. Our analysis shows that, although NBCU suffers
an imitation gap that is larger than BC in the worst case, there exist special
cases where NBCU performs better than or equally well as BC. This discovery
implies that noisy data can also be helpful if utilized elaborately. Therefore,
we further introduce a discriminator-based importance sampling technique to
re-weight the supplementary data, proposing the WBCU method. With our newly
developed landscape-based analysis, we prove that WBCU can outperform BC in
mild conditions. Empirical studies show that WBCU simultaneously achieves the
best performance on two challenging tasks where prior state-of-the-art methods
fail.
