  In this paper we study the setting where features are added or change
interpretation over time, which has applications in multiple domains such as
retail, manufacturing, finance. In particular, we propose an approach to
provably determine the time instant from which the new/changed features start
becoming relevant with respect to an output variable in an agnostic
(supervised) learning setting. We also suggest an efficient version of our
approach which has the same asymptotic performance. Moreover, our theory also
applies when we have more than one such change point. Independent post analysis
of a change point identified by our method for a large retailer revealed that
it corresponded in time with certain unflattering news stories about a brand
that resulted in the change in customer behavior. We also applied our method to
data from an advanced manufacturing plant identifying the time instant from
which downstream features became relevant. To the best of our knowledge this is
the first work that formally studies change point detection in a distribution
independent agnostic setting, where the change point is based on the changing
relationship between input and output.
