  We analyze a simple prefiltered variation of the least squares estimator for
the problem of estimation with biased, semi-parametric noise, an error model
studied more broadly in causal statistics and active learning. We prove an
oracle inequality which demonstrates that this procedure provably mitigates the
variance introduced by long-term dependencies. We then demonstrate that
prefiltered least squares yields, to our knowledge, the first algorithm that
provably estimates the parameters of partially-observed linear systems that
attains rates which do not not incur a worst-case dependence on the rate at
which these dependencies decay. The algorithm is provably consistent even for
systems which satisfy the weaker marginal stability condition obeyed by many
classical models based on Newtonian mechanics. In this context, our
semi-parametric framework yields guarantees for both stochastic and worst-case
noise.
