  Off-policy learning from multistep returns is crucial for sample-efficient
reinforcement learning, but counteracting off-policy bias without exacerbating
variance is challenging. Classically, off-policy bias is corrected in a
per-decision manner: past temporal-difference errors are re-weighted by the
instantaneous Importance Sampling (IS) ratio after each action via eligibility
traces. Many off-policy algorithms rely on this mechanism, along with differing
protocols for cutting the IS ratios to combat the variance of the IS estimator.
Unfortunately, once a trace has been fully cut, the effect cannot be reversed.
This has led to the development of credit-assignment strategies that account
for multiple past experiences at a time. These trajectory-aware methods have
not been extensively analyzed, and their theoretical justification remains
uncertain. In this paper, we propose a multistep operator that can express both
per-decision and trajectory-aware methods. We prove convergence conditions for
our operator in the tabular setting, establishing the first guarantees for
several existing methods as well as many new ones. Finally, we introduce
Recency-Bounded Importance Sampling (RBIS), which leverages trajectory
awareness to perform robustly across $\lambda$-values in an off-policy control
task.
