  Off-policy reinforcement learning has many applications including: learning
from demonstration, learning multiple goal seeking policies in parallel, and
representing predictive knowledge. Recently there has been an proliferation of
new policy-evaluation algorithms that fill a longstanding algorithmic void in
reinforcement learning: combining robustness to off-policy sampling, function
approximation, linear complexity, and temporal difference (TD) updates. This
paper contains two main contributions. First, we derive two new hybrid TD
policy-evaluation algorithms, which fill a gap in this collection of
algorithms. Second, we perform an empirical comparison to elicit which of these
new linear TD methods should be preferred in different situations, and make
concrete suggestions about practical use.
