  General Value Functions (GVFs) (Sutton et al, 2011) are an established way to
represent predictive knowledge in reinforcement learning. Each GVF computes the
expected return for a given policy, based on a unique pseudo-reward. Multiple
GVFs can be estimated in parallel using off-policy learning from a single
stream of data, often sourced from a fixed behavior policy or pre-collected
dataset. This leaves an open question: how can behavior policy be chosen for
data-efficient GVF learning? To address this gap, we propose GVFExplorer, which
aims at learning a behavior policy that efficiently gathers data for evaluating
multiple GVFs in parallel. This behavior policy selects actions in proportion
to the total variance in the return across all GVFs, reducing the number of
environmental interactions. To enable accurate variance estimation, we use a
recently proposed temporal-difference-style variance estimator. We prove that
each behavior policy update reduces the mean squared error in the summed
predictions over all GVFs. We empirically demonstrate our method's performance
in both tabular representations and nonlinear function approximation.
