  State abstraction has been an essential tool for dramatically improving the
sample efficiency of reinforcement-learning algorithms. Indeed, by exposing and
accentuating various types of latent structure within the environment,
different classes of state abstraction have enabled improved theoretical
guarantees and empirical performance. When dealing with state abstractions that
capture structure in the value function, however, a standard assumption is that
the true abstraction has been supplied or unrealistically computed a priori,
leaving open the question of how to efficiently uncover such latent structure
while jointly seeking out optimal behavior. Taking inspiration from the bandit
literature, we propose that an agent seeking out latent task structure must
explicitly represent and maintain its uncertainty over that structure as part
of its overall uncertainty about the environment. We introduce a practical
algorithm for doing this using two posterior distributions over state
abstractions and abstract-state values. In empirically validating our approach,
we find that substantial performance gains lie in the multi-task setting where
tasks share a common, low-dimensional representation.
