  We study the role of the representation of state-action value functions in
regret minimization in finite-horizon Markov Decision Processes (MDPs) with
linear structure. We first derive a necessary condition on the representation,
called universally spanning optimal features (UNISOFT), to achieve constant
regret in any MDP with linear reward function. This result encompasses the
well-known settings of low-rank MDPs and, more generally, zero inherent Bellman
error (also known as the Bellman closure assumption). We then demonstrate that
this condition is also sufficient for these classes of problems by deriving a
constant regret bound for two optimistic algorithms (LSVI-UCB and ELEANOR).
Finally, we propose an algorithm for representation selection and we prove that
it achieves constant regret when one of the given representations, or a
suitable combination of them, satisfies the UNISOFT condition.
