  In this paper, we propose a generic framework for devising an adaptive
approximation scheme for value function approximation in reinforcement
learning, which introduces multiscale approximation. The two basic ingredients
are multiresolution analysis as well as tree approximation. Starting from
simple refinable functions, multiresolution analysis enables us to construct a
wavelet system from which the basis functions are selected adaptively,
resulting in a tree structure. Furthermore, we present the convergence rate of
our multiscale approximation which does not depend on the regularity of basis
functions.
