  We consider model selection for classic Reinforcement Learning (RL)
environments -- Multi Armed Bandits (MABs) and Markov Decision Processes (MDPs)
-- under general function approximations. In the model selection framework, we
do not know the function classes, denoted by $\mathcal{F}$ and $\mathcal{M}$,
where the true models -- reward generating function for MABs and and transition
kernel for MDPs -- lie, respectively. Instead, we are given $M$ nested function
(hypothesis) classes such that true models are contained in at-least one such
class. In this paper, we propose and analyze efficient model selection
algorithms for MABs and MDPs, that \emph{adapt} to the smallest function class
(among the nested $M$ classes) containing the true underlying model. Under a
separability assumption on the nested hypothesis classes, we show that the
cumulative regret of our adaptive algorithms match to that of an oracle which
knows the correct function classes (i.e., $\cF$ and $\cM$) a priori.
Furthermore, for both the settings, we show that the cost of model selection is
an additive term in the regret having weak (logarithmic) dependence on the
learning horizon $T$.
