  We present the E-UC$^3$RL algorithm for regret minimization in Stochastic
Contextual Markov Decision Processes (CMDPs). The algorithm operates under the
minimal assumptions of realizable function class and access to \emph{offline}
least squares and log loss regression oracles. Our algorithm is efficient
(assuming efficient offline regression oracles) and enjoys a regret guarantee
of $ \widetilde{O}(H^3 \sqrt{T |S| |A|d_{\mathrm{E}}(\mathcal{P}) \log
(|\mathcal{F}| |\mathcal{P}|/ \delta) )}) , $ with $T$ being the number of
episodes, $S$ the state space, $A$ the action space, $H$ the horizon,
$\mathcal{P}$ and $\mathcal{F}$ are finite function classes used to approximate
the context-dependent dynamics and rewards, respectively, and
$d_{\mathrm{E}}(\mathcal{P})$ is the Eluder dimension of $\mathcal{P}$ w.r.t
the Hellinger distance. To the best of our knowledge, our algorithm is the
first efficient and rate-optimal regret minimization algorithm for CMDPs that
operates under the general offline function approximation setting. In addition,
we extend the Eluder dimension to general bounded metrics which may be of
separate interest.
