  This paper introduces a solution to the problem of selecting dynamically
(online) the ``optimal'' p-norm to combat outliers in linear adaptive filtering
without any knowledge on the probability density function of the outliers. The
proposed online and data-driven framework is built on kernel-based
reinforcement learning (KBRL). To this end, novel Bellman mappings on
reproducing kernel Hilbert spaces (RKHSs) are introduced. These mappings do not
require any knowledge on transition probabilities of Markov decision processes,
and are nonexpansive with respect to the underlying Hilbertian norm. The
fixed-point sets of the proposed Bellman mappings are utilized to build an
approximate policy-iteration (API) framework for the problem at hand. To
address the ``curse of dimensionality'' in RKHSs, random Fourier features are
utilized to bound the computational complexity of the API. Numerical tests on
synthetic data for several outlier scenarios demonstrate the superior
performance of the proposed API framework over several non-RL and KBRL schemes.
