  Nonparametric learning is a fundamental concept in machine learning that aims
to capture complex patterns and relationships in data without making strong
assumptions about the underlying data distribution. Owing to simplicity and
familiarity, one of the most well-known algorithms under this paradigm is the
$k$-nearest neighbors ($k$-NN) algorithm. Driven by the usage of machine
learning in safety-critical applications, in this work, we shed new light on
the traditional nearest neighbors algorithm from the perspective of information
theory and propose a robust and interpretable framework for tasks such as
classification, regression, density estimation, and anomaly detection using a
single model. We can determine data point weights as well as feature
contributions by calculating the conditional entropy for adding a feature
without the need for explicit model training. This allows us to compute feature
contributions by providing detailed data point influence weights with perfect
attribution and can be used to query counterfactuals. Instead of using a
traditional distance measure which needs to be scaled and contextualized, we
use a novel formulation of $\textit{surprisal}$ (amount of information required
to explain the difference between the observed and expected result). Finally,
our work showcases the architecture's versatility by achieving state-of-the-art
results in classification and anomaly detection, while also attaining
competitive results for regression across a statistically significant number of
datasets.
