  A neural network with the widely-used ReLU activation has been shown to
partition the sample space into many convex polytopes for prediction. However,
the parameterized way a neural network and other machine learning models use to
partition the space has imperfections, \textit{e}.\textit{g}., the compromised
interpretability for complex models, the inflexibility in decision boundary
construction due to the generic character of the model, and the risk of being
trapped into shortcut solutions. In contrast, although the non-parameterized
models can adorably avoid or downplay these issues, they are usually
insufficiently powerful either due to over-simplification or the failure to
accommodate the manifold structures of data. In this context, we first propose
a new type of machine learning models referred to as Manifoldron that directly
derives decision boundaries from data and partitions the space via manifold
structure discovery. Then, we systematically analyze the key characteristics of
the Manifoldron such as manifold characterization capability and its link to
neural networks. The experimental results on 4 synthetic examples, 20 public
benchmark datasets, and 1 real-world application demonstrate that the proposed
Manifoldron performs competitively compared to the mainstream machine learning
models. We have shared our code in \url{https://github.com/wdayang/Manifoldron}
for free download and evaluation.
