  Connected decision boundaries are useful in several tasks like image
segmentation, clustering, alpha-shape or defining a region in nD-space.
However, the machine learning literature lacks methods for generating connected
decision boundaries using neural networks. Thresholding an invex function, a
generalization of a convex function, generates such decision boundaries. This
paper presents two methods for constructing invex functions using neural
networks. The first approach is based on constraining a neural network with
Gradient Clipped-Gradient Penality (GCGP), where we clip and penalise the
gradients. In contrast, the second one is based on the relationship of the
invex function to the composition of invertible and convex functions. We employ
connectedness as a basic interpretation method and create connected
region-based classifiers. We show that multiple connected set based classifiers
can approximate any classification function. In the experiments section, we use
our methods for classification tasks using an ensemble of 1-vs-all models as
well as using a single multiclass model on small-scale datasets. The
experiments show that connected set-based classifiers do not pose any
disadvantage over ordinary neural network classifiers, but rather, enhance
their interpretability. We also did an extensive study on the properties of
invex function and connected sets for interpretability and network morphism
with experiments on toy and real-world data sets. Our study suggests that invex
function is fundamental to understanding and applying locality and
connectedness of input space which is useful for various downstream tasks.
