  Feed-forward, fully-connected Artificial Neural Networks (ANNs) or the
so-called Multi-Layer Perceptrons (MLPs) are well-known universal
approximators. However, their learning performance varies significantly
depending on the function or the solution space that they attempt to
approximate. This is mainly because of their homogenous configuration based
solely on the linear neuron model. Therefore, while they learn very well those
problems with a monotonous, relatively simple and linearly separable solution
space, they may entirely fail to do so when the solution space is highly
nonlinear and complex. Sharing the same linear neuron model with two additional
constraints (local connections and weight sharing), this is also true for the
conventional Convolutional Neural Networks (CNNs) and, it is, therefore, not
surprising that in many challenging problems only the deep CNNs with a massive
complexity and depth can achieve the required diversity and the learning
performance. In order to address this drawback and also to accomplish a more
generalized model over the convolutional neurons, this study proposes a novel
network model, called Operational Neural Networks (ONNs), which can be
heterogeneous and encapsulate neurons with any set of operators to boost
diversity and to learn highly complex and multi-modal functions or spaces with
minimal network complexity and training data. Finally, a novel training method
is formulated to back-propagate the error through the operational layers of
ONNs. Experimental results over highly challenging problems demonstrate the
superior learning capabilities of ONNs even with few neurons and hidden layers.
