  Ensemble techniques are powerful approaches that combine several weak
learners to build a stronger one. As a meta-learning framework, ensemble
techniques can easily be applied to many machine learning methods. Inspired by
ensemble techniques, in this paper we propose an ensemble loss functions
applied to a simple regressor. We then propose a half-quadratic learning
algorithm in order to find the parameter of the regressor and the optimal
weights associated with each loss function. Moreover, we show that our proposed
loss function is robust in noisy environments. For a particular class of loss
functions, we show that our proposed ensemble loss function is Bayes consistent
and robust. Experimental evaluations on several datasets demonstrate that our
proposed ensemble loss function significantly improves the performance of a
simple regressor in comparison with state-of-the-art methods.
