  A general method to combine several estimators of the same quantity is
investigated. In the spirit of model and forecast averaging, the final
estimator is computed as a weighted average of the initial ones, where the
weights are constrained to sum to one. In this framework, the optimal weights,
minimizing the quadratic loss, are entirely determined by the mean square error
matrix of the vector of initial estimators. The averaging estimator is built
using an estimation of this matrix, which can be computed from the same
dataset. A non-asymptotic error bound on the averaging estimator is derived,
leading to asymptotic optimality under mild conditions on the estimated mean
square error matrix. This method is illustrated on standard statistical
problems in parametric and semi-parametric models where the averaging estimator
outperforms the initial estimators in most cases.
