  This paper deals with model order selection in context of correlated noise.
More precisely, one considers sources embedded in an additive Complex
Elliptically Symmetric (CES) noise, with unknown parameters. The main
difficultly for estimating the model order lies into the noise correlation,
namely the scatter matrix of the corresponding CES distribution. In this work,
to tackle that problem, one adopts a two-step approach: first, we develop two
different methods based on a Toeplitz-structured model for estimating this
unknown scatter matrix and for whitening the correlated noise. Then, we apply
Maronna's $M$-estimators on the whitened signal to estimate the covariance
matrix of the "decorrelated" signal in order to estimate the model order. The
proposed methodology is based both on robust estimation theory as well as large
Random Matrix Theory, and original results are derived, proving the efficiency
of this methodology. Indeed, the main theoretical contribution is to derive
consistent robust estimators for the covariance matrix of the
signal-plus-correlated noise in a large dimensional regime and to propose
efficient methodology to estimate the rank of signal subspace. Finally, as
shown in the analysis, these results show a great improvement compared to the
state-of-the-art, on both simulated and real hyperspectral images.
