  We are interested in assessing the order of a finite-state Hidden Markov
Model (HMM) with the only two assumptions that the transition matrix of the
latent Markov chain has full rank and that the density functions of the
emission distributions are linearly independent. We introduce a new procedure
for estimating this order by investigating the rank of some well-chosen
integral operator which relies on the distribution of a pair of consecutive
observations. This method circumvents the usual limits of the spectral method
when it is used for estimating the order of an HMM: it avoids the choice of the
basis functions; it does not require any knowledge of an upper-bound on the
order of the HMM (for the spectral method, such an upper-bound is defined by
the number of basis functions); it permits to easily handle different types of
data (including continuous data, circular data or multivariate continuous data)
with a suitable choice of kernel. The method relies on the fact that the order
of the HMM can be identified from the distribution of a pair of consecutive
observations and that this order is equal to the rank of some integral operator
(\emph{i.e.} the number of its singular values that are non-zero). Since only
the empirical counter-part of the singular values of the operator can be
obtained, we propose a data-driven thresholding procedure. An upper-bound on
the probability of overestimating the order of the HMM is established.
Moreover, sufficient conditions on the bandwidth used for kernel density
estimation and on the threshold are stated to obtain the consistency of the
estimator of the order of the HMM. The procedure is easily implemented since
the values of all the tuning parameters are determined by the sample size.
