  We discuss the notorious problem of order selection in hidden Markov models,
i.e. of selecting an adequate number of states, highlighting typical pitfalls
and practical challenges arising when analyzing real data. Extensive
simulations are used to demonstrate the reasons that render order selection
particularly challenging in practice despite the conceptual simplicity of the
task. In particular, we demonstrate why well-established formal procedures for
model selection, such as those based on standard information criteria, tend to
favor models with numbers of states that are undesirably large in situations
where states shall be meaningful entities. We also offer a pragmatic
step-by-step approach together with comprehensive advice for how practitioners
can implement order selection. Our proposed strategy is illustrated with a
real-data case study on muskox movement.
