  Hidden Markov models (HMMs) are flexible time series models in which the
distributions of the observations depend on unobserved serially correlated
states. The state-dependent distributions in HMMs are usually taken from some
class of parametrically specified distributions. The choice of this class can
be difficult, and an unfortunate choice can have serious consequences for
example on state estimates, on forecasts and generally on the resulting model
complexity and interpretation, in particular with respect to the number of
states. We develop a novel approach for estimating the state-dependent
distributions of an HMM in a nonparametric way, which is based on the idea of
representing the corresponding densities as linear combinations of a large
number of standardized B-spline basis functions, imposing a penalty term on
non-smoothness in order to maintain a good balance between goodness-of-fit and
smoothness. We illustrate the nonparametric modeling approach in a real data
application concerned with vertical speeds of a diving beaked whale,
demonstrating that compared to parametric counterparts it can lead to models
that are more parsimonious in terms of the number of states yet fit the data
equally well.
