  Different disciplines pursue the aim to develop models which characterize
certain phenomena as accurately as possible. Climatology is a prime example,
where the temporal evolution of the climate is modeled. In order to compare and
improve different models, methodology for a fair model evaluation is
indispensable. As models and forecasts of a phenomenon are usually associated
with uncertainty, proper scoring rules, which are tools that account for this
kind of uncertainty, are an adequate choice for model evaluation. However,
under the presence of non-stationarity, such a model evaluation becomes
challenging, as the characteristics of the phenomenon of interest change. We
provide methodology for model evaluation in the context of non-stationary time
series. Our methodology assumes stationarity of the time series in shorter
moving time windows. These moving windows, which are selected based on a
changepoint analysis, are used to characterize the uncertainty of the
phenomenon/model for the corresponding time instances. This leads to the
concept of moving scores allowing for a temporal assessment of the model
performance. The merits of the proposed methodology are illustrated in a
simulation and a case study.
