  Recurrent neural networks (RNNs) with rich feature vectors of past values can
provide accurate point forecasts for series that exhibit complex serial
dependence. We propose two approaches to constructing deep time series
probabilistic models based on a variant of RNN called an echo state network
(ESN). The first is where the output layer of the ESN has stochastic
disturbances and a shrinkage prior for additional regularization. The second
approach employs the implicit copula of an ESN with Gaussian disturbances,
which is a deep copula process on the feature space. Combining this copula with
a non-parametrically estimated marginal distribution produces a deep
distributional time series model. The resulting probabilistic forecasts are
deep functions of the feature vector and also marginally calibrated. In both
approaches, Bayesian Markov chain Monte Carlo methods are used to estimate the
models and compute forecasts. The proposed models are suitable for the complex
task of forecasting intraday electricity prices. Using data from the Australian
National Electricity Market, we show that our deep time series models provide
accurate short term probabilistic price forecasts, with the copula model
dominating. Moreover, the models provide a flexible framework for incorporating
probabilistic forecasts of electricity demand as additional features, which
increases upper tail forecast accuracy from the copula model significantly.
