  It is well known that modeling and forecasting realized covariance matrices
of asset returns play a crucial role in the field of finance. The availability
of high frequency intraday data enables the modeling of the realized covariance
matrices directly. However, most of the models available in the literature
depend on strong structural assumptions and they often suffer from the curse of
dimensionality. We propose an end-to-end trainable model built on the CNN and
Convolutional LSTM (ConvLSTM) which does not require to make any distributional
or structural assumption but could handle high-dimensional realized covariance
matrices consistently. The proposed model focuses on local structures and
spatiotemporal correlations. It learns a nonlinear mapping that connect the
historical realized covariance matrices to the future one. Our empirical
studies on synthetic and real-world datasets demonstrate its excellent
forecasting ability compared with several advanced volatility models.
