  Supervised classification of temporal sequences of astronomical images into
meaningful transient astrophysical phenomena has been considered a hard problem
because it requires the intervention of human experts. The classifier uses the
expert's knowledge to find heuristic features to process the images, for
instance, by performing image subtraction or by extracting sparse information
such as flux time series, also known as light curves. We present a successful
deep learning approach that learns directly from imaging data. Our method
models explicitly the spatio-temporal patterns with Deep Convolutional Neural
Networks and Gated Recurrent Units. We train these deep neural networks using
1.3 million real astronomical images from the Catalina Real-Time Transient
Survey to classify the sequences into five different types of astronomical
transient classes. The TAO-Net (for Transient Astronomical Objects Network)
architecture outperforms the results from random forest classification on light
curves by 10 percentage points as measured by the F1 score for each class; the
average F1 over classes goes from $45\%$ with random forest classification to
$55\%$ with TAO-Net. This achievement with TAO-Net opens the possibility to
develop new deep learning architectures for early transient detection. We make
available the training dataset and trained models of TAO-Net to allow for
future extensions of this work.
