  We propose a new sequential classification model for astronomical objects
based on a recurrent convolutional neural network (RCNN) which uses sequences
of images as inputs. This approach avoids the computation of light curves or
difference images. This is the first time that sequences of images are used
directly for the classification of variable objects in astronomy. The second
contribution of this work is the image simulation process. We generate
synthetic image sequences that take into account the instrumental and observing
conditions, obtaining a realistic, set of movies for each astronomical object.
The simulated dataset is used to train our RCNN classifier. This approach
allows us to generate datasets to train and test our RCNN model for different
astronomical surveys and telescopes. We aim at building a simulated dataset
whose distribution is close enough to the real dataset, so that a fine tuning
could match the distributions between real and simulated dataset. To test the
RCNN classifier trained with the synthetic dataset, we used real-world data
from the High cadence Transient Survey (HiTS) obtaining an average recall of
85%, improved to 94% after performing fine tuning with 10 real samples per
class. We compare the results of our model with those of a light curve random
forest classifier. The proposed RCNN with fine tuning has a similar performance
on the HiTS dataset compared to the light curve classifier, trained on an
augmented training set with 10 real samples per class. The RCNN approach
presents several advantages in an alert stream classification scenario, such as
a reduction of the data pre-processing, faster online evaluation and easier
performance improvement using a few real data samples. These results encourage
us to use this method for alert brokers systems that will process alert streams
generated by new telescopes such as the Large Synoptic Survey Telescope.
