  Deep convolutional neural networks (DCNNs) have become the most common
solution for automatic image annotation due to their non-parametric nature,
good performance, and their accessibility through libraries such as TensorFlow.
Among other fields, DCNNs are also a common approach to the annotation of large
astronomical image databases acquired by digital sky surveys. One of the main
downsides of DCNNs is the complex non-intuitive rules that make DCNNs act as a
``black box", providing annotations in a manner that is unclear to the user.
Therefore, the user is often not able to know what information is used by the
DCNNs for the classification. Here we demonstrate that the training of a DCNN
is sensitive to the context of the training data such as the location of the
objects in the sky. We show that for basic classification of elliptical and
spiral galaxies, the sky location of the galaxies used for training affects the
behavior of the algorithm, and leads to a small but consistent and
statistically significant bias. That bias exhibits itself in the form of
cosmological-scale anisotropy in the distribution of basic galaxy morphology.
Therefore, while DCNNs are powerful tools for annotating images of extended
sources, the construction of training sets for galaxy morphology should take
into consideration more aspects than the visual appearance of the object. In
any case, catalogs created with deep neural networks that exhibit signs of
cosmological anisotropy should be interpreted with the possibility of
consistent bias.
