  Artificial neural network (ANN) is a versatile tool to study the neural
representation in the ventral visual stream, and the knowledge in neuroscience
in return inspires ANN models to improve performance in the task. However, it
is still unclear how to merge these two directions into a unified framework. In
this study, we propose an integrated framework called Deep Autoencoder with
Neural Response (DAE-NR), which incorporates information from ANN and the
visual cortex to achieve better image reconstruction performance and higher
neural representation similarity between biological and artificial neurons. The
same visual stimuli (i.e., natural images) are input to both the mice brain and
DAE-NR. The encoder of DAE-NR jointly learns the dependencies from neural spike
encoding and image reconstruction. For the neural spike encoding task, the
features derived from a specific hidden layer of the encoder are transformed by
a mapping function to predict the ground-truth neural response under the
constraint of image reconstruction. Simultaneously, for the image
reconstruction task, the latent representation obtained by the encoder is
assigned to a decoder to restore the original image under the guidance of
neural information. In DAE-NR, the learning process of encoder, mapping
function and decoder are all implicitly constrained by these two tasks. Our
experiments demonstrate that if and only if with the joint learning, DAE-NRs
can improve the performance of visual image reconstruction and increase the
representation similarity between biological neurons and artificial neurons.
The DAE-NR offers a new perspective on the integration of computer vision and
neuroscience.
