  Convolutional neural network (CNN) driven by image recognition has been shown
to be able to explain cortical responses to static pictures at ventral-stream
areas. Here, we further showed that such CNN could reliably predict and decode
functional magnetic resonance imaging data from humans watching natural movies,
despite its lack of any mechanism to account for temporal dynamics or feedback
processing. Using separate data, encoding and decoding models were developed
and evaluated for describing the bi-directional relationships be-tween the CNN
and the brain. Through the encoding models, the CNN-predicted areas covered not
only the ventral stream, but also the dorsal stream, albe-it to a lesser
degree; single-voxel response was visualized as the specific pixel pattern that
drove the response, revealing the distinct representation of individual
cortical location; cortical activation was synthesized from natural images with
high-throughput to map category representation, con-trast, and selectivity.
Through the decoding models, fMRI signals were directly decoded to estimate the
feature representations in both visual and semantic spaces, for direct visual
reconstruction and seman-tic categorization, respectively. These results
cor-roborate, generalize, and extend previous findings, and highlight the value
of using deep learning, as an all-in-one model of the visual cortex, to
understand and decode natural vision.
