  What information from an act of sentence understanding is robustly
represented in the human brain? We investigate this question by comparing
sentence encoding models on a brain decoding task, where the sentence that an
experimental participant has seen must be predicted from the fMRI signal evoked
by the sentence. We take a pre-trained BERT architecture as a baseline sentence
encoding model and fine-tune it on a variety of natural language understanding
(NLU) tasks, asking which lead to improvements in brain-decoding performance.
  We find that none of the sentence encoding tasks tested yield significant
increases in brain decoding performance. Through further task ablations and
representational analyses, we find that tasks which produce syntax-light
representations yield significant improvements in brain decoding performance.
Our results constrain the space of NLU models that could best account for human
neural representations of language, but also suggest limits on the possibility
of decoding fine-grained syntactic information from fMRI human neuroimaging.
