  The Torelli group of a manifold is the group of all diffeomorphisms which act
as the identity on the homology of the manifold. In this paper, we calculate
the invariant part (invariant under the action of the automorphisms of the
homology) of the cohomology of the classifying space of the Torelli group of
certain high-dimensional, highly connected manifolds, with rational
coefficients and in a certain range of degrees. This is based on
Galatius--Randal-Williams' work on the diffeomorphism groups of these
manifolds, Borel's classical results on arithmetic groups, and methods from
surgery theory and pseudoisotopy theory. As a corollary, we find that all
Miller--Morita--Mumford characteristic classes are nontrivial in the cohomology
of the classifying space of the Torelli group, except for those associated with
the Hirzebruch class, whose vanishing is forced by the family index theorem.
