  In this note we introduce the transcendental part $t(X)$ of the motive of a
cubic fourfold $X$ and prove that it is isomorphic to the (twisted)
transcendental part $h_2^{tr}(F(X))$ in a suitable Chow-K\"unneth decomposition
for the motive of the Fano variety of lines $F(X)$. Then we prove that $t(X)$
is isomorphic to the Prym motive associated to the surface $S_l \subset F(X)$
of lines meeting a general line $l$. If $X$ is a special cubic fourfold in the
sense of Hodge theory, and $F(X)\cong S^{[2]}$, with $S$ a $K3$, then we show
that $t(X) \cong t_2(S)(1)$, where $t_2(S)$ is the transcendental motive.
Therefore the motive $h(X)$ is finite dimensional if and only if $S$ has a
finite dimensional motive. If $X$ is very general then $t(X)$ cannot be
isomorphic to the (twisted) transcendental motive of a surface. We relate the
existence of an isomorphism $t(X) \cong t_2(S)(1)$ to conjectures by Hassett
and Kuznetsov on the rationality of a special cubic fourfold. Finally we
consider the case of cubic fourfolds X admitting a fibration over
$\mathbf{P}^2$, whose fibers are either quadrics or del Pezzo surfaces of
degree 6, and prove the isomorphism $t_2(S)(1) \cong t(X)$, with $S$ a K3
surface.
