  We study special cycles on a Shimura variety of orthogonal type over a
totally real field of degree $d$ associated with a quadratic form in $n+2$
variables whose signature is $(n,2)$ at $e$ real places and $(n+2,0)$ at the
remaining $d-e$ real places for $1\leq e <d$. Recently, these cycles were
constructed by Kudla and Rosu-Yott and they proved that the generating series
of special cycles in the cohomology group is a Hilbert-Siegel modular form of
half integral weight. We prove that, assuming the Beilinson-Bloch conjecture on
the injectivity of the higher Abel-Jacobi map, the generating series of special
cycles of codimension $er$ in the Chow group is a Hilbert-Siegel modular form
of genus $r$ and weight $1+n/2$. Our result is a generalization of
\textit{Kudla's modularity conjecture}, solved by Yuan-Zhang-Zhang
unconditionally when $e=1$.
