  Let $X$ be a closed oriented connected topological manifold of dimension
$n\geq 5$. The structure group of $X$ is the abelian group of equivalence
classes of all pairs $(f, M)$ such that $M$ is a closed oriented manifold and
$f\colon M \to X$ is an orientation-preserving homotopy equivalence. The main
purpose of this article is to prove that a higher rho invariant defines a group
homomorphism from the topological structure group of $X$ to the $C^*$-algebraic
structure group of $X$. In fact, we introduce a higher rho invariant map on the
homology manifold structure group of a closed oriented connected
$\textit{topological}$ manifold, and prove its additivity. This higher rho
invariant map restricts to the higher rho invariant map on the topological
structure group. More generally, the same techniques developed in this paper
can be applied to define a higher rho invariant map on the homology manifold
structure group of a closed oriented connected $\textit{homology}$ manifold. As
an application, we use the additivity of the higher rho invariant map to study
non-rigidity of topological manifolds. More precisely, we give a lower bound
for the free rank of the $\textit{algebraically reduced}$ structure group of
$X$ by the number of torsion elements in $\pi_1 X$. Here the algebraic reduced
structure group of $X$ is the quotient of the topological structure group of
$X$ modulo a certain action of self-homotopy equivalences of $X$. We also
introduce a notion of homological higher rho invariant, which can be used to
detect many elements in the structure group of a closed oriented topological
manifold, even when the fundamental group of the manifold is torsion free. In
particular, we apply this homological higher rho invariant to show that the
structure group is not finitely generated for a class of manifolds.
