  We consider a two parameter family of unitarily invariant diffusion processes
on the general linear group $\mathbb{GL}_N$ of $N\times N$ invertible matrices,
that includes the standard Brownian motion as well as the usual unitary
Brownian motion as special cases. We prove that all such processes have
Gaussian fluctuations in high dimension with error of order $O(1/N)$; this is
in terms of the finite dimensional distributions of the process under a large
class of test functions known as trace polynomials. We give an explicit
characterization of the covariance of the Gaussian fluctuation field, which can
be described in terms of a fixed functional of three freely independent free
multiplicative Brownian motions. These results generalize earlier work of
L\'evy and Ma\"ida, and Diaconis and Evans, on unitary groups. Our approach is
geometric, rather than combinatorial.
