  (Abridged) We use high resolution cosmological N-body simulations to study
the growth of intermediate to supermassive black holes from redshift 49 to
zero. We track the growth of black holes from the seeds of population III stars
to black holes in the range of 10^3 < M < 10^7 Msun -- not quasars, but rather
IMBH to low-mass SMBHs. These lower mass black holes are the primary observable
for the Laser Interferometer Space Antenna (LISA). The large-scale dynamics of
the black holes are followed accurately within the simulation down to scales of
1 kpc; thereafter, we follow the merger analytically from the last dynamical
friction phase to black hole coalescence. We find that the merger rate of these
black holes is R~25 per year between 8 < z < 11 and R = 10 per year at z=3.
Before the merger occurs the incoming IMBH may be observed with a next
generation of X-ray telescopes as a ULX source with a rate of about ~ 3 - 7 per
year for 1 < z < 5. We develop an analytic prescription that captures the most
important black hole growth mechanisms: galaxy merger-driven gas accretion and
black hole coalescence. Using this, we find that SMBH at the center of Milky
Way type galaxy was in place with most of its mass by z = 4.7, and most of the
growth was driven by gas accretion excited by major mergers. Hundreds of black
holes have failed to coalesce with the SMBH by z=0, some with masses of 10000
Msun, orbiting within the dark matter halo with luminosities up to ~ 30000
Lsun. These X-ray sources can easily be observed with Chandra at ~ 100 kpc.
