  Understanding how seed black holes grow into intermediate and supermassive
black holes (IMBHs and SMBHs, respectively) has important implications for the
duty-cycle of active galactic nuclei (AGN), galaxy evolution, and gravitational
wave astronomy. Most studies of the cosmological growth and merger history of
black holes have used semianalytic models and have concentrated on SMBH growth
in luminous galaxies. Using high resolution cosmological N-body simulations, we
track the assembly of black holes over a large range of final masses -- from
seed black holes to SMBHs -- over widely varying dynamical histories. We used
the dynamics of dark matter halos to track the evolution of seed black holes in
three different gas accretion scenarios. We have found that growth of
Sagittarius A* - size SMBH reaches its maximum mass M_{SMBH}~10^6Msun at z~6
through early gaseous accretion episodes, after which it stays at near constant
mass. At the same redshift, the duty-cycle of the host AGN ends, hence redshift
z=6 marks the transition from an AGN to a starburst galaxy which eventually
becomes the Milky Way. By tracking black hole growth as a function of time and
mass, we estimate that the IMBH merger rate reaches a maximum of R_{max}=55
yr^-1 at z=11. From IMBH merger rates we calculate N_{ULX}=7 per Milky Way type
galaxy per redshift in redshift range 2<z<6.
