  There exist hitherto unexplained fluctuations in the Cosmic Infrared
Background (CIB) on arcminute scales and larger. These have been shown to
cross-correlate with the Cosmic X-ray Background (CXB), leading several authors
to attribute the excess to a high-redshift growing black hole population. In
order to investigate potential sources that could explain this excess, in this
paper, we develop a new framework to compute the power spectrum of undetected
sources that do not have constant flux as a function of halo mass. In this
formulation, we combine a semi-analytic model for black hole growth and their
simulated spectra from hydrodynamical simulations. Revisiting the possible
contribution of a high-redshift black hole population, we find that too much
black hole growth is required at early epochs for z>6 accretion to explain
these fluctuations. Examining a population of accreting black holes at more
moderate redshifts, z\sim 2-3, we find that such models produce a poor fit to
the observed fluctuations while simultaneously overproducing the local black
hole mass density. Additionally, we rule out the hypothesis of a missing
Galactic foreground of warm dust that produces coherent fluctuations in the
X-ray via reflection of Galactic X-ray binary emission. Although we firmly rule
out accreting massive black holes as the source of these missing fluctuations,
additional studies will be required to determine their origin.
