  The study of the Cosmic Near-Infrared Background (CIB) light after
subtraction of resolved sources can push the limits of current observations and
infer the level of galaxy and black hole activity in the early universe.
However, disentangling the relative contribution from low- and high-redshift
sources is not trivial. Spatial fluctuations of the CIB exhibit a clustering
excess at angular scales $\sim 1^\circ$ whose origin has not been conclusively
identified. We explore the likelihood that this signal is dominated by emission
from galaxies and accreting black holes in the early Universe. We find that, if
the first small mass galaxies have a normal IMF, the light of their ageing
stars (fossils) integrated over cosmic time contributes a comparable amount to
the CIB as their pre-reionization progenitors. However, the measured
fluctuation signal is too large to be produced by galaxies at redshifts $z>8$
unless their star formation efficiencies are much larger than those inferred
from the observed Lyman-dropout population. In order to produce the observed
level of CIB fluctuation without violating constraints from galaxy counts and
the electron optical depth of the IGM, minihalos at $z>12$ must form stars with
efficiency $f_\star \gtrsim 0.1$ and, although a top-heavy IMF is preferred,
have a very low escape fraction of ionizing radiation, $f_{\rm esc}<0.01$. If
instead the CIB fluctuations are produced by high-$z$ black holes, one requires
vigorous accretion in the early universe reaching $\rho_{\rm acc} \gtrsim
10^5M_\odot{\rm Mpc^{-3}}$ by $z\simeq 10$. This growth must stop by $z \sim 6$
and be significantly obscured not to overproduce the soft cosmic X-ray
background (CXB) and its observed coherence with the CIB. We therefore find the
range of suitable possibilities at high-$z$ to be narrow, but could possibly be
widened by including additional physics and evolution at those epochs.
