  The cosmic infrared background (CIB) provides a fundamental observational
constraint on the star-formation history of galaxies over cosmic history. We
estimate the contribution to the CIB from catalogued galaxies in the COSMOS
field by using a novel map fitting technique on the \textit{Herschel} SPIRE
maps. Prior galaxy positions are obtained using detections over a large range
in wavelengths in the $K_{\rm s}$--3\,GHz range. Our method simultaneously fits
the galaxies, the system foreground, and the leakage of flux from galaxies
located in masked areas and corrects for an "over-fitting" effect not
previously accounted for in stacking methods. We explore the contribution to
the CIB as a function of galaxy survey wavelength and depth. We find high
contributions to the CIB with the deep $r$ ($m_{\rm AB} \le 26.5$), $K_{\rm s}$
($m_{\rm AB} \le 24.0$) and 3.6\,$\mu$m ($m_{\rm AB} \le 25.5$) catalogues. We
combine these three deep catalogues and find a total CIB contributions of 10.5
$\pm$ 1.6, 6.7 $\pm$ 1.5 and 3.1 $\pm$ 0.7\,nWm$^{-2}$sr$^{-1}$ at 250, 350 and
500\,$\mu$m, respectively. Our CIB estimates are consistent with recent
phenomenological models, prior based SPIRE number counts and with (though more
precise than) the diffuse total measured by FIRAS. Our results raise the
interesting prospect that the CIB contribution at $\lambda \le 500\,\mu$m from
known galaxies has converged. Future large-area surveys like those with the
Large Synoptic Survey Telescope are therefore likely to resolve a substantial
fraction of the population responsible for the CIB at 250\,$\mu$m $\leq \lambda
\leq$ 500\,$\mu$m.
