  The ultra high-energy (UHE) diffuse gamma-ray background holds important
information on the propagation of cosmic rays in the Galaxy. However, its
measurements suffer from a contamination from unresolved sources whose
importance remains unclear. In this Letter, we propose a novel data-driven
estimate of the contribution of unresolved leptonic sources based on the
information present in the ATNF and the LHAASO catalogs. We find that in the
inner Galaxy at most $\sim60\%$ of the diffuse flux measured by LHAASO at
$10\,\rm{TeV}$ may originate from unresolved leptonic sources, and this
fraction drops with energy to less than $20\%$ at $100\,\rm{TeV}$. In the outer
Galaxy, the contribution of unresolved leptonic sources is always subdominant.
It is less than $\sim 20\%$ at $10\,\rm{TeV}$ and less than $\sim 8\%$ above
$\sim25\,\rm{TeV}$. We conclude that the UHE diffuse background should be
dominated by photons from a hadronic origin above a few tens of $\rm{TeV}$.
