  The CO-to-H$_2$ conversion factor ($X_\mathrm{CO}$) is known to correlate
with the metallicity ($Z$). The dust abundance, which is related to the
metallicity, is responsible for this correlation through dust shielding of
dissociating photons and H$_2$ formation on dust surfaces. In this paper, we
investigate how the relation between dust-to-gas ratio and metallicity
($\mathcal{D}$--$Z$ relation) affects the H$_2$ and CO abundances (and
$X_\mathrm{CO}$) of a `molecular' cloud. For the $\mathcal{D}$--$Z$ relation,
we adopt a dust evolution model developed in our previous work, which treats
the evolution of not only dust abundance but also grain sizes in a galaxy.
Shielding of dissociating photons and H$_2$ formation on dust are solved
consistently with the dust abundance and grain sizes. As a consequence, our
models {predict consistent metallicity dependence of $X_\mathrm{CO}$ with
observational data}. Among various processes driving dust evolution, grain
growth by accretion has the largest impact on the $X_\mathrm{CO}$--$Z$
relation. The other processes also have some impacts on the
$X_\mathrm{CO}$--$Z$ relation, but their effects are minor compared with the
scatter of the observational data at the metallicity range ($Z\gtrsim 0.1$
Z$_\odot$) where CO could be detected. We also find that dust condensation in
stellar ejecta has a dramatic impact on the H$_2$ abundance at low
metallicities ($\lesssim 0.1$ Z$_\odot$), relevant for damped Lyman $\alpha$
systems and nearby dwarf galaxies, and that the grain size dependence of H$_2$
formation rate is also important.
