  We combine samples of nearby galaxies with Herschel photometry selected on
their dust, metal, HI, and stellar mass content, and compare these to chemical
evolution models in order to discriminate between different dust sources. In a
companion paper, we used a HI-selected sample of nearby galaxies to reveal a
sub-sample of very gas rich (gas fraction > 80 per cent) sources with dust
masses significantly below predictions from simple chemical evolution models,
and well below $M_d/M_*$ and $M_d/M_{gas}$ scaling relations seen in dust and
stellar-selected samples of local galaxies. We use a chemical evolution model
to explain these dust-poor, but gas-rich, sources as well as the observed star
formation rates (SFRs) and dust-to-gas ratios. We find that (i) a delayed star
formation history is required to model the observed SFRs; (ii) inflows and
outflows are required to model the observed metallicities at low gas fractions;
(iii) a reduced contribution of dust from supernovae (SNe) is needed to explain
the dust-poor sources with high gas fractions. These dust-poor, low stellar
mass galaxies require a typical core-collapse SN to produce 0.01 - 0.16
$M_{\odot}$ of dust. To match the observed dust masses at lower gas fractions,
significant grain growth is required to counteract the reduced contribution
from dust in SNe and dust destruction from SN shocks. These findings are
statistically robust, though due to intrinsic scatter it is not always possible
to find one single model that successfully describes all the data. We also show
that the dust-to-metals ratio decreases towards lower metallicity.
