  Characterizing the sub-mm Galactic emission has become increasingly critical
especially in identifying and removing its polarized contribution from the one
emitted by the Cosmic Microwave Background (CMB). In this work, we present a
parametric foreground removal performed onto sub-patches identified in the
celestial sphere by means of spectral clustering. Our approach takes into
account efficiently both the geometrical affinity and the similarity induced by
the measurements and the accompanying errors. The optimal partition is then
used to parametrically separate the Galactic emission encoding thermal dust and
synchrotron from the CMB one applied on two nominal observations of forthcoming
experiments from the ground and from the space. Performing the parametric fit
singularly on each of the clustering derived regions results in an overall
improvement: both controlling the bias and the uncertainties in the CMB
$B-$mode recovered maps. We finally apply this technique using the map of the
number of clouds along the line of sight, $\mathcal{N}_c$, as estimated from HI
emission data and perform parametric fitting onto patches derived by clustering
on this map. We show that adopting the $\mathcal{N}_c$ map as a tracer for the
patches related to the thermal dust emission, results in reducing the $B-$mode
residuals post-component separation. The code is made publicly available.
