  A discrete group $\G$ is called rigidly symmetric if the projective tensor
product between the convolution algebra $\ell^1(\G)$ and any $C^*$-algebra $\A$
is symmetric. We show that in each topologically graded $C^*$-algebra over a
rigidly symmetric group there is a $\ell^1$-type symmetric Banach $^*$-algebra,
which is inverse closed in the $C^*$-algebra. This includes new general
classes, as algebras admitting dual actions and partial crossed products.
Results including convolution dominated kernels, inverse closedness with
respect with ideals or weighted versions of the $\ell^1$-decay are included.
Various concrete examples are presented.
