  Let $(G,d)$ be a metric space with the counting measure $\mu$ satisfying some
growth conditions. Let $\omega(x,y)=(1+d(x,y))^\delta$ for some
$0<\delta\leq1$. Let $0<p\leq1$. Let $\mathcal A_{p\omega}$ be the collection
of kernels $K$ on $G\times G$ satisfying $\max\{\sup_x\sum_y
|K(x,y)|^p\omega(x,y)^p, \sup_y\sum_x |K(x,y)|^p\omega(x,y)^p\}<\infty$. Each
$K \in \mathcal A_{p\omega}$ defines a bounded linear operator on $\ell^2(G)$.
If in addition, $\omega$ satisfies the weak growth condition, then we show that
$\mathcal A_{p\omega}$ is inverse closed in $B(\ell^2(G))$. We shall also
discuss inverse-closedness of $p$-Banach algebra of infinite matrices over
$\mathbb Z^d$ and the $p$-Banach algebra of weighted $p$-summable sequences
over $\mathbb Z^{2d}$ with the twisted convolution. In order to show these
results, we prove Hulanicki's lemma and Barnes' lemma for $p$-Banach algebras.
