  The next-generation CMB experiments are expected to constrain the
tensor-to-scalar ratio $r$ with high precision. Delensing is an important
process as the observed CMB $B$-mode polarization that contains the primordial
tensor perturbation signal is dominated by a much larger contribution due to
gravitational lensing. To do so successfully, it is useful to explore methods
for lensing reconstruction beyond the traditional quadratic estimator (QE)
(which will become suboptimal for the next-generation experiments), and the
maximum a posterior estimator (which is still currently under development). In
Caldeira et al. 2020, the authors showed that a neural network (NN) method
using the ResUNet architectrue performs better than the QE and slightly
suboptimally compared to the iterative estimator in terms of the lensing
reconstruction performance. In this work, we take one step further to evaluate
the delensing performance of these estimators on maps with primordial tensor
perturbations using a standard delensing pipeline, and show that the
\emph{delensing} performance of the NN estimator is optimal, agreeing with that
of a converged iterative estimator, when tested on a suite of simulations with
$r=0.01$ and $r=0.001$ for $12.7^{\circ} \times 12.7^{\circ}$ maps at a
CMB-Stage~4 like polarization noise level $1\,\mu \rm{K\,arcmin}$ and 1' beam.
We found that for the purpose of delensing, it is necessary to train and
evaluate the NN on a set of CMB maps with $l<l_{\rm{cut}}$ removed, in order to
avoid spurious correlations on the scales of interest for the final delensed
$B$-mode power spectrum $l<l_{\rm{cut}}$, similar to what was known previously
for the QE and the iterative estimator. We also present various NN training
techniques that can be extended for a simultaneous treatment of foregrounds and
more complex instrumental effects where the modeling is more uncertain.
