  We study the long-term evolution of two or more stellar black holes (BHs) on
initially separated but unstable circular orbits around a supermassive BH
(SMBH). Such a close-packed orbital configuration can naturally arise from BH
migrations in the AGN disk. Dynamical instability of the orbits leads to
recurring close encounters between two BHs, during which the BH separation
$r_{\rm p}$ becomes less than the Hill radius $R_{\rm H}$. In the rare very
close encounters (with $r_{\rm p}$ several orders of magnitude less than
$R_{\rm H}$), a tight merging BH binary can form with the help of gravitational
wave emission. We use $N$-body simulations to study the time evolution of close
encounters of various degrees of "closeness" and the property of the resulting
binary BH mergers. For a typical "SMBH + 2 BHs" system, the averaged cumulative
number of close encounters (with $r_{\rm p} \lesssim R_{\rm H}$) scales
approximately as $\propto t^{0.5}$. The minimum encounter separation $r_{\rm
p}$ follows a linear cumulative distribution $P(<r_{\rm p}) \propto r_{\rm p}$
for $r_{\rm p} \ll R_{\rm H}$. From these, we obtain a semi-analytical
expression for the averaged rate of binary captures that lead to BH mergers.
Our results suggest that close-packed BHs in AGN disks may take a long time
($\gtrsim 10^7$ orbits around the SMBH) to experience a sufficiently close
encounter and form a bound binary, although this time can be shorter if the
initial BH orbits are highly aligned. The BH binary mergers produced in this
scenario always have high eccentricities when entering the LIGO band, and have
a broad distribution of orbital inclinations relative to the original AGN disk.
We also explore the effects of the gas disk and find that simple gas drags on
the BHs do not necessarily lead to an enhanced BH binary capture rate.
