  In this paper, we investigate the power of {\it regularization}, a common
technique in reinforcement learning and optimization, in solving extensive-form
games (EFGs). We propose a series of new algorithms based on regularizing the
payoff functions of the game, and establish a set of convergence results that
strictly improve over the existing ones, with either weaker assumptions or
stronger convergence guarantees. In particular, we first show that dilated
optimistic mirror descent (DOMD), an efficient variant of OMD for solving EFGs,
with adaptive regularization can achieve a fast $\tilde O(1/T)$ last-iterate
convergence in terms of duality gap and distance to the set of Nash equilibrium
(NE) without uniqueness assumption of the NE. Second, we show that regularized
counterfactual regret minimization (\texttt{Reg-CFR}), with a variant of
optimistic mirror descent algorithm as regret-minimizer, can achieve
$O(1/T^{1/4})$ best-iterate, and $O(1/T^{3/4})$ average-iterate convergence
rate for finding NE in EFGs. Finally, we show that \texttt{Reg-CFR} can achieve
asymptotic last-iterate convergence, and optimal $O(1/T)$ average-iterate
convergence rate, for finding the NE of perturbed EFGs, which is useful for
finding approximate extensive-form perfect equilibria (EFPE). To the best of
our knowledge, they constitute the first last-iterate convergence results for
CFR-type algorithms, while matching the state-of-the-art average-iterate
convergence rate in finding NE for non-perturbed EFGs. We also provide
numerical results to corroborate the advantages of our algorithms.
