  Imperfect Information Games (IIGs) offer robust models for scenarios where
decision-makers face uncertainty or lack complete information. Counterfactual
Regret Minimization (CFR) has been one of the most successful family of
algorithms for tackling IIGs. The integration of skill-based strategy learning
with CFR could potentially mirror more human-like decision-making process and
enhance the learning performance for complex IIGs. It enables the learning of a
hierarchical strategy, wherein low-level components represent skills for
solving subgames and the high-level component manages the transition between
skills. In this paper, we introduce the first hierarchical version of Deep CFR
(HDCFR), an innovative method that boosts learning efficiency in tasks
involving extensively large state spaces and deep game trees. A notable
advantage of HDCFR over previous works is its ability to facilitate learning
with predefined (human) expertise and foster the acquisition of skills that can
be transferred to similar tasks. To achieve this, we initially construct our
algorithm on a tabular setting, encompassing hierarchical CFR updating rules
and a variance-reduced Monte Carlo sampling extension. Notably, we offer the
theoretical justifications, including the convergence rate of the proposed
updating rule, the unbiasedness of the Monte Carlo regret estimator, and ideal
criteria for effective variance reduction. Then, we employ neural networks as
function approximators and develop deep learning objectives to adapt our
proposed algorithms for large-scale tasks, while maintaining the theoretical
support.
