  As an essential resource management problem in network virtualization,
virtual network embedding (VNE) aims to allocate the finite resources of
physical network to sequentially arriving virtual network requests (VNRs) with
different resource demands. Since this is an NP-hard combinatorial optimization
problem, many efforts have been made to provide viable solutions. However, most
existing approaches have either ignored the admission control of VNRs, which
has a potential impact on long-term performances, or not fully exploited the
temporal and topological features of the physical network and VNRs. In this
paper, we propose a deep Hierarchical Reinforcement Learning approach to learn
a joint Admission Control and Resource Allocation policy for VNE, named
HRL-ACRA. Specifically, the whole VNE process is decomposed into an upper-level
policy for deciding whether to admit the arriving VNR or not and a lower-level
policy for allocating resources of the physical network to meet the requirement
of VNR through the HRL approach. Considering the proximal policy optimization
as the basic training algorithm, we also adopt the average reward method to
address the infinite horizon problem of the upper-level agent and design a
customized multi-objective intrinsic reward to alleviate the sparse reward
issue of the lower-level agent. Moreover, we develop a deep feature-aware graph
neural network to capture the features of VNR and physical network and exploit
a sequence-to-sequence model to generate embedding actions iteratively.
Finally, extensive experiments are conducted in various settings, and show that
HRL-ACRA outperforms state-of-the-art baselines in terms of both the acceptance
ratio and long-term average revenue. Our code is available at
\url{https://github.com/GeminiLight/hrl-acra}.
