  To efficiently perform inference with neural networks, the underlying tensor
programs require sufficient tuning efforts before being deployed into
production environments. Usually, enormous tensor program candidates need to be
sufficiently explored to find the one with the best performance. This is
necessary to make the neural network products meet the high demand of
real-world applications such as natural language processing, auto-driving, etc.
Auto-schedulers are being developed to avoid the need for human intervention.
However, due to the gigantic search space and lack of intelligent search
guidance, current auto-schedulers require hours to days of tuning time to find
the best-performing tensor program for the entire neural network.
  In this paper, we propose HARL, a reinforcement learning (RL) based
auto-scheduler specifically designed for efficient tensor program exploration.
HARL uses a hierarchical RL architecture in which learning-based decisions are
made at all different levels of search granularity. It also automatically
adjusts exploration configurations in real-time for faster performance
convergence. As a result, HARL improves the tensor operator performance by 22%
and the search speed by 4.3x compared to the state-of-the-art auto-scheduler.
Inference performance and search speed are also significantly improved on
end-to-end neural networks.
