  Recent studies in using deep learning to solve routing problems focus on
construction heuristics, the solutions of which are still far from optimality.
Improvement heuristics have great potential to narrow this gap by iteratively
refining a solution. However, classic improvement heuristics are all guided by
hand-crafted rules which may limit their performance. In this paper, we propose
a deep reinforcement learning framework to learn the improvement heuristics for
routing problems. We design a self-attention based deep architecture as the
policy network to guide the selection of next solution. We apply our method to
two important routing problems, i.e. travelling salesman problem (TSP) and
capacitated vehicle routing problem (CVRP). Experiments show that our method
outperforms state-of-the-art deep learning based approaches. The learned
policies are more effective than the traditional hand-crafted ones, and can be
further enhanced by simple diversifying strategies. Moreover, the policies
generalize well to different problem sizes, initial solutions and even
real-world dataset.
