  The Vehicle Routing Problem with Drones (VRPD) seeks to optimize the routing
paths for both trucks and drones, where the trucks are responsible for
delivering parcels to customer locations, and the drones are dispatched from
these trucks for parcel delivery, subsequently being retrieved by the trucks.
Given the NP-Hard complexity of VRPD, numerous heuristic approaches have been
introduced. However, improving solution quality and reducing computation time
remain significant challenges. In this paper, we conduct a comprehensive
examination of heuristic methods designed for solving VRPD, distilling and
standardizing them into core elements. We then develop a novel reinforcement
learning (RL) framework that is seamlessly integrated with the heuristic
solution components, establishing a set of universal principles for
incorporating the RL framework with heuristic strategies in an aim to improve
both the solution quality and computation speed. This integration has been
applied to a state-of-the-art heuristic solution for VRPD, showcasing the
substantial benefits of incorporating the RL framework. Our evaluation results
demonstrated that the heuristic solution incorporated with our RL framework not
only elevated the quality of solutions but also achieved rapid computation
speeds, especially when dealing with extensive customer locations.
