  This paper presents a novel reinforcement learning (RL) approach called
HAAM-RL (Heuristic Algorithm-based Action Masking Reinforcement Learning) for
optimizing the color batching re-sequencing problem in automobile painting
processes. The existing heuristic algorithms have limitations in adequately
reflecting real-world constraints and accurately predicting logistics
performance. Our methodology incorporates several key techniques including a
tailored Markov Decision Process (MDP) formulation, reward setting including
Potential-Based Reward Shaping, action masking using heuristic algorithms
(HAAM-RL), and an ensemble inference method that combines multiple RL models.
The RL agent is trained and evaluated using FlexSim, a commercial 3D simulation
software, integrated with our RL MLOps platform BakingSoDA. Experimental
results across 30 scenarios demonstrate that HAAM-RL with an ensemble inference
method achieves a 16.25% performance improvement over the conventional
heuristic algorithm, with stable and consistent results. The proposed approach
exhibits superior performance and generalization capability, indicating its
effectiveness in optimizing complex manufacturing processes. The study also
discusses future research directions, including alternative state
representations, incorporating model-based RL methods, and integrating
additional real-world constraints.
