  We introduce the ``soft Deep MaxPain'' (softDMP) algorithm, which integrates
the optimization of long-term policy entropy into reward-punishment
reinforcement learning objectives. Our motivation is to facilitate a smoother
variation of operators utilized in the updating of action values beyond
traditional ``max'' and ``min'' operators, where the goal is enhancing sample
efficiency and robustness. We also address two unresolved issues from the
previous Deep MaxPain method. Firstly, we investigate how the negated
(``flipped'') pain-seeking sub-policy, derived from the punishment action
value, collaborates with the ``min'' operator to effectively learn the
punishment module and how softDMP's smooth learning operator provides insights
into the ``flipping'' trick. Secondly, we tackle the challenge of data
collection for learning the punishment module to mitigate inconsistencies
arising from the involvement of the ``flipped'' sub-policy (pain-avoidance
sub-policy) in the unified behavior policy. We empirically explore the first
issue in two discrete Markov Decision Process (MDP) environments, elucidating
the crucial advancements of the DMP approach and the necessity for soft
treatments on the hard operators. For the second issue, we propose a
probabilistic classifier based on the ratio of the pain-seeking sub-policy to
the sum of the pain-seeking and goal-reaching sub-policies. This classifier
assigns roll-outs to separate replay buffers for updating reward and punishment
action-value functions, respectively. Our framework demonstrates superior
performance in Turtlebot 3's maze navigation tasks under the ROS Gazebo
simulation.
