  We propose a general agent population learning system, and on this basis, we
propose lineage evolution reinforcement learning algorithm. Lineage evolution
reinforcement learning is a kind of derivative algorithm which accords with the
general agent population learning system. We take the agents in DQN and its
related variants as the basic agents in the population, and add the selection,
mutation and crossover modules in the genetic algorithm to the reinforcement
learning algorithm. In the process of agent evolution, we refer to the
characteristics of natural genetic behavior, add lineage factor to ensure the
retention of potential performance of agent, and comprehensively consider the
current performance and lineage value when evaluating the performance of agent.
Without changing the parameters of the original reinforcement learning
algorithm, lineage evolution reinforcement learning can optimize different
reinforcement learning algorithms. Our experiments show that the idea of
evolution with lineage improves the performance of original reinforcement
learning algorithm in some games in Atari 2600.
