  The model-based reinforcement learning paradigm, which uses planning
algorithms and neural network models, has recently achieved unprecedented
results in diverse applications, leading to what is now known as deep
reinforcement learning. These agents are quite complex and involve multiple
components, factors that can create challenges for research. In this work, we
propose a new modular software architecture suited for these types of agents,
and a set of building blocks that can be easily reused and assembled to
construct new model-based reinforcement learning agents. These building blocks
include planning algorithms, policies, and loss functions.
  We illustrate the use of this architecture by combining several of these
building blocks to implement and test agents that are optimized to three
different test environments: Cartpole, Minigrid, and Tictactoe. One particular
planning algorithm, made available in our implementation and not previously
used in reinforcement learning, which we called averaged minimax, achieved good
results in the three tested environments.
  Experiments performed with this architecture have shown that the best
combination of planning algorithm, policy, and loss function is heavily problem
dependent. This result provides evidence that the proposed architecture, which
is modular and reusable, is useful for reinforcement learning researchers who
want to study new environments and techniques.
