  Existing modular Reinforcement Learning (RL) architectures are generally
based on reusable components, also allowing for ``plug-and-play'' integration.
However, these modules are homogeneous in nature - in fact, they essentially
provide policies obtained via RL through the maximization of individual reward
functions. Consequently, such solutions still lack the ability to integrate and
process multiple types of information (i.e., heterogeneous knowledge
representations), such as rules, sub-goals, and skills from various sources. In
this paper, we discuss several practical examples of heterogeneous knowledge
and propose Augmented Modular Reinforcement Learning (AMRL) to address these
limitations. Our framework uses a selector to combine heterogeneous modules and
seamlessly incorporate different types of knowledge representations and
processing mechanisms. Our results demonstrate the performance and efficiency
improvements, also in terms of generalization, that can be achieved by
augmenting traditional modular RL with heterogeneous knowledge sources and
processing mechanisms.
