  Reinforcement Learning (RL) is being increasingly used to learn and adapt
application behavior in many domains, including large-scale and safety critical
systems, as for example, autonomous driving. With the advent of plug-n-play RL
libraries, its applicability has further increased, enabling integration of RL
algorithms by users. We note, however, that the majority of such code is not
developed by RL engineers, which as a consequence, may lead to poor program
quality yielding bugs, suboptimal performance, maintainability, and evolution
problems for RL-based projects. In this paper we begin the exploration of this
hypothesis, specific to code utilizing RL, analyzing different projects found
in the wild, to assess their quality from a software engineering perspective.
Our study includes 24 popular RL-based Python projects, analyzed with standard
software engineering metrics. Our results, aligned with similar analyses for ML
code in general, show that popular and widely reused RL repositories contain
many code smells (3.95% of the code base on average), significantly affecting
the projects' maintainability. The most common code smells detected are long
method and long method chain, highlighting problems in the definition and
interaction of agents. Detected code smells suggest problems in responsibility
separation, and the appropriateness of current abstractions for the definition
of RL algorithms.
