  BCI algorithm development has long been hampered by two major issues: small
sample sets and a lack of reproducibility. We offer a solution to both of these
problems via a software suite that streamlines both the issues of finding and
preprocessing data in a reliable manner, as well as that of using a consistent
interface for machine learning methods. By building on recent advances in
software for signal analysis implemented in the MNE toolkit, and the unified
framework for machine learning offered by the scikit-learn project, we offer a
system that can improve BCI algorithm development. This system is fully
open-source under the BSD licence and available at
https://github.com/NeuroTechX/moabb. To validate our efforts, we analyze a set
of state-of-the-art decoding algorithms across 12 open access datasets, with
over 250 subjects. Our analysis confirms that different datasets can result in
very different results for identical processing pipelines, highlighting the
need for trustworthy algorithm benchmarking in the field of BCIs, and further
that many previously validated methods do not hold up when applied across
different datasets, which has wide-reaching implications for practical BCIs.
