  The widespread adoption of large-scale machine learning models in recent
years highlights the need for distributed computing for efficiency and
scalability. This work introduces a novel distributed machine learning paradigm
-- \emph{consensus learning} -- which combines classical ensemble methods with
consensus protocols deployed in peer-to-peer systems. These algorithms consist
of two phases: first, participants develop their models and submit predictions
for any new data inputs; second, the individual predictions are used as inputs
for a communication phase, which is governed by a consensus protocol. Consensus
learning ensures user data privacy, while also inheriting the safety measures
against Byzantine attacks from the underlying consensus mechanism. We provide a
detailed theoretical analysis for a particular consensus protocol and compare
the performance of the consensus learning ensemble with centralised ensemble
learning algorithms. The discussion is supplemented by various numerical
simulations, which describe the robustness of the algorithms against Byzantine
participants.
