  The ubiquity of microphone-enabled devices has lead to large amounts of
unlabelled audio data being produced at the edge. The integration of
self-supervised learning (SSL) and federated learning (FL) into one coherent
system can potentially offer data privacy guarantees while also advancing the
quality and robustness of speech representations. In this paper, we provide a
first-of-its-kind systematic study of the feasibility and complexities for
training speech SSL models under FL scenarios from the perspective of
algorithms, hardware, and systems limits. Despite the high potential of their
combination, we find existing system constraints and algorithmic behaviour make
SSL and FL systems nearly impossible to build today. Yet critically, our
results indicate specific performance bottlenecks and research opportunities
that would allow this situation to be reversed. While our analysis suggests
that, given existing trends in hardware, hybrid SSL and FL speech systems will
not be viable until 2027. We believe this study can act as a roadmap to
accelerate work towards reaching this milestone much earlier.
