  Federated Learning (FL) enables collaborative optimization of machine
learning models across decentralized data by aggregating model parameters. Our
approach extends this concept by aggregating "knowledge" derived from models,
instead of model parameters. We present a novel framework called CoDream, where
clients collaboratively optimize randomly initialized data using federated
optimization in the input data space, similar to how randomly initialized model
parameters are optimized in FL. Our key insight is that jointly optimizing this
data can effectively capture the properties of the global data distribution.
Sharing knowledge in data space offers numerous benefits: (1) model-agnostic
collaborative learning, i.e., different clients can have different model
architectures; (2) communication that is independent of the model size,
eliminating scalability concerns with model parameters; (3) compatibility with
secure aggregation, thus preserving the privacy benefits of federated learning;
(4) allowing of adaptive optimization of knowledge shared for personalized
learning. We empirically validate CoDream on standard FL tasks, demonstrating
competitive performance despite not sharing model parameters. Our code:
https://mitmedialab.github.io/codream.github.io/
