  Advancement in the field of machine learning is unavoidable, but something of
major concern is preserving the privacy of the users whose data is being used
for training these machine learning algorithms. Federated learning(FL) has
emerged as a promising paradigm for training machine learning models in a
distributed and privacy-preserving manner which enables one to collaborate and
train a global model without sharing local data. But starting this learning
process on each device in the right way, called ``model initialization" is
critical. The choice of initialization methods used for models plays a crucial
role in the performance, convergence speed, communication efficiency, privacy
guarantees of federated learning systems, etc. In this survey, we dive deeper
into a comprehensive study of various ways of model initialization techniques
in FL.Unlike other studies, our research meticulously compares, categorizes,
and delineates the merits and demerits of each technique, examining their
applicability across diverse FL scenarios. We highlight how factors like client
variability, data non-IIDness, model caliber, security considerations, and
network restrictions influence FL model outcomes and propose how strategic
initialization can address and potentially rectify many such challenges. The
motivation behind this survey is to highlight that the right start can help
overcome challenges like varying data quality, security issues, and network
problems. Our insights provide a foundational base for experts looking to fully
utilize FL, also while understanding the complexities of model initialization.
