  Deep Learning (DL) powered by Deep Neural Networks (DNNs) has revolutionized
various domains, yet understanding the intricacies of DNN decision-making and
learning processes remains a significant challenge. Recent investigations have
uncovered an interesting memorization phenomenon in which DNNs tend to memorize
specific details from examples rather than learning general patterns, affecting
model generalization, security, and privacy. This raises critical questions
about the nature of generalization in DNNs and their susceptibility to security
breaches. In this survey, we present a systematic framework to organize
memorization definitions based on the generalization and security/privacy
domains and summarize memorization evaluation methods at both the example and
model levels. Through a comprehensive literature review, we explore DNN
memorization behaviors and their impacts on security and privacy. We also
introduce privacy vulnerabilities caused by memorization and the phenomenon of
forgetting and explore its connection with memorization. Furthermore, we
spotlight various applications leveraging memorization and forgetting
mechanisms, including noisy label learning, privacy preservation, and model
enhancement. This survey offers the first-in-kind understanding of memorization
in DNNs, providing insights into its challenges and opportunities for enhancing
AI development while addressing critical ethical concerns.
