  The characteristics of data like distribution and heterogeneity, become more
complex and counterintuitive as the dimensionality increases. This phenomenon
is known as curse of dimensionality, where common patterns and relationships
(e.g., internal and boundary pattern) that hold in low-dimensional space may be
invalid in higher-dimensional space. It leads to a decreasing performance for
the regression, classification or clustering models or algorithms. Curse of
dimensionality can be attributed to many causes. In this paper, we first
summarize five challenges associated with manipulating high-dimensional data,
and explains the potential causes for the failure of regression, classification
or clustering tasks. Subsequently, we delve into two major causes of the curse
of dimensionality, distance concentration and manifold effect, by performing
theoretical and empirical analyses. The results demonstrate that nearest
neighbor search (NNS) using three typical distance measurements, Minkowski
distance, Chebyshev distance, and cosine distance, becomes meaningless as the
dimensionality increases. Meanwhile, the data incorporates more redundant
features, and the variance contribution of principal component analysis (PCA)
is skewed towards a few dimensions. By interpreting the causes of the curse of
dimensionality, we can better understand the limitations of current models and
algorithms, and drive to improve the performance of data analysis and machine
learning tasks in high-dimensional space.
