  Over the past decades, statisticians and machine-learning researchers have
developed literally thousands of new tools for the reduction of
high-dimensional data in order to identify the variables most responsible for a
particular trait. These tools have applications in a plethora of settings,
including data analysis in the fields of business, education, forensics, and
biology (such as microarray, proteomics, brain imaging), to name a few.
  In the present work, we focus our investigation on the limitations and
potential misuses of certain tools in the analysis of the benchmark colon
cancer data (2,000 variables; Alon et al., 1999) and the prostate cancer data
(6,033 variables; Efron, 2010, 2008). Our analysis demonstrates that models
that produce 100% accuracy measures often select different sets of genes and
cannot stand the scrutiny of parameter estimates and model stability.
  Furthermore, we created a host of simulation datasets and "artificial
diseases" to evaluate the reliability of commonly used statistical and data
mining tools. We found that certain widely used models can classify the data
with 100% accuracy without using any of the variables responsible for the
disease. With moderate sample size and suitable pre-screening, stochastic
gradient boosting will be shown to be a superior model for gene selection and
variable screening from high-dimensional datasets.
