  This paper discusses the potential of graphics processing units (GPUs) in
high-dimensional optimization problems. A single GPU card with hundreds of
arithmetic cores can be inserted in a personal computer and dramatically
accelerates many statistical algorithms. To exploit these devices fully,
optimization algorithms should reduce to multiple parallel tasks, each
accessing a limited amount of data. These criteria favor EM and MM algorithms
that separate parameters and data. To a lesser extent block relaxation and
coordinate descent and ascent also qualify. We demonstrate the utility of GPUs
in nonnegative matrix factorization, PET image reconstruction, and
multidimensional scaling. Speedups of 100 fold can easily be attained. Over the
next decade, GPUs will fundamentally alter the landscape of computational
statistics. It is time for more statisticians to get on-board.
