  We describe a massively parallel implementation of the recently developed
discontinuous Galerkin density functional theory (DGDFT) [J. Comput. Phys.
2012, 231, 2140] method, for efficient large-scale Kohn-Sham DFT based
electronic structure calculations. The DGDFT method uses adaptive local basis
(ALB) functions generated on-the-fly during the self-consistent field (SCF)
iteration to represent the solution to the Kohn-Sham equations. The use of the
ALB set provides a systematic way to improve the accuracy of the approximation.
It minimizes the number of degrees of freedom required to represent the
solution to the Kohn-Sham problem for a desired level of accuracy. In
particular, DGDFT can reach the planewave accuracy with far fewer numbers of
degrees of freedom. By using the pole expansion and selected inversion (PEXSI)
technique to compute electron density, energy and atomic forces, we can make
the computational complexity of DGDFT scale at most quadratically with respect
to the number of electrons for both insulating and metallic systems. We show
that DGDFT can achieve 80% parallel efficiency on 128,000 high performance
computing cores when it is used to study the electronic structure of
two-dimensional (2D) phosphorene systems with 3,500-14,000 atoms. This high
parallel efficiency results from a two-level parallelization scheme that we
will describe in detail.
