  Inference of population structure from genetic data plays an important role
in population and medical genetics studies. With the advancement and decreasing
cost of sequencing technology, the increasingly available whole genome
sequencing data provide much richer information about the underlying population
structure. The traditional method (Patterson, Price, and Reich, 2006)
originally developed for array-based genotype data for computing and selecting
top principal components that capture population structure may not perform well
on sequencing data for two reasons. First, the number of genetic variants p is
much larger than the sample size n in sequencing data such that the
sample-to-marker ratio n/p is nearly zero, violating the assumption of the
Tracy-Widom test used in their method. Second, their method might not be able
to handle the linkage disequilibrium well in sequencing data. To resolve those
two practical issues, we propose a new method called ERStruct to determine the
number of top informative principal components based on sequencing data. More
specifically, we propose to use the ratio of consecutive eigenvalues as a more
robust test statistic, and then we approximate its null distribution using
modern random matrix theory. Both simulation studies and applications to two
public data sets from the HapMap 3 and the 1000 Genomes Projects demonstrate
the empirical performance of our ERStruct method.
