  Modern population genetics studies typically involve genome-wide genotyping
of individuals from a diverse network of ancestries. An important, unsolved
problem is how to formulate and estimate probabilistic models of observed
genotypes that allow for complex population structure. We formulate two general
probabilistic models, and we propose computationally efficient algorithms to
estimate them. First, we show how principal component analysis (PCA) can be
utilized to estimate a general model that includes the well-known
Pritchard-Stephens-Donnelly mixed-membership model as a special case. Noting
some drawbacks of this approach, we introduce a new "logistic factor analysis"
(LFA) framework that seeks to directly model the logit transformation of
probabilities underlying observed genotypes in terms of latent variables that
capture population structure. We demonstrate these advances on data from the
human genome diversity panel and 1000 genomes project, where we are able to
identify SNPs that are highly differentiated with respect to structure while
making minimal modeling assumptions.
