  Linear mixed models (LMMs) are widely used for heritability estimation in
genome-wide association studies (GWAS). In standard approaches to heritability
estimation with LMMs, a genetic relationship matrix (GRM) must be specified. In
GWAS, the GRM is frequently a correlation matrix estimated from the study
population's genotypes, which corresponds to a normalized Euclidean distance
kernel. In this paper, we show that reliance on the Euclidean distance kernel
contributes to several unresolved modeling inconsistencies in heritability
estimation for GWAS. These inconsistencies can cause biased heritability
estimates in the presence of linkage disequilibrium (LD), depending on the
distribution of causal variants. We show that these biases can be resolved (at
least at the modeling level) if one adopts a Mahalanobis distance-based GRM for
LMM analysis. Additionally, we propose a new definition of partitioned
heritability -- the heritability attributable to a subset of genes or single
nucleotide polymorphisms (SNPs) -- using the Mahalanobis GRM, and show that it
inherits many of the nice consistency properties identified in our original
analysis. Partitioned heritability is a relatively new area for GWAS analysis,
where inconsistency issues related to LD have previously been known to be
especially pernicious.
