  Recent technological advances coupled with large sample sets have uncovered
many factors underlying the genetic basis of traits and the predisposition to
complex disease, but much is left to discover. A common thread to most genetic
investigations is familial relationships. Close relatives can be identified
from family records, and more distant relatives can be inferred from large
panels of genetic markers. Unfortunately these empirical estimates can be
noisy, especially regarding distant relatives. We propose a new method for
denoising genetically - inferred relationship matrices by exploiting the
underlying structure due to hierarchical groupings of correlated individuals.
The approach, which we call Treelet Covariance Smoothing, employs a multiscale
decomposition of covariance matrices to improve estimates of pairwise
relationships. On both simulated and real data, we show that smoothing leads to
better estimates of the relatedness amongst distantly related individuals. We
illustrate our method with a large genome-wide association study and estimate
the "heritability" of body mass index quite accurately. Traditionally
heritability, defined as the fraction of the total trait variance attributable
to additive genetic effects, is estimated from samples of closely related
individuals using random effects models. We show that by using smoothed
relationship matrices we can estimate heritability using population-based
samples. Finally, while our methods have been developed for refining genetic
relationship matrices and improving estimates of heritability, they have much
broader potential application in statistics. Most notably, for
error-in-variables random effects models and settings that require
regularization of matrices with block or hierarchical structure.
