  Graphical models are a powerful tool to estimate a high-dimensional inverse
covariance (precision) matrix, which has been applied for a portfolio
allocation problem. The assumption made by these models is a sparsity of the
precision matrix. However, when stock returns are driven by common factors,
such assumption does not hold. We address this limitation and develop a
framework, Factor Graphical Lasso (FGL), which integrates graphical models with
the factor structure in the context of portfolio allocation by decomposing a
precision matrix into low-rank and sparse components. Our theoretical results
and simulations show that FGL consistently estimates the portfolio weights and
risk exposure and also that FGL is robust to heavy-tailed distributions which
makes our method suitable for financial applications. FGL-based portfolios are
shown to exhibit superior performance over several prominent competitors
including equal-weighted and Index portfolios in the empirical application for
the S&P500 constituents.
