  Visual representation based on covariance matrix has demonstrates its
efficacy for image classification by characterising the pairwise correlation of
different channels in convolutional feature maps. However, pairwise correlation
will become misleading once there is another channel correlating with both
channels of interest, resulting in the ``confounding'' effect. For this case,
``partial correlation'' which removes the confounding effect shall be estimated
instead. Nevertheless, reliably estimating partial correlation requires to
solve a symmetric positive definite matrix optimisation, known as sparse
inverse covariance estimation (SICE). How to incorporate this process into CNN
remains an open issue. In this work, we formulate SICE as a novel structured
layer of CNN. To ensure end-to-end trainability, we develop an iterative method
to solve the above matrix optimisation during forward and backward propagation
steps. Our work obtains a partial correlation based deep visual representation
and mitigates the small sample problem often encountered by covariance matrix
estimation in CNN. Computationally, our model can be effectively trained with
GPU and works well with a large number of channels of advanced CNNs.
Experiments show the efficacy and superior classification performance of our
deep visual representation compared to covariance matrix based counterparts.
