  The problem of multi-domain learning of deep networks is considered. An
adaptive layer is induced per target domain and a novel procedure, denoted
covariance normalization (CovNorm), proposed to reduce its parameters. CovNorm
is a data driven method of fairly simple implementation, requiring two
principal component analyzes (PCA) and fine-tuning of a mini-adaptation layer.
Nevertheless, it is shown, both theoretically and experimentally, to have
several advantages over previous approaches, such as batch normalization or
geometric matrix approximations. Furthermore, CovNorm can be deployed both when
target datasets are available sequentially or simultaneously. Experiments show
that, in both cases, it has performance comparable to a fully fine-tuned
network, using as few as 0.13% of the corresponding parameters per target
domain.
