  Feature embeddings are one of the most essential steps when training deep
learning based Click-Through Rate prediction models, which map high-dimensional
sparse features to dense embedding vectors. Classic human-crafted embedding
size selection methods are shown to be "sub-optimal" in terms of the trade-off
between memory usage and model capacity. The trending methods in Neural
Architecture Search (NAS) have demonstrated their efficiency to search for
embedding sizes. However, most existing NAS-based works suffer from expensive
computational costs, the curse of dimensionality of the search space, and the
discrepancy between continuous search space and discrete candidate space. Other
works that prune embeddings in an unstructured manner fail to reduce the
computational costs explicitly. In this paper, to address those limitations, we
propose a novel strategy that searches for the optimal mixed-dimension
embedding scheme by structurally pruning a super-net via Hard Auxiliary Mask.
Our method aims to directly search candidate models in the discrete space using
a simple and efficient gradient-based method. Furthermore, we introduce
orthogonal regularity on embedding tables to reduce correlations within
embedding columns and enhance representation capacity. Extensive experiments
demonstrate it can effectively remove redundant embedding dimensions without
great performance loss.
