  In today's context, deploying data-driven services like recommendation on
edge devices instead of cloud servers becomes increasingly attractive due to
privacy and network latency concerns. A common practice in building compact
on-device recommender systems is to compress their embeddings which are
normally the cause of excessive parameterization. However, despite the vast
variety of devices and their associated memory constraints, existing
memory-efficient recommender systems are only specialized for a fixed memory
budget in every design and training life cycle, where a new model has to be
retrained to obtain the optimal performance while adapting to a smaller/larger
memory budget. In this paper, we present a novel lightweight recommendation
paradigm that allows a well-trained recommender to be customized for arbitrary
device-specific memory constraints without retraining. The core idea is to
compose elastic embeddings for each item, where an elastic embedding is the
concatenation of a set of embedding blocks that are carefully chosen by an
automated search function. Correspondingly, we propose an innovative approach,
namely recommendation with universally learned elastic embeddings (RULE). To
ensure the expressiveness of all candidate embedding blocks, RULE enforces a
diversity-driven regularization when learning different embedding blocks. Then,
a performance estimator-based evolutionary search function is designed,
allowing for efficient specialization of elastic embeddings under any memory
constraint for on-device recommendation. Extensive experiments on real-world
datasets reveal the superior performance of RULE under tight memory budgets.
