  Human brains are known to be capable of speeding up visual recognition of
repeatedly presented objects through faster memory encoding and accessing
procedures on activated neurons. For the first time, we borrow and distill such
a capability into a semantic memory design, namely SMTM, to improve on-device
CNN inference. SMTM employs a hierarchical memory architecture to leverage the
long-tail distribution of objects of interest, and further incorporates several
novel techniques to put it into effects: (1) it encodes high-dimensional
feature maps into low-dimensional, semantic vectors for low-cost yet accurate
cache and lookup; (2) it uses a novel metric in determining the exit timing
considering different layers' inherent characteristics; (3) it adaptively
adjusts the cache size and semantic vectors to fit the scene dynamics. SMTM is
prototyped on commodity CNN engine and runs on both mobile CPU and GPU.
Extensive experiments on large-scale datasets and models show that SMTM can
significantly speed up the model inference over standard approach (up to 2X)
and prior cache designs (up to 1.5X), with acceptable accuracy loss.
