  Early exiting allows instances to exit at different layers according to the
estimation of difficulty. Previous works usually adopt heuristic metrics such
as the entropy of internal outputs to measure instance difficulty, which
suffers from generalization and threshold-tuning. In contrast, learning to
exit, or learning to predict instance difficulty is a more appealing way.
Though some effort has been devoted to employing such "learn-to-exit" modules,
it is still unknown whether and how well the instance difficulty can be
learned. As a response, we first conduct experiments on the learnability of
instance difficulty, which demonstrates that modern neural models perform
poorly on predicting instance difficulty. Based on this observation, we propose
a simple-yet-effective Hash-based Early Exiting approach (HashEE) that replaces
the learn-to-exit modules with hash functions to assign each token to a fixed
exiting layer. Different from previous methods, HashEE requires no internal
classifiers nor extra parameters, and therefore is more efficient. Experimental
results on classification, regression, and generation tasks demonstrate that
HashEE can achieve higher performance with fewer FLOPs and inference time
compared with previous state-of-the-art early exiting methods.
