  Lottery Ticket Hypothesis (LTH) raises keen attention to identifying sparse
trainable subnetworks, or winning tickets, which can be trained in isolation to
achieve similar or even better performance compared to the full models. Despite
many efforts being made, the most effective method to identify such winning
tickets is still Iterative Magnitude-based Pruning (IMP), which is
computationally expensive and has to be run thoroughly for every different
network. A natural question that comes in is: can we "transform" the winning
ticket found in one network to another with a different architecture, yielding
a winning ticket for the latter at the beginning, without re-doing the
expensive IMP? Answering this question is not only practically relevant for
efficient "once-for-all" winning ticket finding, but also theoretically
appealing for uncovering inherently scalable sparse patterns in networks. We
conduct extensive experiments on CIFAR-10 and ImageNet, and propose a variety
of strategies to tweak the winning tickets found from different networks of the
same model family (e.g., ResNets). Based on these results, we articulate the
Elastic Lottery Ticket Hypothesis (E-LTH): by mindfully replicating (or
dropping) and re-ordering layers for one network, its corresponding winning
ticket could be stretched (or squeezed) into a subnetwork for another deeper
(or shallower) network from the same family, whose performance is nearly the
same competitive as the latter's winning ticket directly found by IMP. We have
also extensively compared E-LTH with pruning-at-initialization and dynamic
sparse training methods, as well as discussed the generalizability of E-LTH to
different model families, layer types, and across datasets. Code is available
at https://github.com/VITA-Group/ElasticLTH.
