  Due to their architecture and how they are trained, artificial neural
networks are typically not robust toward pruning, replacing, or shuffling
layers at test time. However, such properties would be desirable for different
applications, such as distributed neural network architectures where the order
of execution cannot be guaranteed or parts of the network can fail during
inference. In this work, we address these issues through a number of proposed
training approaches for vision transformers whose most important component is
randomizing the execution order of attention modules at training time. We show
that with our proposed approaches, vision transformers are indeed capable to
adapt to arbitrary layer execution orders at test time assuming one tolerates a
reduction (about 20\%) in accuracy at the same model size. We also find that
our trained models can be randomly merged with each other resulting in
functional ("Frankenstein") models without loss of performance compared to the
source models. Finally, we layer-prune our models at test time and find that
their performance declines gracefully.
