  Natural Language Processing (NLP) inference is seeing increasing adoption by
mobile applications, where on-device inference is desirable for crucially
preserving user data privacy and avoiding network roundtrips. Yet, the
unprecedented size of an NLP model stresses both latency and memory, creating a
tension between the two key resources of a mobile device. To meet a target
latency, holding the whole model in memory launches execution as soon as
possible but increases one app's memory footprints by several times, limiting
its benefits to only a few inferences before being recycled by mobile memory
management. On the other hand, loading the model from storage on demand incurs
IO as long as a few seconds, far exceeding the delay range satisfying to a
user; pipelining layerwise model loading and execution does not hide IO either,
due to the high skewness between IO and computation delays.
  To this end, we propose Speedy Transformer Inference (STI). Built on the key
idea of maximizing IO/compute resource utilization on the most important parts
of a model, STI reconciles the latency v.s. memory tension via two novel
techniques. First, model sharding. STI manages model parameters as
independently tunable shards, and profiles their importance to accuracy.
Second, elastic pipeline planning with a preload buffer. STI instantiates an
IO/compute pipeline and uses a small buffer for preload shards to bootstrap
execution without stalling at early stages; it judiciously selects, tunes, and
assembles shards per their importance for resource-elastic execution,
maximizing inference accuracy.
  Atop two commodity SoCs, we build STI and evaluate it against a wide range of
NLP tasks, under a practical range of target latencies, and on both CPU and
GPU. We demonstrate that STI delivers high accuracies with 1-2 orders of
magnitude lower memory, outperforming competitive baselines.
