  Edge applications generate a large influx of sensor data at massive scales.
Under many time-critical scenarios, these massive data streams must be
processed in a very short time to derive actionable intelligence. However,
traditional data processing systems (e.g., stream processing systems,
cloud-based IoT data processing systems) are not well-suited for these edge
applications. This is because they often do not scale well with a large number
of concurrent stream queries, do not support low-latency processing under
limited edge computing resources, and do not adapt to the level of
heterogeneity and dynamicity commonly present in edge computing environments.
These gaps suggest a need for a new edge stream processing system that advances
the stream processing paradigm to achieve efficiency and flexibility under the
constraints presented by edge computing architectures.
  We present AgileDart, an agile and scalable edge stream processing engine
that enables fast stream processing of a large number of concurrently running
low-latency edge applications' queries at scale in dynamic, heterogeneous edge
environments. The novelty of our work lies in a dynamic dataflow abstraction
that leverages distributed hash table (DHT) based peer-to-peer (P2P) overlay
networks to automatically place, chain, and scale stream operators to reduce
query latencies, adapt to workload variations, and recover from failures; and a
bandit-based path planning model that can re-plan the data shuffling paths to
adapt to unreliable and heterogeneous edge networks. We show analytically and
empirically that AgileDart outperforms Storm and EdgeWise on query latency and
significantly improves scalability and adaptability when processing a large
number of real-world edge stream applications' queries.
