  Distributed actor languages are an effective means of constructing scalable
reliable systems, and the Erlang programming language has a well-established
and influential model. While Erlang model conceptually provides reliable
scalability, it has some inherent scalability limits and these force developers
to depart from the model at scale. This article establishes the scalability
limits of Erlang systems, and reports the work to improve the language
scalability.
  We systematically study the scalability limits of Erlang and address the
issues at the virtual machine (VM), language, and tool levels. More
specifically: (1) We have evolved the Erlang VM so that it can work effectively
in large scale single-host multicore and NUMA architectures. We have made
important architectural improvements to the Erlang/OTP. (2) We have designed
and implemented Scalable Distributed (SD) Erlang libraries to address
language-level scalability issues, and provided and validated a set of
semantics for the new language constructs. (3) To make large Erlang systems
easier to deploy, monitor, and debug we have developed and made open source
releases of five complementary tools, some specific to SD Erlang.
  Throughout the article we use two case studies to investigate the
capabilities of our new technologies and tools: a distributed hash table based
Orbit calculation and Ant Colony Optimisation (ACO). Chaos Monkey experiments
show that two versions of ACO survive random process failure and hence that SD
Erlang preserves the Erlang reliability model. Even for programs with no global
recovery data to maintain, SD Erlang partitions the network to reduce network
traffic and hence improves performance of the Orbit and ACO benchmarks above 80
hosts. ACO measurements show that maintaining global recovery data dramatically
limits scalability; however scalability is recovered by partitioning the
recovery data.
