  A high performance Layer-4 load balancer (LB) is one of the most important
components of a cloud service infrastructure. Such an LB uses network and
transport layer information for deciding how to distribute client requests
across a group of servers. A crucial requirement for a stateful LB is per
connection consistency (PCC); namely, that all the packets of the same
connection will be forwarded to the same server, as long as the server is
alive, even if the pool of servers or the assignment function changes. The
challenge is in designing a high throughput, low latency solution that is also
scalable. This paper proposes a highly scalable LB, called Prism, implemented
using a programmable switch ASIC. As far as we know, Prism is the first
reported LB that can process millions of connections per second and hundreds of
millions connections in total, while ensuring PCC. This is due to the fact that
Prism forwards all the packets in hardware, even during server pool changes,
while avoiding the need to maintain a hardware state per every active
connection. We implemented a prototype of the proposed architecture and showed
that Prism can scale to 100 million simultaneous connections, and can
accommodate more than one pool update per second.
