Question: I am designing a network for my private cloud. Should I use Layer 2 switches or Layer 3 routers for my cloud network architecture?
Since the dawn of the Internet there has been an on-going debate over whether to use Layer 2 (Ethernet) or Layer 3 (IP) networking inside the data center. In the beginning, yes, we are talking about the 1990’s, for the most part networks were built on Layer 3 protocols. L2 switches were only used for internal LAN’s or very small installations. Does anyone really fondly remember NetBIOS or SPX/IPX? While Layer 2 switches were easy to deploy – one brand was appropriately named Black Box — they were impossibly slow and unreliable once you scaled past more than a couple hundred machines. Before the development of public IP address sparing protocols such as CIDR, DCHP and NAT, if you wanted to have an internet connection you had to assign each system a public IP address anyway.
Fast forward 20 years and many network protocols later, data centers are now typically architected to use L2 switches rather than L3 routers. The reasoning seems to be that Ethernet is faster because you don’t have the overhead of the IP hierarchy, and you don’t have to worry about reconfiguring IP addresses as systems get moved around. For me the second argument doesn’t hold up, since that is exactly what DHCP and DNS are designed for! The simplicity of Layer 2 protocols might work well in a data center with hundreds of physical machines, but cloud data centers have the additional burden of needing to keep track of all the virtual machine addresses and networks as well. It is not uncommon for one physical node to support 30-40 VM’s. Layer 2 switching protocols have improved mostly by adding “bolt-ons” such as VLAN’s, RBridges, or Cisco’s L2MP. I would argue that these are all proprietary patches to the fundamental scale and complexity problem. They still don’t have the built-in hierarchy and resiliency of a fully routed IP network.
A better paradigm is to think of cloud data centers as miniature (or in the case of Amazon, not so miniature) versions of the Internet. Thus applying the inherent scalability and flexibility of the IP address based Internet to a cloud data center network architecture makes perfect sense.
Cloud Network Architecture Basic Principles
- The cloud means, “Any server, any service, any time”
- Scalability through hierarchy
- Simplified network management
- Maximum network traffic flexibility
- Flattened traffic flow over the entire network mesh
- Minimize amount of state information maintained in network by keeping VM state (VM MACs and IPs) out of core network
- Reduce number of protocols to manage
Layer 2 Architecture Limitations
- Number of VLANs is limited to 4096
- Number of MACs stored in switch tables is limited
- Need to maintain a set of Layer 4 devices to handle traffic control
- MLAG (which is used for switch redundancy) is a proprietary solution that doesn’t scale beyond two devices and forces vendor lock-in
- Difficult to troubleshoot network without IP addresses and ICMP
- Configuring ARP is tricky on large L2 networks
- All network devices need to be aware of all MACs, even VM MAC’s, so there is constant churn in MAC tables and network state changes as VM’s are started or stopped
- Migrating MACs (VM migration) to different physical locations could be a problem if ARP table timeouts aren’t set properly
Layer 3 Architecture Advantages
- Provides the same level of resiliency and scalability as the Internet
- Easy to control traffic with routing metrics
- Can use BGP confederation for scalability, so core routers have state proportional to number of racks, not to number of servers or VMs
- It keeps per VM state (VM MACs and IPs) out of the network core, to reduce state churn. The only routing state changes are in case of a ToR failure or backbone link failure
- Uses ICMP to monitor and manage traffic
In future articles, there will be further dives into some of the ways that Layer 3 networks, virtual networks, and the most exciting new networking development, software only networks can be used to successfully address the needs of a cloud data center network.