The data centre network: spine-leaf, VXLAN and EVPN
Lesson 1 of 5
1The data centre network: spine-leaf, VXLAN and EVPN
The big pictureIn an ordinary company most traffic goes from offices to the internet (north-south). In a data centre it is the reverse: servers talk to each other all the time, the web server asks the database, which copies to another, while storage syncs (east-west). The old tiered design with STP blocks half the links and makes the distance between two servers uneven. A senior builds a fabric where every link works.
What the picture shows: Any server to any server: leaf, then spine, then leaf. With ECMP flows spread over every spine at once, so no link is blocked and no road is longer than another.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Picture a hospital with floors for departments (leaves) and central lifts (spines). Every floor connects to every lift, so a patient goes from any department to any other in one lift ride, and if one lift breaks the others carry on. VXLAN makes two departments on different floors feel like one room, by putting the room's message in an envelope that rides the lifts. EVPN is the front desk that knows where everyone sits, so nobody shouts on every floor to find someone.
A two-layer Clos design: every leaf connects to every spine, with no leaf-to-leaf or spine-to-spine links. The underlay is fully routed, usually eBGP with one AS per leaf (RFC 7938) or OSPF/IS-IS, with ECMP across all spines and BFD. The overlay: VXLAN (RFC 7348) wraps an Ethernet frame in UDP on port 4789, with a 24-bit VNI (over 16 million networks against 4094 VLANs), adding 50 bytes, so set the underlay MTU to 9216. The wrapping point is the VTEP on each leaf. EVPN (RFC 7432 and RFC 8365) spreads MAC and IP addresses over BGP instead of flood-and-learn: type 2 for MAC/IP, type 3 for flood lists, type 5 for prefixes. Anycast gateway: the same gateway IP and MAC on every leaf. The oversubscription ratio = server-facing capacity ÷ spine-facing capacity.
Before You Start
🔁STP and EtherChannel
You know why STP blocks links.
🌐BGP
You know eBGP and AS_PATH.
The Core Idea
1From three tiers to spine-leaf
The old design (access, distribution, core) was built for north-south traffic, leans on STP and so blocks half the links, and the distance between two servers changes with where they sit. Spine-leaf fixes that: every leaf links to every spine, so the road between any two servers is always three hops (leaf-spine-leaf), and every link works at once with ECMP because the whole fabric is routed at layer 3, with no STP between switches.
2Sizing and the oversubscription ratio
The number of spines equals each leaf's uplink count, and the largest number of leaves equals each spine's port count. The oversubscription ratio says how much servers share the capacity upwards: 1:1 means none (for storage and AI), and 3:1 is common for general apps.
In the first case 1200G faces servers and 600G goes up: 2:1. AI training clusters are built with no oversubscription because every card talks to every card.
def fabric(leaf_down, down_speed, leaf_up, up_speed, spine_ports, servers_per_leaf):
down = leaf_down * down_speed
up = leaf_up * up_speed
leaves = spine_ports # each spine gives one port to every leaf
print(f"spines: {leaf_up} | max leaves: {leaves} | max servers: {leaves * servers_per_leaf:,}")
print(f"per leaf: {down} G down, {up} G up -> oversubscription {down / up:.1f}:1")
print("general apps:")
fabric(leaf_down=48, down_speed=25, leaf_up=6, up_speed=100, spine_ports=64, servers_per_leaf=48)
print("AI training cluster (no sharing):")
fabric(leaf_down=32, down_speed=400, leaf_up=32, up_speed=400, spine_ports=64, servers_per_leaf=4)3VXLAN: a layer 2 network over a routed one
Virtual machines move between hosts and want to stay on the same network with the same address. But the fabric is routed at layer 3. VXLAN squares the circle: the leaf (the VTEP) takes the whole Ethernet frame, puts it inside UDP/IP addressed to the other leaf, it crosses the spines as an ordinary packet, then is unwrapped there. Every virtual network has a VNI.
A frame's trip through VXLAN
VM-A on LEAF1 sends to VM-B on LEAF3, both in VNI 10010. Press a stage or walk in order.
The idea: the original frame never changes. The routed network carries an "envelope" around it, opened only at the two ends.
1 · The frame
VM-A sends an ordinary Ethernet frame to VM-B's MAC, as if both sat on one switch. It knows nothing of VXLAN.
2 · Lookup
LEAF1 looks up the destination MAC in its EVPN table: "VM-B is behind VTEP 10.0.0.13 in VNI 10010". It learned that from an EVPN type-2 route, not from flooding.
3 · Wrap
It adds a VXLAN header (8 bytes, holding the VNI), UDP to port 4789 (8), IP from 10.0.0.11 to 10.0.0.13 (20) and an outer Ethernet (14). 50 bytes in all, so a 1500 frame becomes 1550.
4 · Cross
The outer packet is routed like any IP packet. ECMP picks a spine by a hash over the UDP ports, so different flows spread over all spines.
5 · Unwrap
LEAF3 sees UDP 4789 addressed to it, strips the outer headers, reads VNI 10010 and hands the original frame, untouched, to VM-B.
Forget to raise the underlay MTU and small packets (ping) work while big ones fail: a deceptive fault.
OUTER = {"outer Ethernet": 14, "outer IP": 20, "UDP (dst 4789)": 8, "VXLAN (VNI)": 8}
overhead = sum(OUTER.values())
for part, size in OUTER.items():
print(f"{part:<16} {size:>3} bytes")
print("overhead :", overhead, "bytes")
print("underlay MTU needed for 1500-byte guests:", 1500 + overhead)
print("VNIs available :", f"{2 ** 24:,}", "vs VLANs:", 4094)4EVPN and the anycast gateway
VXLAN alone learns by flooding: it sends unknown frames to every VTEP. That does not scale. EVPN makes BGP carry the knowledge: when a VM appears behind a leaf, the leaf announces it with a type-2 route (MAC and IP), so everyone knows where it is before they need it, and each leaf answers ARP locally. The anycast gateway: every leaf holds the same gateway (IP and MAC), so a VM finds its gateway at the first hop wherever it moves.
| EVPN route type | Carries | What it is for |
|---|---|---|
| Type 2 | A device's MAC and IP | Where each VM is, and stopping ARP flooding |
| Type 3 | A VTEP wanting flood traffic for a VNI | Who receives the remaining broadcast |
| Type 5 | An IP prefix | Routing between networks and to the outside |
Worked example
The task: a company builds a data centre for 1500 servers on 25G ports, running general apps. You have leaves with 48×25G and 8×100G, and spines with 32×100G. Design the fabric.
The answer: each leaf serves 48 servers, so you need 1500 ÷ 48 ≈ 32 leaves, exactly the spine port count (32), leaving no room to grow; take 4 spines now with four uplinks per leaf (400G up against 1200G down = 3:1), and keep 4 uplinks free to add 4 spines later (then 1.5:1). Underlay eBGP with one AS per leaf, BFD and MTU 9216. Overlay VXLAN with EVPN, an anycast gateway on every leaf, and a pair of border leaves for the outside. To grow beyond 32 leaves you need bigger spines or a super-spine tier.
Myth vs Reality
| Myth | Reality |
|---|---|
| Spine-leaf means pricier gear. | It uses identical mid-size switches instead of one huge core, often cheaper for the same capacity and easier to grow. |
| VXLAN removes the need for routing. | VXLAN lives on top of a good routed network, and the underlay's quality sets the quality of everything. |
| BGP is only for the internet. | The largest data centres run eBGP inside the fabric itself for its simplicity and control. |
Where you meet it at work
☁️Cloud providers
Every cloud region is thousands of leaves in multi-tier Clos fabrics.
🧠AI clusters
Fabrics with no oversubscription at 400G and 800G for model training.
🏦A bank data centre
VXLAN EVPN separates production and test into separate VNIs on one fabric.
Test your understanding
How many hops between any two servers in a two-layer spine-leaf?
Symmetry is the point: always the same distance.
Which port does VXLAN use?
RFC 7348 sets UDP 4789.
Why raise the underlay MTU in a VXLAN fabric?
A 1500 frame becomes 1550 after wrapping.
A leaf has 48×25G to servers and 4×100G up. What is the oversubscription ratio?
1200G ÷ 400G = 3.
What does EVPN add to VXLAN?
Type-2 routes tell every leaf where each device is in advance.
Quick review cards
Check Your Understanding
Why is a leaf never linked straight to another leaf?
Because it breaks the symmetry and creates roads of different lengths; all traffic goes through the spines.
Ping works between two VMs in one VNI but file transfers hang. Most likely cause?
The underlay MTU is 1500, so big packets are dropped after wrapping.
How do you grow the fabric when server ports run out?
Add a new leaf and wire it to every spine, as long as the spines have free ports.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Done with this lesson?
Sign in to save your progress.