Top-of-rack (ToR) switching places one or more access switches in or near each equipment rack, with short connections to local servers and uplinks into the rest of the data-center fabric. It is a physical and operational design pattern, not a specific routing protocol or a promise that every packet stays inside the rack. A ToR design can simplify patching and modular expansion, but its value depends on uplink capacity, failure domains, switch density, optics costs and management overhead.
Designers should compare ToR with middle-of-row or end-of-row (EoR) switching using real server counts and traffic patterns. The right answer for a densely populated rack can be different from the right answer for an underfilled rack that shares infrastructure with adjacent cabinets. For the topology beyond the access switch, the site’s spine-and-leaf discussion helps frame how leaf switches connect into a larger fabric.
What a top-of-rack switch actually connects
A typical ToR switch terminates short server connections inside one rack and provides higher-speed uplinks to aggregation or spine devices. Some designs use a pair of ToR switches for resilience; others use one device with multiple upstream links and accept the corresponding single-switch failure domain. The form factor and position alone do not determine whether the device acts at Layer 2 or Layer 3, whether traffic passes through a firewall, or whether servers can reach each other without leaving the rack.
Physical location matters because copper, passive direct-attach copper, active optical cables, and pluggable optics have different reach and manageability characteristics. A ToR design can limit repeated copper moves to a rack and aggregate long horizontal runs over fiber. But many small switches create more operating systems, optics, control-plane endpoints and power supplies to monitor. Those costs must be included, not hidden behind the phrase “simpler cable management.”
Uplink oversubscription is a workload decision
Oversubscription compares the potential server-facing bandwidth with the available uplink capacity. Suppose a rack has 32 servers with 25 Gbps ports, for a combined nominal 800 Gbps of downlink capability, but only two 100 Gbps uplinks carrying 200 Gbps in aggregate. The nominal bandwidth ratio is 4:1. That is not necessarily four times too much oversubscription: few workloads drive all links simultaneously, and east-west flows within the same switching domain may avoid some uplinks. It is still a useful starting constraint to validate against actual application peaks and failure behavior.
Now assume one uplink fails. The remaining capacity may be only 100 Gbps, making the nominal ratio 8:1 if all 32 downlinks still operate. A design that performs acceptably in steady state can struggle during maintenance or an uplink failure. Calculate the worst relevant failure mode, not only the aggregated physical port count on a product sheet.
Server teaming and link aggregation require platform- and topology-aware configuration. Putting two physical links into two separate switches does not automatically create a single working LACP bundle; multi-chassis solutions and host policies have specific compatibility and failure-mode requirements. For a related check, see link aggregation assumptions and limits.
Top-of-rack versus end-of-row layouts
With ToR, each rack typically houses more switching equipment but uses shorter local server cables. With EoR, fewer centralized switches can serve multiple racks through longer structured copper or fiber runs. ToR can simplify rack-at-a-time rollout and moves, while EoR can make sense when equipment is sparse and centralized port utilization is more efficient. The comparison must include the cabling tray routes, power and cooling, port count, optic reach, rack space, maintenance ownership and expected growth.
For a large installation, labor and failure isolation may matter more than initial hardware list price. Replacing an individual ToR unit can be operationally straightforward when its rack and workloads are documented. Conversely, maintaining identical device configurations across dozens of racks needs tested automation, backups, image management and inventory. A centralized EoR switch may reduce device count but also concentrate more servers within a switch or cable-path failure domain.
Where east-west traffic goes
Not all traffic between neighboring servers stays in the rack, and not all traffic between separate racks traverses the same sequence of switches. VLAN membership, IP subnets, routing, distributed gateways, overlay encapsulation and security policy determine the forwarding path. A server connected to one ToR may send traffic through the fabric to reach a server attached to another ToR, even if the machines are physically adjacent in the data center.
Applications that communicate heavily between components can reveal oversubscription or latency limits in unexpected places. Measure observed flow distributions and congestion counters during representative workloads instead of designing solely for an abstract “east-west percentage.” Modern VXLAN/EVPN fabrics may route or bridge traffic in different locations depending on the control-plane and gateway design.
Availability depends on more than dual switches
Two ToR switches are useful only when the servers and upstream fabric can actually survive a single device failure. Verify power feeds, network adapter teaming behavior, multipath settings, upstream routing, topology loops, maintenance windows and detection times. A shared power distribution unit or a misconfigured LACP peer can defeat the apparent redundancy. Test the server’s real application path during a controlled switch maintenance event.
When selecting physical links, ensure that the server adapter and switch transceivers support the intended mode and distance. For high-speed server connectivity, direct-attach copper can be economical over short supported reaches; longer or denser runs may justify active optical cables or removable optics. The total-cost comparison should include spare optics, cable replacement labor and rack airflow, not merely per-meter media cost.
Manage the lifecycle of rack switches
A repeatable ToR design should define a standard port map, rack naming, VLAN or routed interface policy, uplink labeling, telemetry, credentials and backup procedure. Automation can apply those definitions consistently, but it also makes an incorrect template capable of affecting many racks at once. Staging, validation, rollback and protected out-of-band access remain important.
Port utilization dashboards alone are not sufficient. Watch link errors, queue congestion, power and thermal conditions, platform resource usage, uplink failure events and version-specific software issues. The site’s data-center telemetry troubleshooting guide gives a useful framework for distinguishing physical problems from routing or application issues.
A rack design checklist before procurement
- Count expected servers, their port speeds and the realistic growth envelope for each rack.
- Calculate both normal and single-failure uplink ratios using plausible workload traffic, not just nominal downlink rates.
- Confirm cabling, optics, power, cooling, rack-space and maintenance requirements.
- Draw server-to-ToR, ToR-to-fabric, and alternate failure paths, including any routed or overlay gateway placement.
- Test supported host teaming and multi-chassis arrangements; never assume that physically redundant cables guarantee logical redundancy.
- Document the monitoring and rollback process for a failed ToR switch and rehearse a representative maintenance event.
Estimate port capacity without hiding the failure case
A rack contains 24 active servers, each with dual 25-Gbps ports. The theoretical total of all server-facing links is 1.2 Tbps, but this does not mean applications routinely transmit at that aggregate rate, or that two adapters on one host necessarily double the usable rate for any single flow. For uplink planning, first decide which ports and flows can actually be active simultaneously and how the host teaming mode distributes traffic.
If two 100-Gbps uplinks serve the rack under normal operation, their usable aggregate capacity depends on the hashing or forwarding behavior and the traffic mix. A single flow may be limited to one member link, while many independent flows can distribute across them. With one uplink down, observe whether routing or teaming converges quickly enough for the applications’ timeout budgets. The switch’s port-speed total is a capacity ceiling, not a measured application service level.
Track a more realistic design metric: the highest expected traffic during backups, replica synchronization, migrations and major workload incidents, alongside the capacity remaining when a switch or link is lost. This gives the procurement team an auditable reason for choosing two versus four uplinks, 100 versus 400 Gbps optics, or a different rack placement strategy.
A small rack may justify a different architecture
Consider a row in which every second rack contains only a few servers. Buying two switches per rack for a high-availability ToR design could leave most ports and power capacity unused. A middle-of-row switch arrangement can share access-port density more efficiently, at the cost of longer cable routes and a larger affected population if the centralized device fails. Compare power, support, optics and replacement windows rather than using ToR as a universal rule.
Conversely, a site that replaces whole racks frequently may value repeatable in-rack cabling and pretested switch configurations over saving a few switch ports. With correctly engineered upstream redundancy, the rack can become a predictable deployment unit. The choice is ultimately about change frequency and failure domain as much as it is about bandwidth.
A practical starting point is to model one representative rack in detail and compare its full lifecycle cost under ToR and EoR. Then test how the topology behaves when an uplink fails, a switch is upgraded, or a server moves to another cabinet. Top-of-rack switching is compelling when short cabling and modular operations outweigh the added device management; it is not universally cheaper or faster merely because switches sit closer to servers.