Congestion on a Cisco Catalyst network does not always appear as a saturated uplink. A voice call can break up while average interface utilization remains low because traffic arrives in short bursts, a narrow egress queue drops frames, or classification fails to recognize the application’s packets. Troubleshooting requires evidence about where frames enter, how they are marked, which hardware queue receives them, and what happens under competing traffic.
QoS mechanisms cannot create additional bandwidth. They allocate scarce forwarding resources according to policy. A successful diagnosis therefore separates a faulty marking decision, an overloaded path, buffer behavior, microbursts, and genuine capacity limits before adjusting the configuration.
Establish a repeatable congestion symptom
Start with the application and affected path. Document endpoint address, VLAN, DSCP marking, traffic direction, interface speed, timing, and whether the problem is loss, jitter, or delay. A conferencing session degraded for three seconds every minute may not correlate with a five-minute interface average. Time resolution is a critical part of proving the cause.
Collect interface counters, queue statistics, application telemetry, and relevant switch CPU or hardware resource health for the same incident window. A physical layer error, output queue drop, and ingress policy drop imply different repairs. It is unsafe to interpret every rising discard counter as QoS congestion without checking the hardware feature and software version that produced that counter.
Compare a known-good endpoint and a failing one with the same application. If one device enters a different access policy, it may lose its intended marking at the first hop. If both endpoints degrade only across one downstream link, the contested bottleneck is likely farther along the path. The comparison narrows where an engineer should inspect queue behavior.
Follow classification and marking across trust boundaries
Traffic can be classified by DSCP, class maps, ACLs, application recognition, or a combination supported by the platform. Check what the switch actually recognizes, not merely what the policy author intended. A packet marked EF by a phone may be rewritten at an access port because the configured trust state differs from the deployment design.
The DSCP model represents service intent, but a DSCP value is not a reservation. Downstream devices can map the same marking into different local queues. Trace the packet from access edge through distribution to WAN or service-provider handoff, inspecting policies where trust or rewriting changes.
Distinguish trusted voice devices from arbitrary endpoints. Trusting all markings from untrusted laptops allows one compromised or misconfigured application to claim priority service at the expense of legitimate traffic. A sound access design applies classification close to the traffic source while limiting who can influence priority mappings.
Imagine a 10-gigabit access uplink carrying bursts from storage transfers and time-sensitive meetings. A five-minute graph averaging 35% utilization seems healthy, yet one egress queue may briefly fill while many storage senders synchronize writes. Capturing millisecond-level telemetry or queue-specific hardware counters can reveal the burst coincidence. If the affected traffic is correctly classified but drops because the physical path is momentarily oversubscribed, changing DSCP alone will not solve the problem. Buffer allocation, scheduling, shaping, and capacity each address a different part of the failure.
Inspect egress queues rather than averages
Hardware queuing is commonly the decisive congestion point. Determine the configured queue sets, scheduler behavior, buffer allocation, drop thresholds, and how traffic classes map into them on that exact Catalyst platform. The same high-level policy language may compile into different hardware resources on different switch generations.
A queue can drop during a microburst even when the link’s average utilization looks comfortable. Inspect high-resolution counters or telemetry, calculate offered load over useful intervals, and correlate discard spikes with application latency. An interface carrying short bursts at line rate can exhaust a small queue long before a coarse utilization graph shows saturation.
The queuing fundamentals matter here: a packet needs to reach the intended class before priority scheduling can help it. Raising queue weight without confirming classification can improve the wrong traffic and leave the affected flow unchanged. Verify actual class and drop counts before and after the proposed repair.
Distinguish policing from shaping
Policing enforces an allowed rate by dropping or remarking traffic when its contract is exceeded. Shaping temporarily queues traffic and regulates transmission rate, adding delay while aiming to reduce downstream drops. A policing violation can produce an abrupt loss pattern even when the physical interface itself has capacity.
Inspect token-bucket parameters, class-specific match counts, conformed and exceeded byte statistics, and whether burst allowances fit the real application. A camera that briefly transmits a burst after motion detection behaves differently from a constant-bitrate voice stream. Treat traffic envelopes and burst size as measured properties rather than configuration guesses.
Shaping introduces its own trade-offs. If a WAN handoff uses an effective provider rate lower than the local physical port, shaping below the provider contract may protect priority flows from opaque upstream policing. But shaping at an unnecessarily low rate can create avoidable delay. Validate with the provider’s committed rate and with endpoint measurements.
A design review should name specific traffic examples for each service class. If a softphone’s voice stream, a screen-sharing feed, and a large video upload all arrive with the same priority marking, they may behave very differently under sustained contention. Capture each class’s actual bitrate and burst profile in a controlled trial. Then confirm the access switch distinguishes trusted telephone ports from workstation-generated DSCP and that the downstream queue map preserves the intended hierarchy. Measuring one successful call in isolation cannot validate class fairness during a busy business day.
Check priority class starvation and fairness
A strict priority class can protect latency-sensitive traffic during congestion, but assigning too much traffic to it defeats that purpose. If bulk flows or video streams are incorrectly marked as voice, they can compete inside the priority treatment. Confirm both the volume and legitimacy of packets entering that class.
Inspect the available fairness mechanisms for nonpriority traffic. A healthy QoS design should not allow one flow to monopolize all capacity merely because it arrived first. Distinguish weighted service, queue depth, drop thresholds, and class reservations; they address related but separate congestion behavior.
The objective is not zero drops under every overload. Some low-priority packet loss may be the deliberate outcome that prevents latency-sensitive sessions from collapsing. Evaluate whether the actual class outcomes match policy: voice remains intelligible, essential transactions finish, and backup or bulk transfers slow predictably instead of causing widespread application failure.
Test at the physical and overlay boundaries
A downstream LAG member may be oversubscribed even when aggregate bundle utilization is low. Hashing can place several large flows onto the same member, making a local congestion issue invisible in the port-channel total. Compare member-interface counters, hash inputs, and flow distribution before increasing aggregate capacity.
Encapsulation changes packet size and sometimes which header’s DSCP is trusted. On routed overlays, tunnels, and service-provider interconnects, verify whether the outer marking matches the intended service treatment. A QoS design that works before encapsulation may fail after a platform update or handoff configuration change.
For a meaningful load test, reproduce competing traffic with similar packet sizes and flow counts to production. Send only a bulk throughput benchmark and a real-time call and the queue results may differ from the busy-hour mix of short transactions, video, and numerous small flows. Test the failure mode actually observed.
Keep platform-specific hardware constraints visible
Catalyst product families differ in queuing ASICs, QoS support, buffer architecture, and syntax. Commands and counters from a Catalyst 9K deployment should not be generalized to older access switches without checking the platform guide. Some queue measurements require a more specific hardware-level command or are sampled differently from interface-level statistics.
When an engineer changes class maps or queue allocations, record the hardware resource impact and whether another policy feature shares the same TCAM or buffer resources. An apparently harmless policy expansion can fail compilation, change default class behavior, or consume scarce classification entries. Validate the active hardware programming rather than relying solely on the running configuration.
QoS can appear configured correctly while hardware marks or queues real traffic differently, so 350-401 ENCOR congestion diagnosis tests classification counters, drops, and available scheduling behavior. A technically correct configuration is still unsuccessful if real traffic is classified differently from the design or the hardware cannot implement the intended queue discipline.
A disciplined acceptance record lists before-and-after values for the exact queue, class, test load, interface speed, and application latency. If the voice class improves but the interactive data class becomes unusable, the network team should review the overall policy against agreed business priorities. Keep a rollback threshold tied to observable user impact rather than only a CLI configuration diff. This avoids retaining a technically successful change whose hidden cost is a different outage several hours later.
Verify the repair under the original conditions
State the hypothesis before modifying the policy. For example, “priority packets are being policed at the WAN edge due to a burst allowance that is too small.” The test should confirm the offending class counter rises with application loss, change only the relevant contract parameter, and reproduce the same application load to measure the new outcome.
After remediation, compare drops, queue occupancy, class counters, throughput, jitter, and end-user performance. A policy may reduce one queue’s drops by pushing congestion into another. Check adjacent links and lower-priority services to avoid declaring victory based on a single statistic.
Reliable QoS operations preserve causality: traffic identity, marking, hardware queue selection, offered load, and observed application result must line up. That evidence lets teams change a classifier, scheduler, policing rate, or capacity plan for the reason the network actually requires instead of making increasingly complex queue configurations by trial and error.
