A deployment can have five healthy replicas and still lack meaningful fault tolerance if all five run in one failure domain. Kubernetes topology spread constraints let a workload state how it wants matching Pods distributed across node labels such as zones or hostnames. The scheduler uses that intent together with resource availability, node affinity, taints, and other scheduling rules. Successful scheduling alone does not establish that the resulting distribution meets the workload’s availability objective.
The difficult cases occur at the boundaries: an unavailable zone, a new rollout whose Pods have changed labels, a constrained node pool, or a constraint that favors spreading but does not require it. To diagnose these cases, an administrator must understand eligible domains, maximum skew, selection rules, and the tradeoff between strict placement and service capacity.
Define the failure domain and selected Pod population
The topologyKey identifies a node label that defines a domain. topology.kubernetes.io/zone is commonly used for zone spreading, whereas kubernetes.io/hostname distinguishes nodes. A constraint counts matching Pods in eligible domains, not necessarily every Pod across the whole cluster. A useful design starts by stating whether the service needs resilience against one node, one zone, or a different infrastructure boundary.
An application might have multiple Deployments serving separate roles, such as API gateways and asynchronous workers. If their label selectors overlap unintentionally, the scheduler may count both sets when calculating distribution. Review the workload’s template labels, labelSelector, and any matchLabelKeys values together. The reference population should be exactly the replicas whose distribution supports the stated availability goal.
For a rollout, consider the pod-template-hash label often used to distinguish revisions. Without revision-aware counting where appropriate, old and new replicas may compete in one skew calculation, obscuring an imbalance among the new revision alone. Kubernetes supports matchLabelKeys with documented constraints and version-dependent behavior. Validate the deployed Kubernetes version and actual selector merging before relying on a copied manifest.
Calculate skew rather than eyeballing placements
maxSkew constrains the difference between Pod counts across considered domains under documented scheduling semantics. For a zone-level strict placement with three eligible zones, a distribution of four, one, and one may violate a desired maximum skew of one even though every zone contains at least one Pod. Count Pods selected by the configured selector and determine the eligible domain set before computing the discrepancy.
A common mistake is counting all nodes as eligible even when node selectors or taints exclude some of them. nodeAffinityPolicy and nodeTaintsPolicy determine how those boundaries contribute to spread calculations. A zone with only untolerated nodes may not belong to the candidate set under the chosen policies. Inspect both node labels and the incoming Pod’s tolerations to explain why the scheduler considers the available landscape differently from a human diagram.
minDomains matters when the design expects a minimum number of eligible fault domains. When fewer domains are available than the specified minimum, the global-minimum calculation follows special rules that can prevent placements under strict policy. Rehearse loss of a whole zone, not just normal steady state. Record whether delaying a new Pod is preferable to accepting a concentration of replicas when one zone is unavailable.
Distinguish DoNotSchedule from ScheduleAnyway
With whenUnsatisfiable: DoNotSchedule, the scheduler must reject a placement that violates the strict topology spread rule. This can protect distribution but can also keep Pods Pending during a capacity incident. The tradeoff is deliberate: an additional replica does not count toward resilience if placement would concentrate it beyond the approved limit. Capacity planning must consider this possibility rather than treating Pending as a scheduler failure by default.
ScheduleAnyway makes spreading a preference instead of a hard admission condition, allowing the scheduler to favor better distribution when other constraints permit it. This can improve availability during constrained periods, yet may leave too many replicas in one zone. A rollout may appear healthy while its failure-domain safety margin is reduced. Monitor the actual domain distribution alongside readiness and desired replica count.
A workload can combine multiple spread constraints, such as per-zone and per-node rules. Evaluate their intersection with resource requests, affinity, and anti-affinity. A configuration that seems individually reasonable may be impossible when two constraints require incompatible placements. Use Pod events and scheduler messages to separate an unsatisfied spread rule from CPU pressure, untolerated taints, unavailable volumes, or affinity exclusions.
Investigate Pending Pods systematically
Begin with kubectl describe pod and the scheduling events for a representative Pending replica. Identify the exact rejected node classes and whether spread constraints, resource limits, affinity, or storage topology caused the rejection. Then collect the node labels and the count of selected Pods per topology domain. A listing of Ready nodes is not enough because a node may be present but ineligible for a specific workload.
Compare the constraint with the actual Deployment template, not a stale local YAML file. A controller or Helm change can alter labels or inherited scheduling defaults. If the selector no longer matches new Pods, the scheduler may calculate misleading skew. If the topology label is missing on a subset of nodes, those nodes cannot be treated as interchangeable with correctly labeled zones without understanding the specific constraint behavior.
The Kubernetes scheduling problem is often described as insufficient capacity when it is really a mismatch between desired topology and the nodes available to satisfy it. Confirm whether adding a node in the correct zone would solve the problem, versus adding an arbitrary node elsewhere. A tested hypothesis saves resources and avoids weakening the topology rule merely to make a rollout green.
A practical readiness gate can compare the distribution of Ready replicas across zones after the rollout completes. If two zones contain only newly restarted Pods that have not passed readiness, a balanced count of scheduled objects can exaggerate real serving capacity. Record both scheduled and service-ready counts per domain, then test the result when one zone disappears. This couples placement with client availability and prevents operators from declaring resilience merely because every intended topology label appears in kubectl get pods -o wide output.
Measure placement during scaling and rolling updates
Autoscaling changes the denominator of a spread calculation. A HorizontalPodAutoscaler may request additional replicas in response to demand, while the cluster autoscaler supplies nodes only after capacity pressure is observed. Strict spreading can delay those replicas until a missing zone receives eligible capacity. Track the scheduling delay, node provisioning time, and load distribution together to distinguish policy enforcement from an autoscaling loop that cannot converge.
During a rolling update, old replicas, unavailable Pods, surge capacity, and revision labels can affect the transient distribution. Plan for peak replica count rather than sizing nodes only for steady state. If a Deployment asks for an extra surge Pod in every zone but has no spare capacity, rollout progress may depend on removals that other safety policies restrict. Test the proposed maxSurge, maxUnavailable, and topology rules in the same scenario.
A regional incident is the most revealing test. Remove or cordon the nodes representing one failure domain in a nonproduction environment, then observe which replicas remain, which replacements become Pending, and whether traffic still meets service-level objectives. Preserve the before-and-after distribution, failed scheduling events, and response-time data. The goal is not an aesthetically balanced chart; it is predictable application behavior under a realistic failure.
Align node labels with infrastructure reality
Topology labels are only as accurate as the infrastructure metadata behind them. A manually assigned zone-a label on a node actually hosted in another site can create false redundancy. Establish who sets standard topology labels, how changes are audited, and which automation validates their consistency with provider or datacenter metadata. A scheduler cannot infer an unmodeled shared power, network, or storage dependency from the labels alone.
A multi-zone architecture may still share a single storage controller or control-plane path. Replica spreading helps only for failures that are independent across the chosen domains. Pair placement analysis with application data replication, load-balancer health checks, and persistent-volume attachment limitations. An apparently well-distributed stateful workload can still be vulnerable if every replica requires one unavailable backend volume.
A topology-spread rule can leave replicas Pending if eligible nodes lack labels or capacity, even when healthy nodes exist elsewhere; CKA scheduling diagnosis uses actual placement constraints. Administrators should explain why an otherwise healthy node was skipped, how eligible domains affect skew, and which constraint is safe to change during an incident. Memorizing a YAML fragment without interpreting its node and Pod selection is insufficient for operational troubleshooting.
A deployment owner should also distinguish rapid recovery from deliberate geographical spreading. A service that stores sessions on local files cannot gain transparent failover merely by placing Pods in three zones; on a zone loss, clients may still lose state or reconnect through a single inaccessible load balancer. Include recovery tests for DNS resolution, service endpoint convergence, and session persistence when defining success. A topology rule is one input to availability engineering, and the acceptance evidence should show that an actual client request survives the selected failure mode.
Document the availability contract behind the rule
Write a policy statement that names the protected failure, required minimum zone coverage, permitted skew, and expected behavior when capacity is missing. A stateless customer-facing API may prefer serving an extra replica in a crowded zone over rejecting load, while a critical quorum-sensitive component may require stricter placement. Neither choice is universally correct. The workload owner’s recovery objectives should determine the constraint mode.
Monitor drift from the desired topology after scale-in, node maintenance, and rescheduling. The scheduler normally acts when placing Pods, not by continuously rebalancing every already-running Pod to achieve a perfect distribution. A stable cluster can therefore accumulate imbalance after failures or operational changes. Decide whether a controlled rollout or descheduling strategy is justified before taking action that might cause unnecessary disruption.
The acceptance check should include normal scaling, one node failure, one zone failure, a rollout at capacity, and a deliberately mislabeled node. For each, document the expected scheduling outcome and the service behavior that follows. Topology spread constraints then become part of a tested resilience design rather than decorative fields in a manifest.
