Troubleshooting a failing Kubernetes workload is fastest when operators classify the symptom before changing anything. The current CKA exam gives Troubleshooting the largest domain weight, and Kubernetes’ own debugging guidance starts by asking whether the failure belongs to the Pod, workload controller, Service, cluster, or another dependency. That sequence matters because a Pending Pod, CrashLoopBackOff, failing readiness probe, image-pull error, and unreachable Service point to different evidence.
The component map in Kubernetes cluster anatomy provides the first diagnostic boundary: the API server stores desired state; controllers create or reconcile objects; the scheduler binds Pods to nodes; kubelet and the container runtime start them; networking and storage attach dependencies; Services route to ready endpoints.
A disciplined habit is symptom → expected baseline → events/status → owning component → hypothesis → smallest safe remediation → validation. Commands are useful only when they answer one of those questions. A long command list without a fault model creates activity and little diagnosis.
First classify where the workload stopped
A Pod can fail before scheduling, after scheduling but before container start, during container startup, after becoming Running, or only when traffic reaches it.
Check phase, conditions, container states, restart count, node assignment, and recent events. That tells you which component most recently made a decision.
The distinction between Pods and containers is useful: the scheduler places a Pod, while container-level failures happen later on the selected node. Do not investigate scheduler policy for a container that is already bound and crashing.
Classification should also separate controller health from Pod health. A Deployment can be unable to create replacement Pods because its ReplicaSet is blocked by quota or admission policy, while the existing Pods continue to serve. Conversely, the controller can report desired replicas while every new container crashes. Inspect the owning controller’s conditions and recent events so the incident is not reduced to one Pod snapshot. Workload objects explain why Pods exist and whether replacement behavior is functioning as intended.
Pending usually means the scheduler or a prerequisite cannot satisfy the Pod
Pending can come from insufficient requested CPU/memory, taints without tolerations, affinity or topology constraints, unbound PVCs, unavailable nodes, or admission/preemption conditions.
Read scheduler events before changing resources. The event often lists how many nodes failed and the reasons.
Do not remove every constraint at once. Change one failed predicate or validate one missing prerequisite, then observe whether the scheduler advances to a new reason.
Scheduling evidence should be captured before cluster conditions change. Autoscaling, another workload finishing, or a node recovering can make a previously Pending Pod schedule successfully and erase the immediate symptom. The original events still explain the rejected nodes. If the condition is recurring, compare scheduler reasons over time and look for one constraint that appears across incidents. Repeated CPU shortage suggests capacity; repeated untolerated taint suggests node-pool policy; repeated PVC topology conflict suggests stateful placement design.
Image and container startup failures point to node-side execution
ImagePullBackOff and ErrImagePull suggest image name, registry access, credentials, network, or policy issues.
CreateContainerConfigError and related states can point to missing Secrets, ConfigMaps, or invalid runtime configuration.
Once the container starts, CrashLoopBackOff usually means the process exits repeatedly. Read previous-container logs, termination reason, exit code, command/args, mounted configuration, and dependency availability before increasing restart delays.
Startup failures should be read in order. Image pull establishes whether bytes can reach the node; container creation checks configuration and runtime setup; process execution reaches application startup. Jumping straight to application logs when the image never pulled wastes time. Likewise, rebuilding an image does not fix a missing Secret reference. Treat each state transition as a gate and stop at the first one that failed. Kubernetes surfaces those gates explicitly in container state and events if operators read them sequentially.
Readiness and liveness answer different questions
Readiness controls whether a Pod should receive traffic; liveness can trigger container restart when the process is considered unhealthy.
A bad readiness probe creates unavailable endpoints without necessarily restarting the container. A bad liveness probe can create a restart loop around an otherwise recoverable service.
Compare the probe’s target with real application behavior. If the endpoint depends on a slow external service, a tiny timeout may turn a dependency hiccup into a self-inflicted availability incident.
Probe failures should be compared with direct application tests from the same network context. A readiness probe using localhost or Pod IP can succeed while users fail through the Service, and a probe using a dependency endpoint can fail because the dependency is slow even though the local process is healthy. Decide what the probe is supposed to protect. Then test that exact condition manually. Probe configuration is part of release behavior and should be reviewed after application endpoints or startup timing change.
Events usually have higher value than a broad log sweep
Object events capture scheduler rejection, mount failure, image pull, failed probes, eviction, and controller actions.
Start with the failing Pod and related controller events, then inspect component logs when the event narrows the layer.
Collect events early during intermittent problems because retention is limited. A Pod can later recover and leave operators without the original scheduling or mount reason if evidence was not captured.
Controller and kubelet logs become valuable after object events narrow the layer. Scheduler logs can explain unusual placement decisions; kubelet logs can reveal node-local mount/runtime issues; container runtime and CNI logs can expose lower-level failure. Collecting every control-plane log first creates noise. Use the Kubernetes object status as an index into the component architecture: the object says which controller most recently attempted an action, and that component’s logs are then likely to contain higher-value detail.
Service symptoms need an inside-out path
An application can be healthy directly on the Pod IP and unreachable through the Service.
Verify Pods are serving, then Service selectors and EndpointSlices, then DNS and the Service dataplane, then external Ingress/Gateway or load balancer.
The general network-connectivity troubleshooting principle applies: prove each hop before changing the next layer. A failed external request does not automatically mean the Pod is broken.
Service debugging should also verify the application port contract. A Pod can listen on 8080, Service targetPort can point to 80, and EndpointSlice can still exist. Kubernetes objects are syntactically valid while packets reach the wrong port. Compare container listening sockets, Pod spec ports where documented, Service port/targetPort, and route/backend configuration. Port names matter when other resources reference them. Misaligned naming is especially easy to miss when manifests are copied between workloads with different container conventions.
Storage errors have their own sequence
Inspect PVC state, StorageClass, volume binding, scheduling topology, attach/mount events, CSI components, and node/kubelet evidence.
A Pod that cannot mount its volume is not fixed by restarting the Deployment repeatedly.
Preserve data safety while troubleshooting. Deleting PVCs or recreating storage as an experiment can turn an availability problem into irreversible data loss depending on reclaim policy.
Storage diagnosis should preserve the difference between provisioning and application data correctness. A volume can attach and mount successfully while the filesystem is read-only, full, corrupt, or contains stale data from the wrong claim. After mount succeeds, inspect filesystem/application evidence. Conversely, application errors about missing files should not cause immediate PVC deletion. State changes are higher risk than stateless restarts, so troubleshooting should prefer observation, snapshots/backups, and driver evidence before destructive cleanup.
False leads are often the most familiar subsystem
Network teams blame CNI, storage teams blame PVCs, application teams blame the image, and platform teams blame scheduling because those are the areas each team knows best.
Use evidence to force the fault into one layer before escalating.
Compare a healthy replica, node, namespace, or previous revision with the failing instance. Differences in status, labels, mounts, image digest, environment, or node conditions are often more revealing than reading every possible dashboard.
Comparing healthy and failing replicas is powerful only when the replicas are actually comparable. They can run different revisions, nodes, zones, injected sidecars, Secrets, or ConfigMap versions. Record those differences before assuming the healthy Pod represents the same release. A rollout can produce one good old replica and one failing new replica; comparing their logs without noticing the image digest explains little. Establish version equivalence first, then use differential diagnosis to isolate environment-specific versus release-specific causes.
Close the incident by proving recovery and removing the workaround
Re-run the original failing request, confirm the controller has the desired replicas, Pods stay healthy through the observation window, Services expose the expected endpoints, and metrics return to baseline.
Remove temporary tolerations, resource changes, disabled probes, elevated permissions, or manual edits used during diagnosis.
The broader Kubernetes foundations matter because Kubernetes continually reconciles desired and actual state. Troubleshooting is complete when the team understands which reconciliation or dependency failed, why the remediation corrected it, and whether the declarative source now matches the healthy runtime.
Validation should include recurrence triggers. If the incident appeared during node drain, deployment, autoscaling, secret rotation, or traffic spike, recreate that transition in a controlled way where possible. A Pod that stays healthy for five quiet minutes after remediation does not prove the scheduling or lifecycle failure is gone. The final test should resemble the condition that exposed the bug, and monitoring should remain focused through the next comparable event.
Troubleshooting practice improves when each incident produces one stronger diagnostic shortcut. If several failures were ultimately caused by stale Secret mounts, add a comparison or alert that makes that state visible sooner. If scheduler events repeatedly expire before investigation, improve event retention. The goal is not to build an enormous runbook for every error string; it is to convert recurring evidence gaps into better observability, safer defaults, and faster isolation the next time a similar symptom appears.
