Monitoring a Kubernetes cluster is not the same thing as detecting a security incident. A normal health dashboard can show that all pods are ready while an attacker uses an overprivileged service account to list Secrets or launches a container with unexpected host access. Effective Kubernetes security work connects cluster state, API activity, workload behavior and a documented response process. Those connections are particularly relevant to the monitoring, logging and runtime-security component of the Certified Kubernetes Security Specialist (CKS) examination.
CKS is a remote, command-line performance assessment administered through the Cloud Native Computing Foundation and The Linux Foundation. The live curriculum allocates 20% to Monitoring, Logging and Runtime Security; the remaining domains cover cluster setup, cluster hardening, system hardening, microservice vulnerabilities and supply-chain security. Candidates must previously have passed the Certified Kubernetes Administrator (CKA) exam, although CKA need not still be active. This article explores the monitoring and response work in depth without implying that a few logging tools constitute the entire CKS exam.
Start with a threat and a logging objective
A cluster produces many kinds of signals. The correct collection strategy depends on what must be discovered. If the threat is a stolen service-account credential, a node CPU graph alone will not answer who read a Kubernetes Secret. If the threat is code execution inside a container, an API audit record might show nothing suspicious after the workload was created. A monitoring design must state its detection question and identify a source capable of answering it.
Record the expected workload owner, namespace, allowed network connections, service identity and deployment method. That baseline gives analysts something against which to judge deviations. It is also necessary for distinguishing routine operator activity from unusual actions. A deployment system may create many short-lived Pods; the security question is whether they were created by an authorized identity, from an expected image and under an acceptable policy.
Separate API audit records, container logs and node evidence
Kubernetes API audit logs describe requests made through the API server, including identity, verb, resource and the disposition of the request when auditing is configured. They are useful for investigating role bindings, privilege changes, Secret access and attempts to modify cluster resources. An audit policy decides which events and stages are recorded. Capturing every request body would be expensive and may expose sensitive information; select levels and resources with both investigative value and privacy in mind.
Container and application logs record what programs report. Use kubectl logs to examine a pod and the appropriate container; use the previous-container option where available to inspect a restarted container. An application log may show failed authentication, unusual requests or an exception, but it will not reliably report every syscall or cluster API operation. Container logs should be forwarded from transient nodes when the incident-retention requirement exceeds the life of a Pod.
Node, runtime and network evidence includes kernel or system service logs, container-runtime events and relevant flow records. A process escaping its expected behavior may require host-level telemetry. A Pod repeatedly trying an unfamiliar destination may warrant network-flow analysis, but network policy enforcement must be supported by the installed networking implementation. Logging a denied connection and enforcing a policy are separate activities.
Preserve timestamps, cluster and namespace identifiers, workload identity and retention rules. In hybrid or managed clusters, the control-plane audit log may be available through the provider rather than a file on the Kubernetes node. Do not assume the same audit backend or administrative privileges exist in every platform. Test that a recorded incident can be reconstructed after its original Pod has been deleted.
Find risky identities with RBAC evidence
Role-based access control limits which Kubernetes resources an identity can read or change. An unexpected ClusterRoleBinding to a broad role is worth investigating because it can expand the reach of a compromised account. Use targeted authorization checks such as kubectl auth can-i and inspect relevant RoleBindings rather than inferring permissions from a username. Service accounts should be granted the minimum permissions necessary for their workload.
For investigation, pair changes to RBAC objects with API requests from the affected identity. Distinguish a rejected authorization attempt from a permitted but unusual operation; both may be informative, but they imply different control failures. Review automated controllers separately, because their activity can be frequent and legitimately privileged. A monitoring rule that fires on every controller change will generate noise and may obscure a genuine incident.
Understand runtime alerts without assuming a required vendor
Runtime security inspects workload behavior after deployment. Examples of suspicious events include a new shell launched in a container that normally serves only HTTP traffic, writing to sensitive host paths, or a privileged process with unexpected network access. Falco can detect such activity using syscall and other data sources, but its presence is not a universal CKS requirement or a guarantee that an alert indicates compromise. Other runtime telemetry tools can support equivalent investigations.
Good alert rules are tuned against how the application should behave. A debugging shell in a development namespace differs from an unsolicited shell in a locked-down production payment service. Use workload labels, container images, deployment accounts, timestamps and known maintenance windows to improve context. False positives consume responder time, while exclusions broad enough to hide real attacks create dangerous blind spots.
Host and container hardening affect what runtime events are possible. Pod Security Standards define privileged, baseline and restricted profiles; Kubernetes workload security contexts can prohibit privileged operation and reduce capabilities. A restricted security context does not eliminate all vulnerabilities, and a default-deny NetworkPolicy only works when the relevant policy engine enforces it. Review those protections as part of Kubernetes security, not as a replacement for monitoring.
Build an incident workflow around evidence preservation
When a high-confidence alert arrives, identify the cluster, namespace, affected Pod, service account, source image and deployment controller. Record the first observed time and the specific behavior that triggered the alert. Collect the appropriate API audit events, Pod logs, relevant events and configuration state before deleting resources. Because Kubernetes workloads are ephemeral, evidence may disappear quickly when a deployment restarts or autoscaling removes a node.
Containment is risk-dependent. In one case, a temporary NetworkPolicy and revoked service-account token may reduce further access. In another, draining or isolating an affected node could interrupt healthy services and erase volatile evidence. A responder should coordinate with the incident commander and application owner, record the reason for each action and avoid blanket deletion that destroys useful evidence. Never copy sensitive Secrets into an unsecured incident notebook.
After containment, identify the entry path and authorization boundary. Was a container image vulnerable, a credential exposed, a role binding excessive or an admission control absent? Recovery should replace compromised workloads from trusted images, restore minimally scoped access and verify normal data and network behavior. An incident is not complete because an alert stops firing; the original access path must be tested as closed and affected services must pass functional validation.
Hold a post-incident review that ties each observation to an actionable change: an audit policy gap, missing runtime alert, weak credential rotation, unclear escalation, inadequate backup or insufficient application ownership. A useful detection improvement specifies what event will trigger, who receives it and how the recipient distinguishes actual risk from expected operation. That feedback loop makes repeated incidents less likely and keeps monitoring proportional to operational needs.
Practise with a small, isolated CKS lab
Create a disposable cluster or a dedicated training namespace. Deploy a simple application under a narrowly scoped service account. Enable or locate the available API audit facility in the lab, inspect application and Pod logs, and confirm where a routine RoleBinding creation appears. Where a cluster provider abstracts control-plane configuration, use its supported audit service instead of assuming direct access to API server flags.
Introduce three controlled scenarios: an RBAC authorization denial, a test Pod that performs an unexpected command, and a service that tries to contact a blocked endpoint. Write down which telemetry source actually captured each behavior. Avoid running potentially destructive test commands on production systems, and verify that all training telemetry is deleted according to the organization’s retention rules.
For a fourth scenario, simulate a service identity that can read more objects than intended. Document the minimal RBAC change that would correct it, compare before-and-after authorization checks, and determine whether prior requests remain visible in the configured audit log. The purpose is to demonstrate cause and effect, not merely to produce a green dashboard. Practice completing the exercise under time pressure, but never sacrifice cluster recovery or evidence quality for speed.
Connect this skill to the complete examination
The CKS assessment uses practical command-line tasks and lasts two hours under current CNCF guidance. Preparation should include the whole published curriculum, including workload security, image provenance, cluster control-plane hardening and supported admission-policy mechanisms. The monitoring/logging/runtime segment carries 20%, so overinvesting in one SIEM product while ignoring the other five domains is not a sound exam strategy.
Use the official current curriculum and exam-environment notes to decide which tools may be used during the supervised test. Do not assume that a particular commercial product, managed-cloud console or plug-in will be installed. Rehearse with standard Kubernetes tooling, read command output carefully and verify your changes. In real operations, the strongest security response is a repeatable process that can explain what happened, why it matters and which controls now prevent a recurrence.
