Mastering Kubernetes Log Troubleshooting With Kubectl

Kubernetes log troubleshooting starts when a pod is restarting, stuck in CrashLoopBackOff, or failing to pull its image and nobody can tell whether the problem is code, configuration, or the cluster itself. The fastest path is usually to combine kubectl logs, pod troubleshooting, container logs, and Kubernetes debugging commands with describe and events, then narrow the issue from symptom to root cause.

Introduction

Most Kubernetes incidents are not “log problems.” They are symptom problems that happen to show up in logs, events, or restart counters. A pod can fail because an app crashed on startup, because a container image cannot be pulled, because a readiness probe is too aggressive, or because the node is under pressure.

This guide focuses on practical Kubernetes debugging with kubectl logs, describe, and related commands. It is built for people who need to diagnose pod failures fast, not for people who want a theory lesson they will never use during an outage.

Kubernetes troubleshooting works best when you treat the pod as evidence, not as the whole story.

The right mindset is simple: inspect symptoms, gather evidence, narrow the scope, and confirm the root cause. That means looking at application-level failures first, then container runtime symptoms, then scheduling or cluster-state problems if the pod-level clues do not close the case.

For reference on the platform itself, the official Kubernetes documentation is the best source for command behavior, object status fields, and workload concepts. For the operational side of cloud-native logging and observability, the Cloud Native Computing Foundation ecosystem remains the most relevant place to track current tooling patterns.

Understanding Kubernetes Logs And Pod Failure Symptoms

Kubernetes logs are one source of troubleshooting data, but they are not the only source. The most useful inputs are application logs, container logs, pod events, node status, and resource metrics. A good investigation connects all five instead of over-trusting one signal.

Common pod symptoms tell you where to start. CrashLoopBackOff usually means the container starts, exits, and gets restarted repeatedly. ImagePullBackOff and ErrImagePull point to image reference, registry, or credential problems. OOMKilled usually means memory pressure. Pending often means scheduling constraints, missing resources, or node issues.

What each symptom usually means

• CrashLoopBackOff often indicates the process is crashing during startup, before it becomes stable.
• ImagePullBackOff usually means Kubernetes cannot fetch the image from the registry.
• Pending means the pod has not been scheduled or cannot be placed on a node.
• OOMKilled means the container exceeded its memory limit or the node ran out of memory headroom.
• Error often appears when the container exits with a non-zero code after startup.

Logs help distinguish a code bug from a dependency outage or a bad configuration. A stack trace can point to an application exception. A timeout may indicate a downstream API is unavailable. An authentication failure may reveal a missing secret or environment variable. A DNS lookup error can indicate cluster network problems or service discovery issues.

Logs alone are not enough because they do not always show scheduling failures, mount errors, pull issues, or probe failures. That is why pod troubleshooting should be layered: start with the pod, then the container, then the node, then the cluster. This approach avoids the common mistake of tuning the wrong setting because the wrong symptom looked important.

For container and Kubernetes security context, the official CNCF Kubernetes project page and the Kubernetes Pods documentation are good grounding references for how pod state and workload behavior are represented.

Getting Started With Kubectl For Log Investigation

The essential command set is small, which is why it is worth memorizing. kubectl get pods shows status and restart count. kubectl describe pod shows container state, events, image references, probes, and scheduling details. kubectl logs pulls stdout and stderr from the container so you can read what the application actually reported.

Before you fetch logs, identify the correct namespace, pod name, container name, and restart count. In real incidents, people often grab the wrong pod because the deployment has multiple replicas or the same app exists in several namespaces. Copy the exact pod name from kubectl get pods -n your-namespace and check the RESTARTS column before going any further.

Commands that save time

• kubectl get pods -n app-namespace -o wide to see node placement and pod IP.
• kubectl describe pod pod-name -n app-namespace to inspect status and events.
• kubectl logs pod-name -n app-namespace to read the current container output.
• kubectl logs pod-name -n app-namespace --previous to read the last crashed instance.
• kubectl get pods -n app-namespace -l app=myapp to filter by label.

Useful output formats include -o wide for quick node visibility, -o yaml for full object details, and -o json when you want to pipe output into jq. Field selectors and label selectors help you isolate exactly the pods you care about instead of scrolling through everything in the namespace.

For example, a label selector like kubectl get pods -n prod -l app=payments is faster than hunting by name in a busy namespace. A timestamp check matters too, because event order often tells you whether the failure happened during image pull, container start, probe execution, or shutdown.

For command syntax and object formatting details, the official kubectl reference is the right source. For the broader platform behavior, the Microsoft Learn Kubernetes content is a useful vendor-neutral place to review workload and cluster concepts without guessing about command behavior.

Using Kubectl Logs To Read Application Output

kubectl logs retrieves a container’s stdout and stderr stream. That means it shows what the application wrote to standard output, not every possible file on disk. If your app writes logs to a file inside the container but never sends them to stdout, kubectl logs will look empty or incomplete.

The basic syntax is straightforward: kubectl logs pod-name. From there, you can narrow the time window and reduce noise. Use --tail=100 to see the last 100 lines, --since=10m to see recent history, and -f to follow streaming output during a live incident.

Reading the right log stream

1. Run kubectl logs pod-name -n namespace to get the current container output.
2. Add --tail or --since to reduce irrelevant history.
3. Use -f when the issue is still happening and you want live output.
4. If the pod has more than one container, add -c container-name.
5. If the pod is restarting, repeat the command with --previous.

Multi-container pods require extra care. If you have a main app container plus a sidecar, kubectl logs needs the container name or you may read the wrong stream. That matters in service mesh setups, log shipping sidecars, and jobs that use init containers or helper containers.

What should you look for? Stack traces, authentication failures, timeouts, parsing errors, missing file references, and dependency failures are all high-value clues. A message like “connection refused” may point to the wrong service name, a dead dependency, or a port mismatch. A “permission denied” message often points to a security context, volume mount, or file-system issue rather than an application bug.

For structured logging practices and log ingestion patterns, the official Elastic documentation is a useful reference, and the Grafana Loki project shows why label-based log querying has become common in Kubernetes environments.

Diagnosing CrashLoopBackOff And Restarting Containers

CrashLoopBackOff usually means the container starts, fails, and gets restarted repeatedly by Kubernetes. The restart backoff is Kubernetes trying to avoid hammering a failing workload. In practice, this is one of the most common pod failure patterns you will see during Kubernetes debugging.

The first move is usually kubectl logs pod-name --previous. The current container may already be in a fresh restart, which means its current logs are sparse or misleading. The previous instance often contains the actual stack trace, startup exception, or configuration error that caused the crash.

What to correlate when a container keeps restarting

• Startup log timestamps against container restart timestamps.
• Readiness and liveness probe failures.
• Missing environment variables or secret references.
• Bad config files, malformed YAML, or invalid JSON input.
• Entrypoint errors, port binding failures, and uncaught exceptions.

Probe timing is especially important. A liveness probe that starts too early can kill a healthy application before it finishes booting. A readiness probe that fails because the app still depends on a database or cache will keep the pod out of service even if the process itself is fine.

Many teams misdiagnose a probe problem as an application crash. The difference is important. If the app logs show normal startup and then the kubelet kills the container because the probe failed, the fix is usually probe tuning. If the app logs show an exception or exit code before probes even matter, the bug is in the application or startup configuration.

For probe behavior and pod lifecycle semantics, the official Kubernetes probes documentation is the best reference. For code-level failure patterns and container startup conventions, the OWASP Top 10 is also useful when the failure involves insecure defaults, broken input handling, or poor error handling.

Investigating ImagePullBackOff, ErrImagePull, And Start-Up Failures

ImagePullBackOff and ErrImagePull mean the container never gets to application startup because Kubernetes cannot fetch the image. The failure may look like a pod problem, but the real issue is often an image reference, registry credential, or node connectivity problem.

kubectl describe pod is the fastest way to diagnose these cases. The Events section usually shows whether the image tag does not exist, the registry rejected authentication, or the node could not reach the registry. If the manifest looks correct but the pull still fails, the node’s network path or DNS resolution may be broken.

Things to verify in the manifest

• The image name is spelled correctly.
• The tag exists and matches what was pushed.
• imagePullPolicy fits the release process.
• The image pull secret is attached to the service account or pod spec.
• The registry endpoint is reachable from the node.

Private registry access is a frequent culprit. If the workload can run on one node but not another, the issue may be node-level DNS, firewall, or credential distribution rather than the manifest itself. In some environments, the image pull secret is correct but not mounted on the right namespace or service account, so the failure appears only after deployment.

Events often tell the story more clearly than application logs because the image never started. If there is no container process, there may be no container log stream at all. That is why image pull incidents are a reminder that pod troubleshooting must include control-plane and node signals, not just application output.

For image and registry behavior, the official Kubernetes container images documentation is the primary reference. For registry access patterns and image delivery architecture, the Docker documentation remains a practical reference point even when the runtime is not Docker itself.

Using Pod Events And Describe Output To Find Root Causes

kubectl describe pod is one of the most underrated troubleshooting commands in Kubernetes. It surfaces the pod’s timeline: scheduling, image pulling, container creation, readiness checks, volume mounts, and failure messages. If you want a concise incident narrative, this is usually where it lives.

The Events section often reveals what logs do not. You may see back-off messages, volume mount failures, permission issues, scheduling constraints, or repeated probe failures. The order of those events matters because it tells you what happened first and what failed later.

What describe output can expose

• Node affinity conflicts that keep a pod unscheduled.
• Taints and tolerations that prevent placement on a node.
• Secret or ConfigMap references that are missing or invalid.
• Volume mount errors caused by bad paths or permissions.
• Security context issues that block file or process access.

When a pod stays Pending, check resource requests, node selectors, affinity rules, and tolerations before assuming the cluster is broken. A pod can remain unscheduled simply because no node satisfies its constraints or because the requested CPU and memory are larger than anything available in the cluster.

Describe output also helps with hidden startup problems. A pod may look healthy at first glance but fail because a secret reference is wrong, a ConfigMap key is missing, or the container security context does not allow the application to write to a mounted directory. Those failures often surface in events before the application even begins to log.

For event and scheduling behavior, the official Kubernetes scheduling and eviction documentation is the authoritative source. If you need a broader operational lens on incident handling, the NIST Cybersecurity Framework is a useful model for identifying, protecting, detecting, responding, and recovering from cluster issues.

Troubleshooting Resource Pressure, OOMKills, And Node Problems

Resource pressure can make healthy workloads look broken. If memory is too tight, the kernel can kill the process and Kubernetes reports OOMKilled. If CPU is constrained, the app may become slow, miss probe deadlines, or appear unstable even though it is technically running.

When multiple pods fail at once, move beyond the pod and inspect node health. kubectl get nodes tells you whether nodes are Ready, NotReady, or under pressure. If several workloads on the same node start failing together, the root cause may be node disk pressure, memory pressure, network issues, or kubelet instability.

How to confirm resource-related failures

1. Check the pod status for OOMKilled or eviction messages.
2. Compare container memory limits with the application’s actual usage pattern.
3. Inspect node status with kubectl get nodes and kubectl describe node.
4. Look for disk, memory, or PID pressure on the node.
5. Review requests and limits in the deployment spec.

Resource sizing is not just about capacity planning. It is also about behavior under load, startup spikes, and garbage collection patterns. A Java app may need a higher memory limit than its average usage suggests. A bursty API may need CPU headroom to avoid probe failures during traffic spikes. A log-heavy container may pressure disk faster than expected.

Comparing requests and limits against real usage is the practical way to detect mismatches. If a pod gets evicted or killed repeatedly, do not assume the app is unstable until you rule out inadequate limits, node pressure, or poor bin-packing caused by aggressive resource requests.

For workload resource management, the official Kubernetes resource management documentation is the best reference. For broader cluster operations and node health, Red Hat’s Kubernetes overview is a practical vendor source that explains how pods, nodes, and scheduling interact.

Working With Multi-Container Pods And Sidecars

Pods with sidecars, init containers, or helper containers require container-specific log inspection. A healthy main application container does not guarantee the pod is healthy. If an init container fails, the pod may never start. If a sidecar fails, readiness may never go green.

Use kubectl logs -c container-name to inspect each container independently. That is essential when a pod includes a service mesh proxy, log shipper, configuration helper, or database migration container. The failure may be in the supporting container, not the application container people assume is guilty.

Common multi-container failure patterns

• Init containers fail on config bootstrapping or secret retrieval.
• Sidecars fail to become ready and block pod readiness.
• Log shippers crash because they cannot access mounted paths.
• Service mesh proxies delay startup or cause connection interception issues.
• One container succeeds while another quietly fails in the background.

Init container logs are especially valuable for setup failures. If a migration container cannot connect to a database, the main app may never start. If a bootstrap container cannot read a secret, the pod may loop before the primary process ever launches. This is a common blind spot in pod troubleshooting because the “real app” is not always the first thing to fail.

Sidecars can also hide problems. A tracing agent, proxy, or log collector may be healthy enough to keep the pod running but unhealthy enough to make the application unreachable or slow. The cure is to inspect each container separately and correlate the output with pod events and readiness conditions.

For service mesh and sidecar patterns, the Istio documentation is a useful operational reference. For container runtime behavior and pod design principles, the official Kubernetes sidecar documentation is the more direct source.

Building A Repeatable Kubernetes Log Troubleshooting Workflow

A repeatable workflow is the difference between fast incident response and random command spamming. The best Kubernetes debugging habits are simple: identify the pod, inspect status, read events, view logs, and then confirm the underlying cause. If the first layer does not explain the issue, move outward one level at a time.

Use a checklist to sort the problem into one of a few buckets: application, configuration, image, resource, networking, or cluster. That classification prevents over-fixing. If the failure is image-related, don’t waste time tuning probes. If the problem is a bad secret, don’t start resizing memory limits.

A practical incident workflow

1. Find the failing pod with the correct namespace and labels.
2. Check restart count, age, and current status.
3. Read kubectl describe pod and note the event timeline.
4. Pull current logs and, if needed, previous logs from the failed container.
5. Inspect workload, node, and service dependencies only if the pod evidence points there.
6. Validate the fix by watching the workload become Ready and stay stable.

When should you escalate? If the pod-level evidence points to deployment, replica set, service, ingress, or node-level behavior, move up the stack. If every replica fails the same way, the issue may be in the deployment template or image. If only one node has the problem, the issue is likely node-specific. If traffic cannot reach a healthy pod, the service or ingress may be at fault.

Keep an incident note template with timestamps, commands run, symptoms observed, and findings. That sounds basic, but it is how teams reduce repeated incidents and avoid losing the root cause after the pressure drops. A short record also helps when the same failure returns three weeks later with a slightly different symptom.

For operational process and response discipline, the ISACA COBIT framework is useful when you need a control-oriented approach to incident handling, while NIST guidance reinforces structured detection and response habits.

Advanced Tips, Tooling, And Best Practices

Once you are comfortable with the basics, the next gain comes from better tooling and better log hygiene. Tools like stern, kail, and kubetail help aggregate logs across multiple pods, which is useful when a deployment has many replicas and the issue only appears intermittently. jq and jsonpath help filter object output when you need exactly one field instead of a page of YAML.

Centralized logging makes historical analysis much easier. Elasticsearch, Loki, and cloud-native logging stacks let you search across restarts, correlate pods with nodes, and inspect events after the pod is gone. That matters because kubectl logs only works while the log source is still available and the container history has not been lost.

Best practices that pay off later

• Use structured logs so fields can be queried reliably.
• Add correlation IDs so one request can be traced across services.
• Set log levels consistently across applications.
• Redact secrets and sensitive data before logs leave the pod.
• Define retention rules so logs stay available long enough for incident review.

Probes should be added thoughtfully. Liveness, readiness, and startup probes improve reliability when they reflect actual application behavior. They become a problem when they are too strict, too shallow, or tuned to the wrong path. The goal is to detect real failure, not to create a self-inflicted restart loop.

For log pipeline and search design, the Splunk log management guidance is a practical external reference, and the OpenTelemetry documentation is the right source for traces, metrics, and logs correlation patterns that improve incident debugging.

If your organization is building Kubernetes security training or broader NICE Workforce Framework-aligned skills, include log troubleshooting in the core labs. A kubernetes specialist should be able to explain why a pod is failing, not just recite the command that displays the failure.

Conclusion

The core troubleshooting pattern is consistent: use kubectl logs, describe, events, and pod status together to isolate the failure. That combination usually tells you whether the issue is inside the application, in the container runtime, in the manifest, or somewhere lower in the cluster.

Logs matter, but they only become useful when you read them in context. A crash trace without events can mislead you. An image pull error without pod state can send you down the wrong path. A restart loop without previous logs can hide the real failure entirely.

Adopt a consistent workflow, keep your incident notes clean, and standardize the tools you use for pod troubleshooting and Kubernetes debugging. That habit will save time the next time a workload fails during a deploy, a node gets noisy, or a sidecar starts blocking readiness.

Before you close an incident, verify the fix with fresh logs, healthy pod status, and stable restarts. If the pod stays Ready and the log stream remains clean under normal traffic, you have not just masked the symptom—you have likely fixed the root cause.

CompTIA®, Cisco®, Microsoft®, AWS®, EC-Council®, ISC2®, ISACA®, and PMI® are trademarks of their respective owners.