AI TL;DR Too lazy to read? Summarize this post with: ChatGPT Claude Grok

Blog Post Publish Date: 2025/06/22


Kubernetes Logs Simplified: Everything You Need to Know About Pod Logging#

This blog post explains where Kubernetes stores container logs and how $ kubectl logs and log shippers read them. It also covers the CRI log line format, how log rotation works, and how to ship pod logs to an external system with a DaemonSet. Kubernetes logging can look complex, but knowing these details makes troubleshooting much easier.

How Kubernetes Stores Container Logs#

Kubernetes runs containers through a container runtime, such as containerd or CRI-O. The Container Runtime Interface (CRI) is the API that the kubelet uses to talk to that runtime. As the CRI documentation describes, one part of it defines how container logs are handled: file names, directory locations, and line format. The CRI logging design proposal and the Kubernetes logging architecture page have the details.

Each container in a pod writes its logs to its own file, with everything the application sends to stdout and stderr (the standard streams). The files live on the node where the pod runs:

# pod container log file pattern
/var/log/pods/<namespace>_<podname>_<uid>/<container_name>/<restart-count>.log

# example
/var/log/pods/default_nginx-7f5c7d4f9b-abcde_12345/nginx-container/
  ├── 0.log   # first run of the container
  ├── 1.log   # run after the first restart
  └── ...

The file name is the container's restart count. The first run writes 0.log. If the container crashes or restarts, the new run writes 1.log, then 2.log, and so on, while the previous file stays on disk. The kubelet keeps the logs of the most recent terminated run, which is what $ kubectl logs --previous <pod> -c <container> reads.

To make ingestion easier for third-party tools, the kubelet also creates a symbolic link to each container's current log file in /var/log/containers/. The link name includes the pod name, namespace, container name, and container ID, so log shippers can identify each container:

# >>> symbolic link log file name pattern
/var/log/containers/<pod_name>_<namespace>_<container_name>-<container_id>.log

# >>> symbolic link log file name example
/var/log/containers/nginx-7f5c7d4f9b-abcde_default_nginx-container-12345.log

# >>> real file that the symbolic link points to
/var/log/pods/default_nginx-7f5c7d4f9b-abcde_12345/nginx-container/0.log

$ kubectl logs doesn't use these links. They exist for log shippers such as Fluent Bit, Vector, and Filebeat. A link always points to the log file of the current run and doesn't concatenate older runs. The container ID is part of the link name, so a restarted container gets a new link that points to its new log file. The same happens when the kubelet rotates a file: the path stays the same, but it's a new file.

Log shippers must handle that. Fluent Bit, for example, tracks the inode of each file it reads. When a rotated file has the same name but a new inode, Fluent Bit treats it as a new file and reads it from the start. See the Tail input reference for the details.

The Kubernetes Log Line Format#

The CRI defines the format of each line in the log file:

# >>> log line pattern
<timestamp> <stream> <flags> <message>

# >>> log line examples
2023-10-06T00:17:09.669794202Z stdout F Your log message here
2023-10-06T00:17:09.669794202Z stdout P Another log pt 1
  • timestamp: when the line was written, in RFC 3339 format with nanosecond precision.

  • stream: stdout or stderr.

  • flags: F for a full line, P for a partial line.

  • message: the raw log message from the container.

The P flag appears when a single line is longer than the runtime's maximum log line size, 16 KiB by default in containerd (max_container_log_line_size). The runtime splits the line into chunks: each chunk except the last is written with P, and the last one with F. For example, an application that prints one 40 KB line produces three entries:

2025-06-21T22:15:10.123456789Z stdout P <first 16 KiB of the line>
2025-06-21T22:15:10.123456790Z stdout P <next 16 KiB of the line>
2025-06-21T22:15:10.123456791Z stdout F <rest of the line>

$ kubectl logs joins the partial chunks before showing them, so you see one line. Log shippers have to do the same. Fluent Bit, for example, does it with its built-in CRI multiline parser.

Managing Log Growth with Log Rotation#

Logs can grow until they fill the node's disk, which hurts observability in the cluster and the performance of the applications on that node. Even if an application logs very little, enforce size limits and rotation.

Log rotation for container logs is done by the kubelet. According to the log rotation documentation, the kubelet rotates each container log file when it reaches containerLogMaxSize (10Mi by default) and keeps at most containerLogMaxFiles files (5 by default). Configure these limits in the kubelet instead of using a third-party tool such as logrotate. Renaming, truncating, or closing files behind the kubelet's back can corrupt its log management, and the log layout may change in future Kubernetes versions.

You set the limits in the kubelet configuration file:

/var/lib/kubelet/config.yaml (excerpt)#
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
containerLogMaxSize: 100Mi
containerLogMaxFiles: 5

Or with the equivalent kubelet flags, for example through the kubelet's systemd unit:

Environment="KUBELET_EXTRA_ARGS=--container-log-max-size=100Mi --container-log-max-files=5"

If you change the kubelet unit, put the change in a drop-in file instead of editing the unit itself, as I explain in Systemd Drop-in Overrides.

Shipping Kubernetes Logs to an External System#

For search and long-term analysis, ship workload logs to a log analytics system such as OpenSearch, Elasticsearch, Grafana Loki, Splunk, or Datadog. Inside the cluster, tools such as Fluent Bit, Filebeat, and Vector read, process, and send the logs.

Whatever shipper you choose, the deployment pattern is the same. A DaemonSet is a Kubernetes workload that runs one pod on every node, and the DaemonSet for the shipper mounts the node's /var/log directory with a hostPath volume. The shipper reads /var/log/containers/*.log, and because those links point into /var/log/pods/, the mount must include both directories. Mounting /var/log covers them. For example, a Fluent Bit DaemonSet:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: default
  labels:
    app: fluent-bit
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      containers:
        - name: fluent-bit
          image: docker.io/bitnami/fluent-bit:4.0.3-debian-12-r0
          resources:
            limits:
              memory: "200Mi"
              cpu: "100m"
            requests:
              memory: "100Mi"
              cpu: "50m"
          volumeMounts: 
            - name: config
              mountPath: /opt/bitnami/fluent-bit/conf/fluent-bit.conf
              subPath: fluent-bit.conf
            # >>> MOUNTS THE NODE'S LOG DIRECTORY (BOTH /var/log/containers AND /var/log/pods)
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: fluent-bit-config
        # >>> ALLOWS THE FLUENT BIT POD TO READ THE LOG FILES OF PODS SCHEDULED ON THE NODE
        - name: varlog
          hostPath:
            path: /var/log
            type: Directory

Many log shippers can also add metadata such as pod labels and annotations to each log. For that, the shipper needs permission to read from the Kubernetes API, usually through a ClusterRole bound to its service account. This Fluent Bit ClusterRole is an example. I'll cover metadata enrichment with Fluent Bit in a future post.

Once the logs are flowing, two problems usually come next. Fluent Bit can deliver a log more than once, which I address in Generating Log IDs with SHA-256 Hashes. And teams start counting log lines to build dashboards, which I argue against in Stop Extracting Metrics from OpenSearch Logs.

Key Takeaways#

  • The container runtime writes each container's stdout and stderr to /var/log/pods/<namespace>_<pod>_<uid>/<container>/<restart-count>.log on the node.

  • Each restart writes a new file. kubectl logs --previous reads the log of the previous run.

  • /var/log/containers/ holds symbolic links to the current log file of each container, for log shippers.

  • Each line follows the CRI format <timestamp> <stream> <flags> <message>. P marks a chunk of a line longer than the runtime's limit, and F marks the end.

  • The kubelet rotates container logs with containerLogMaxSize and containerLogMaxFiles. Don't use logrotate on these files.

  • Ship logs with a DaemonSet that mounts the node's /var/log, not only /var/log/containers, because the links point into /var/log/pods.

References#


I'm Carlos Neto, and I've been running OpenSearch on EKS for over five years. More about me on my home page. If you have questions or a different approach, contact me on LinkedIn.