Posts tagged logs
OpenSearch Storage Tiering for Log Analytics: A Hands-On Guide
- 12 November 2026
Blog Post Publish Date: 2026/11/12
This blog post is a hands-on implementation of storage tiering in OpenSearch, using a log analytics scenario based on my experience. In the example, log indexes move through three tiers: hot nodes on local NVMe for the first 3 days, warm nodes on EBS until 90 days, and S3 snapshots that come back as searchable snapshots only when someone needs old data. You will configure the node attributes, index template, and Index State Management (ISM) policy that automate the flow, and see how I run the searchable tier on short-lived Kubernetes nodes and which costs and failures I ran into.
Avoiding Over-Sharding in OpenSearch with Filtered Aliases per Environment
- 11 November 2026
Blog Post Publish Date: 2026/11/11
This blog post covers one of the strategies I use to avoid over-sharding in OpenSearch. Instead of per-environment indexes, I write all logs into one index series and expose each environment as a filtered alias. The rule behind it is simple: the environment is an attribute of each document, so it belongs in a field, not in the index name. The cluster keeps fewer, fuller shards and one mapping, and OpenSearch Dashboards users still get a per-environment view that behaves like a dedicated index.
Kubernetes Logs Simplified: Everything You Need to Know About Pod Logging
- 22 June 2025
Blog Post Publish Date: 2025/06/22
This blog post explains where Kubernetes stores container logs and how $ kubectl logs and log shippers read them. It also covers the CRI log line format, how log rotation works, and how to ship pod logs to an external system with a DaemonSet. Kubernetes logging can look complex, but knowing these details makes troubleshooting much easier.
Fluent Bit: Generating Log IDs with SHA-256 Hashes for Log Deduplication
- 19 June 2025
Blog Post Publish Date: 2025/06/19
This blog post explains why Fluent Bit can deliver the same log more than once, and why deduplication needs a unique ID derived from the log content. It then shows how to generate a SHA-256 hash of each log line with only native Fluent Bit features, without external scripts or plugins.
Logstash DB Enrichment: Tips and Traps
- 16 February 2025
Blog Post Publish Date: 2025/02/16
This blog post covers the lessons from enriching logs in Logstash with data from a PostgreSQL database. It explains hidden behaviors of the jdbc_streaming filter that can stop log ingestion, and the JDBC connection string parameters that prevent them. These problems are hard to reproduce and not clearly documented, so I hope these tips save you hours of troubleshooting and forum searching.
Logstash Sniffing: Improving Log Ingestion in OpenSearch Scalable Clusters
- 08 June 2024
Blog Post Publish Date: 2024/06/08
This blog post covers a common problem when Logstash ships logs to an OpenSearch cluster with autoscaling ingest nodes behind a load balancer: one node takes all the traffic while the others sit idle. It shows how the output plugin's sniffing options fix that by discovering the nodes from the OpenSearch API.
Logstash Modular Pipelines: Reusing Inputs, Filters, and Outputs Without Duplication
- 12 November 2023
Blog Post Publish Date: 2023/11/12
This blog post covers a Logstash pipeline structure that removes duplicated code. Each input, filter, and output lives in its own file, and every application's pipeline is composed from those modules with a glob pattern and an environment variable. It includes an example configuration, a Docker Compose lab to try it, and my opinion on when the structure is worth it.