Posts in Observability
A Hands-On Example of Decoupling App Instrumentation with OpenTelemetry
- 18 November 2026
Blog Post Publish Date: 2026/11/18
This post builds a small Python service instrumented with OpenTelemetry, one piece at a time. It starts with the API only, turns on the SDK without changing the code, routes the metrics through the OpenTelemetry Collector to Prometheus, and then switches to Datadog by changing only the Collector configuration.
OpenSearch Storage Tiering for Log Analytics: A Hands-On Guide
- 12 November 2026
Blog Post Publish Date: 2026/11/12
This blog post is a hands-on implementation of storage tiering in OpenSearch, using a log analytics scenario based on my experience. In the example, log indexes move through three tiers: hot nodes on local NVMe for the first 3 days, warm nodes on EBS until 90 days, and S3 snapshots that come back as searchable snapshots only when someone needs old data. You will configure the node attributes, index template, and Index State Management (ISM) policy that automate the flow, and see how I run the searchable tier on short-lived Kubernetes nodes and which costs and failures I ran into.
Avoiding Over-Sharding in OpenSearch with Filtered Aliases per Environment
- 11 November 2026
Blog Post Publish Date: 2026/11/11
This blog post covers one of the strategies I use to avoid over-sharding in OpenSearch. Instead of per-environment indexes, I write all logs into one index series and expose each environment as a filtered alias. The rule behind it is simple: the environment is an attribute of each document, so it belongs in a field, not in the index name. The cluster keeps fewer, fuller shards and one mapping, and OpenSearch Dashboards users still get a per-environment view that behaves like a dedicated index.
Scaling Down and Resizing OpenSearch Data Nodes Without Losing Data
- 10 November 2026
Blog Post Publish Date: 2026/11/10
This blog post covers the patterns and API calls I use to scale down and resize OpenSearch data nodes without losing data. The rule behind all of them is simple: the data on a node you remove has to end up somewhere else first. If you take nodes away faster than the cluster can relocate their shards, you lose data.
OpenSearch + EC2 Instance Store: A Disruptive Approach to Hot Tier Storage
- 08 November 2026
Blog Post Publish Date: 2026/11/08
This blog post covers running the OpenSearch hot data tier on EC2 Instance Store (local NVMe) instead of EBS. It explains why local disks perform better for this workload, and how to manage the risk of data loss with OpenSearch replication, Kubernetes, and Karpenter.
OpenSearch Shard Allocation: Balanced by Count, Rebalanced by Watermarks
- 04 November 2026
Blog Post Publish Date: 2026/11/04
This blog post explains how OpenSearch distributes shards across data nodes. It covers how the default allocator works, why it can leave disk usage uneven in clusters with many small indices, and how disk watermarks correct that. It also covers other allocation mechanisms and a limitation I run into often: watermarks are one cluster-wide setting, even in clusters with different storage tiers.
Stop Extracting Metrics from OpenSearch Logs
- 03 November 2026
Blog Post Publish Date: 2026/11/03
This blog post explains why counting log lines in OpenSearch is an expensive way to produce a metric. The same number captured as a Prometheus counter is stored once and read almost for free. When you cannot change the application, Fluent Bit can extract that metric while the logs are ingested, so OpenSearch never has to rebuild it at query time.
OpenTelemetry: A Framework for Decoupling Your Observability
- 29 October 2026
Blog Post Publish Date: 2026/10/29
OpenTelemetry is not a database, and it doesn't compete with Fluent Bit, Logstash, or Jaeger. It defines how metrics, traces, and logs are structured and transported, so your telemetry isn't tied to any vendor. The tools you already know can be the backends that receive it. This post explains that idea, the one component OpenTelemetry runs on its own, and where it fits.
Kubernetes Logs Simplified: Everything You Need to Know About Pod Logging
- 22 June 2025
Blog Post Publish Date: 2025/06/22
This blog post explains where Kubernetes stores container logs and how $ kubectl logs and log shippers read them. It also covers the CRI log line format, how log rotation works, and how to ship pod logs to an external system with a DaemonSet. Kubernetes logging can look complex, but knowing these details makes troubleshooting much easier.
Fluent Bit: Generating Log IDs with SHA-256 Hashes for Log Deduplication
- 19 June 2025
Blog Post Publish Date: 2025/06/19
This blog post explains why Fluent Bit can deliver the same log more than once, and why deduplication needs a unique ID derived from the log content. It then shows how to generate a SHA-256 hash of each log line with only native Fluent Bit features, without external scripts or plugins.
Logstash DB Enrichment: Tips and Traps
- 16 February 2025
Blog Post Publish Date: 2025/02/16
This blog post covers the lessons from enriching logs in Logstash with data from a PostgreSQL database. It explains hidden behaviors of the jdbc_streaming filter that can stop log ingestion, and the JDBC connection string parameters that prevent them. These problems are hard to reproduce and not clearly documented, so I hope these tips save you hours of troubleshooting and forum searching.
OpenSearch Concepts: Index and Shards
- 05 January 2025
Blog Post Publish Date: 2025/01/05
This blog post explains two of the most basic concepts in OpenSearch administration: indices and shards. It covers what primary and replica shards are, how their counts affect performance and fault tolerance, and the API calls I use most to inspect indices and shards while troubleshooting.
Logstash Sniffing: Improving Log Ingestion in OpenSearch Scalable Clusters
- 08 June 2024
Blog Post Publish Date: 2024/06/08
This blog post covers a common problem when Logstash ships logs to an OpenSearch cluster with autoscaling ingest nodes behind a load balancer: one node takes all the traffic while the others sit idle. It shows how the output plugin's sniffing options fix that by discovering the nodes from the OpenSearch API.
Logstash Modular Pipelines: Reusing Inputs, Filters, and Outputs Without Duplication
- 12 November 2023
Blog Post Publish Date: 2023/11/12
This blog post covers a Logstash pipeline structure that removes duplicated code. Each input, filter, and output lives in its own file, and every application's pipeline is composed from those modules with a glob pattern and an environment variable. It includes an example configuration, a Docker Compose lab to try it, and my opinion on when the structure is worth it.