Posts tagged logstash

Logstash DB Enrichment: Tips and Traps

Blog Post Publish Date: 2025/02/16

This blog post covers the lessons from enriching logs in Logstash with data from a PostgreSQL database. It explains hidden behaviors of the jdbc_streaming filter that can stop log ingestion, and the JDBC connection string parameters that prevent them. These problems are hard to reproduce and not clearly documented, so I hope these tips save you hours of troubleshooting and forum searching.

Read more ...


Logstash Sniffing: Improving Log Ingestion in OpenSearch Scalable Clusters

Blog Post Publish Date: 2024/06/08

This blog post covers a common problem when Logstash ships logs to an OpenSearch cluster with autoscaling ingest nodes behind a load balancer: one node takes all the traffic while the others sit idle. It shows how the output plugin's sniffing options fix that by discovering the nodes from the OpenSearch API.

Read more ...


Logstash Modular Pipelines: Reusing Inputs, Filters, and Outputs Without Duplication

Blog Post Publish Date: 2023/11/12

This blog post covers a Logstash pipeline structure that removes duplicated code. Each input, filter, and output lives in its own file, and every application's pipeline is composed from those modules with a glob pattern and an environment variable. It includes an example configuration, a Docker Compose lab to try it, and my opinion on when the structure is worth it.

Read more ...