Posts tagged logstash
Logstash DB Enrichment: Tips and Traps
- 16 February 2025
Blog Post Publish Date: 2025/02/16
This blog post covers the lessons from enriching logs in Logstash with data from a PostgreSQL database. It explains hidden behaviors of the jdbc_streaming filter that can stop log ingestion, and the JDBC connection string parameters that prevent them. These problems are hard to reproduce and not clearly documented, so I hope these tips save you hours of troubleshooting and forum searching.
Logstash Sniffing: Improving Log Ingestion in OpenSearch Scalable Clusters
- 08 June 2024
Blog Post Publish Date: 2024/06/08
This blog post covers a common problem when Logstash ships logs to an OpenSearch cluster with autoscaling ingest nodes behind a load balancer: one node takes all the traffic while the others sit idle. It shows how the output plugin's sniffing options fix that by discovering the nodes from the OpenSearch API.
Logstash Modular Pipelines: Reusing Inputs, Filters, and Outputs Without Duplication
- 12 November 2023
Blog Post Publish Date: 2023/11/12
This blog post covers a Logstash pipeline structure that removes duplicated code. Each input, filter, and output lives in its own file, and every application's pipeline is composed from those modules with a glob pattern and an environment variable. It includes an example configuration, a Docker Compose lab to try it, and my opinion on when the structure is worth it.