AI TL;DR
Too lazy to read? Summarize this post with:
Blog Post Publish Date: 2024/06/08
Logstash Sniffing: Improving Log Ingestion in OpenSearch Scalable Clusters#
This blog post covers a common problem when Logstash ships logs to an OpenSearch cluster with autoscaling ingest nodes behind a load balancer: one node takes all the traffic while the others sit idle. It shows how the output plugin's sniffing options fix that by discovering the nodes from the OpenSearch API.
The Problem: One Ingest Node Takes All the Traffic#
An ingest node is an OpenSearch node that preprocesses documents before they are indexed. The node types documentation describes how a cluster can have dedicated ingest nodes, and on Kubernetes they are often scaled with a Horizontal Pod Autoscaler (HPA) and reached through a load balancer.
That setup has a catch when Logstash is the shipper. Shipping logs over HTTP is stateless, but Logstash keeps a persistent connection to the node the load balancer picked first. As long as that connection stays open, every log goes to the same node. That node gets overloaded while the other ingest nodes are underused. If the HPA scales on average resource usage, the imbalance also hides the real load, so the autoscaler reacts too late or not at all.
The Solution: Let Logstash Discover the Nodes#
The hosts parameter of the output plugin accepts one address or several. With several addresses, Logstash balances the traffic across them. For a fixed set of ingest nodes, listing them all solves the problem. For nodes that scale up and down, a static list goes stale quickly.
Sniffing is the output plugin's answer for dynamic clusters: Logstash periodically asks the cluster which nodes exist and updates its host list. The sniffing parameters read the node addresses from the Nodes Info API. The OpenSearch output plugin is a fork of the Elasticsearch one and keeps these options. For example:
output {
opensearch {
hosts => "https://opensearch-load-balancer:9200"
sniffing => true
sniffing_delay => 30
sniffing_path => "/_nodes/ingest:true"
...
}
}
When Logstash starts, it calls /_nodes/ingest:true to get the addresses of all ingest nodes, and adds them to its host list. Every 30 seconds, it calls the API again, adding new nodes and removing the ones that are gone. The load balancer address is only used to reach that API.
The
sniffing_pathparameter selects which nodes to use. For example,/_nodes/data:truesends logs straight to the data nodes.
Two things are worth checking before you enable it. Logstash connects to the addresses the nodes publish, so they must be reachable from where Logstash runs. If Logstash runs outside the Kubernetes cluster, pod IPs usually aren't. And if the connection uses TLS, the node certificates must be valid for those addresses, not only for the load balancer's hostname.
Why I Use Sniffing#
With sniffing enabled, the traffic spreads evenly across the ingest nodes. It also takes the load balancer out of the data path, removes an extra network hop, and makes better use of resources. The HPA also gets a realistic view of the load, so scaling works the way it should.
It's simple and effective enough that I miss it in other log shippers, such as Fluent Bit. I have been using this approach, and the results are really nice!
For more on how OpenSearch places the data once it arrives, see OpenSearch Concepts: Index and Shards. If your Logstash configuration is growing across many applications, Logstash Modular Pipelines shows how I keep it organized.
Key Takeaways#
Behind a load balancer, Logstash keeps a persistent connection to one OpenSearch node and sends all logs to it.
That overloads one ingest node, leaves the others idle, and gives the autoscaler a misleading average.
A static list in
hostsworks for a fixed set of nodes, but not for nodes that scale up and down.sniffing => truewithsniffing_path => "/_nodes/ingest:true"makes Logstash discover the ingest nodes and refresh the list everysniffing_delayseconds.The load balancer address is only used to reach the Nodes Info API.
The published node addresses must be reachable from Logstash, and TLS certificates must cover them.
References#
I'm Carlos Neto, and I've been running OpenSearch on EKS for over five years. More about me on my home page. If you have questions or a different approach, contact me on LinkedIn.