Carlos Neto

Vim for Kubernetes Certification Exams (CKA, CKAD & CKS)

2026-07-15T00:00:00+00:00

Blog Post Publish Date: 2026/07/15

This post focuses on the minimal Vim configuration and editing techniques I use during Kubernetes certification exams from the Linux Foundation. The goal is not to master Vim, but to learn a small set of commands that can improve your editing speed during the exam.

Vim: Less Mouse, More Efficiency

The exam workstation includes VSCodium (the free and open-source binaries of VS Code), but it’s not a practical choice during the exam. While VSCodium is available on the workstation, all exam tasks are performed on remote nodes via the terminal over SSH, where it is unavailable. In addition, the connection to the exam workstation often has noticeable latency, making mouse operations slow and unreliable. The less you rely on the mouse, the faster you’ll be able to edit files. This is where Vim shines.

You don’t need to become a Vim expert to pass the exam. This guide covers everything you need to efficiently edit Kubernetes YAML manifests under exam conditions. Focus on building muscle memory by practicing these commands in the simulation labs.

Vim Configuration

This configuration is intentionally minimal and optimized for editing Kubernetes YAML manifests. Two-space indentation helps prevent invalid YAML formatting, line numbers make it easier to jump to specific locations, syntax highlighting improves readability, and cursor highlighting helps you keep track of indentation levels.

~/.vimrc

set nu        " Show line numbers
set ai        " Preserve indentation while editing
set et        " Convert tabs into spaces
set ts=2      " Display tabs as two spaces
set sw=2      " Indent and unindent using two spaces
set sts=2     " Make Tab and Backspace use two-space indentation
set hls       " Highlight search matches
set cuc       " Highlight the current cursor column
syntax on     " Enable syntax highlighting

Avoid using set mouse=a due to the exam workstation’s high connection latency. Also, the clipboard settings clipboard+=unnamedplus,autoselect do not work on the exam workstations.

Each exam task is performed on a different remote node. Before starting a task, copy your local ~/.vimrc to the target question machine:

scp ~/.vimrc node01:~/.vimrc

Essential Editing Commands

These are the commands you’ll use most often. They let you navigate, copy, delete, replace, and repeat edits without relying on the mouse.

o           # Open a new line below
Shift+o     # Open a new line above
i           # Enter Insert mode
Shift+a     # Enter Insert mode at the end of the line
cW          # Cut the word under the cursor and enter Insert mode (for word replacement)
Shift+c     # Cut to the end of the line and enter Insert mode (for line replacement)
Ctrl+y      # Copy the character from the line above

y           # Yank (copy)
p           # Paste below
Shift+p     # Paste above
dd          # Cut the current line
dG          # Cut to the end of the file
dgg         # Cut to the beginning of the file

u           # Undo
Ctrl+r      # Redo

Shift+v     # Enter Visual Line mode (for indentation and copying)
Ctrl+v      # Enter Visual Block mode (for commenting lines)
>           # Indent to the right
<           # Indent to the left
.           # Repeat the last command
~           # Toggle character case

:retab      # Convert tabs to spaces
:wq         # Save and quit (or ZZ)

Cursor Movement

As mentioned previously, I strongly recommend not using set mouse=a during the exam. The remote exam workstations have higher latency than the Killer Shell simulation environment, making mouse interactions in Vim slow and unreliable. Use the mouse only for scrolling. For navigation, use the following commands:

/greeting   # Search for "greeting"
/foo/bar    # Search for "foo/bar"
n           # Jump to the next match
N           # Jump to the previous match
12gg        # Jump to line 12
0           # Jump to the beginning of the line
gg          # Jump to the beginning of the file
G           # Jump to the end of the file
W           # Jump to the next word
B           # Jump to the previous word

YAML Indentation

Indentation mistakes are one of the most common causes of invalid Kubernetes manifests. Vim makes it easy to indent entire YAML blocks to the right or left while preserving their structure.

To increase the indentation:

Shift+v   # 1. Select lines in Visual Line mode (↑↓)
>         # 2. Shift indentation to the right
ESC       # 3. Apply the change
.         # 4. Repeat the indentation

If you need to reverse the indentation or perform another operation on the same lines, use gv to restore the previous Visual selection:

gv      # 1. Reselect the previous Visual selection
<       # 2. Shift indentation to the left
ESC     # 3. Apply the change
.       # 4. Repeat the operation

Commenting Multiple Lines

Visual Block mode allows you to insert the same text across multiple lines simultaneously. This is particularly useful for commenting or uncommenting YAML blocks.

Ctrl+v    # 1. Select columns across lines in Visual Block mode (↑↓)
Shift+i   # 2. Enter Insert mode
#         # 3. Type the comment character
ESC       # 4. Apply the change to all selected lines

The same technique can also be used to insert identical text across multiple lines, such as prefixes, labels, or environment variables.

Search and Replace

When renaming resources, labels, namespaces, image names, or environment variables, replacing every occurrence manually is both slow and error-prone.

To replace every occurrence of a string:

:%s/foo/bar/g                   # Replace "foo" with "bar"
:%s#/kubernets/#/kubernetes/#g  # Replace "/kubernets/" with "/kubernetes/"

Sometimes, however, you only want to replace one occurrence at a time. In that case, use the following workflow:

*         # Search for the word under the cursor (or use "/greeting")
cW        # Replace the current occurrence
foobar    # Type the replacement text
n         # Jump to the next match
.         # Repeat the replacement
n         # Jump to the next match
n         # Skip this match and jump to the next one
Shift+n   # Go back to the previous match
.         # Repeat the replacement here

This approach lets you review each occurrence before replacing it, making it safer than a global search-and-replace.

Working with YAML Blocks

Many editing operations can be performed directly on a range of line numbers without entering Visual mode. This is especially useful when moving, copying, deleting, or reindenting large YAML blocks.

:%<         # Unindent all lines
:10,15>     # Indent lines 10-15
:10,15<     # Unindent lines 10-15
:30,50x     # Delete lines 30-50
:30,50t70   # Copy lines 30-50 and paste below line 70
:30,50m70   # Cut lines 30-50 and paste below line 70
:30,50d     # Cut lines 30-50
:10,15y     # Copy lines 10-15
:50put      # Paste lines below line 50

Working with Numbers

Kubernetes manifests frequently contain numeric values such as replica counts, ports, resource requests, limits, and probe timings. Instead of deleting and retyping numbers, Vim can increment or decrement them directly.

Place the cursor before or on a number, then use Ctrl+a to increment it or Ctrl+x to decrement it. You can also provide a count to change the value by a larger amount:

50 » Ctrl+a     # Increment the next number by 50
40 » Ctrl+x     # Decrement the next number by 40

For example, with the cursor on replicas:

replicas: 3

Pressing Ctrl+a changes it to:

replicas: 4

Likewise, 10 » Ctrl+a changes it from 3 to 13.

Running Shell Commands in Vim

You can leverage operating system CLI tools to process and edit file content directly inside Vim. For example, you can execute yq on the current buffer to check for syntax errors and format the file:

:%!yq

You can also apply the same approach to a selection of lines. For example, to sort and remove duplicate lines, select the lines and pipe them through sort and uniq:

Shift+v         # 1. Select lines in Visual Line mode (↑↓)
:!sort | uniq   # 2. Sort and remove duplicates from the selection

Vim can also execute commands without modifying the current buffer. This is useful when you need to quickly inspect information from the cluster while editing a manifest. For example, when editing a NetworkPolicy, you can check pod and namespace labels with:

:!kubectl get pods,ns --show-labels -n foobar

Another useful trick is saving protected files without opening them with sudo vim. This is particularly handy when you forget to open a file with elevated privileges and have already made several changes. Instead of reopening the file with sudo vim and repeating your edits, you can write the current buffer using:

:w !sudo tee %

The :w normally saves the buffer to a file. When followed by !, it sends the buffer’s content to an external command’s standard input (STDIN) instead of writing to the file directly.

Learn More

Take a look at the complete Vim cheatsheet at https://vim.rtorr.com/ to learn more Vim commands and tricks.

For more tips on how I configure my Kubernetes exam workstation, including Bash aliases and search history shortcuts, take a look at my previous blog post Workstation Setup for Kubernetes Exams (CKA, CKAD & CKS).

Workstation Setup for Kubernetes Exams (CKA, CKAD & CKS)

2026-07-09T00:00:00+00:00

Blog Post Publish Date: 2026/07/09

This blog post walks through the workstation optimizations I use to work more efficiently during the exam, including Vim configuration, Bash keybindings, shell aliases, and other tweaks that help reduce friction and save valuable time.

Time management is one of the most critical factors for passing the Linux Foundation Kubernetes certification exams. The CKA, CKAD, and CKS are hands-on exams conducted on remote Ubuntu instances running the XFCE desktop environment, where network latency and the remote desktop experience can slow you down. Check out the following tips to optimize your setup.

Configure the U.S. Keyboard Layout (MacBooks)

If you’re taking the exam on a MacBook, make sure your keyboard layout is set to U.S.. Otherwise, the remote Linux environment may not recognize certain characters correctly, especially ~ and `. This is a common issue with layouts such as U.S. International - PC and Brazilian – ABNT2.

Before the exam, perform the following steps:

Open System Settings » Keyboard » Input Sources.
Add U.S. input source.
Remove any other input sources so only U.S. remains.

This helps prevent unexpected keyboard mapping issues during the exam.

Disable Mission Control Shortcuts (MacBooks)

By default, pressing CTRL + ←→ switches between macOS Spaces. During the exam, this can be interpreted by the proctor as leaving the exam environment, potentially triggering a warning or even interrupting your session.

Before the exam, perform the following steps:

Open System Settings » Keyboard » Keyboard Shortcuts.
Select Mission Control.
Disable all shortcuts under Mission Control.

This helps prevent accidental desktop switching and reduces the risk of unnecessary proctor alerts or exam suspension on suspicion of consulting unauthorized material.

Disable Window Tiling Apps (MacBook)

If you use window tiling apps such as Rectangle or Magnet, it’s strongly recommended to disable/close them before the exam. Their keyboard shortcuts can conflict with the exam workstation, causing unexpected behavior when using shortcuts such as ⌘ + ↑↓←→.

History Search by Prefix

This is probably my most valuable tip!

The exam environment uses Bash as the default shell. Behind the scenes, Bash relies on the GNU Readline library, which provides command-line editing, key bindings, and history navigation. For example, the interactive reverse search that appears when you press CTRL + R is implemented by GNU Readline.

One of Readline’s most useful features is history search by prefix, that is disabled by default. The history-search-backward and history-search-forward functions search your command history using the text that already exists before the cursor, instead of simply moving to the previous or next command.

For example, imagine your command history contains:

kubectl get pods
kubectl get nodes
kubectl describe pod nginx
helm list

If you type:

kubectl g

and press ↑, history-search-backward cycles only through commands that begin with kubectl g:

kubectl get nodes
kubectl get pods

Commands such as kubectl describe pod nginx or helm list are skipped because they do not match the typed prefix.

This is much faster than the default ↑ behavior, which walks through every command in your history, and often more convenient than repeatedly using CTRL + R during the exam.

To enable this behavior, create the ~/.inputrc file and add the following configuration:

~/.inputrc

"\e[A": history-search-backward   # ↑ (previous matching command)
"\e[B": history-search-forward    # ↓ (next matching command)

The file is automatically loaded whenever a new Bash session starts. If you modify it during an existing session, reload it with:

bind -f ~/.inputrc

Setting Up Vim

The exam workstation includes VSCodium. However, every exam task requires you to connect to a remote node via SSH, and VSCodium is not available on those remote machines. In addition, the exam workstation often has high latency, making mouse operation unreliable, this is where Vim shines.

The following settings provide a much better editing experience for Kubernetes YAML manifests.

Create the ~/.vimrc file and add the following configuration:

~/.vimrc

set nu        " Show line numbers
set ai        " Preserve indentation while editing
set et        " Convert tabs into spaces
set ts=2      " Display tabs as two spaces
set sw=2      " Indent and unindent using two spaces
set sts=2     " Make Tab and Backspace use two-space indentation
set hls       " Highlight search matches
set cuc       " Highlight the current cursor column
syntax on     " Enable syntax highlighting

Avoid using set mouse=a due to the exam workstation’s high connection latency. Also, the clipboard settings with clipboard+=unnamedplus,autoselect does not work on the exam workstations.

I have created a blog post Vim for Kubernetes Certification Exams (CKA, CKAD & CKS) with editing techniques I use most frequently during exams.

Kubectl Aliases and Shortcuts

Each remote exam node already contains a preconfigured ~/.bashrc file with several useful settings. A handy trick is to copy it to main workstation, append your own aliases, and then copy it back to the question nodes.

First, copy the ~/.bashrc from question node:

scp cks1234:~/.bashrc ~/.bashrc

Append the following aliases to the end of the file:

~/.bashrc

### omitted the .bashrc copied from question node

alias kgp="kubectl get pods"

alias kaf="kubectl apply -f"
alias kdel="kubectl delete --now"
alias kdelp="kubectl delete pod --now"

alias kns="kubectl config set-context --current --namespace"
alias kctx="kubectl config use-context"

export x="--dry-run=client -oyaml"

INFO: The export x is useful shorthand for generating YAML in imperative kubectl commands such as k run nginx --image nginx:latest $x > nginx.yaml

Copy Settings to Exam Question Node

All exam tasks are required to be performed on the remote question node via SSH. A convenient workflow is to create your configuration files once on the main instance and then copy them to the question node before starting the task.

scp ~/.inputrc ~/.vimrc ~/.bashrc cks1234:~/
ssh cks1234

References

Helm Tips: Inspect Your Chart Context with JSON

2026-02-25T00:00:00+00:00

Blog Post Publish Date: 2026/02/25

Debugging Helm charts can be frustrating when starting out, especially if you come from programming languages with debuggers and IDE auto-completion that provide a smoother development experience. It is common to face issues when you create a template that tries to access a parameter defined in the .Values files, and for any reason, whether a syntax error or typo, the $ helm template command fails, and the stack trace shows confusing messages that don’t point to the root cause. This blog post provides an alternative approach to inspect the Helm Chart context with JSON, helping you understand how to access chart properties for use in your templates.

Are There IDEs for Helm Charts?

Unfortunately, there is no definitive IDE specifically for Helm. There are plugins for VSCode and JetBrains IDEA that provide some tooling for auto-completion, but they don’t come close to the code experience available with popular programming languages like Python and Java. I use VSCode’s Kubernetes (by Microsoft) + YAML (by Red Hat), which helps avoid typos in .Values usage and checks available Go template functions. However, there are limitations, some variables are omitted in auto-complete, which diminishes the experience. I tried the JetBrains Helm Plugin, but it doesn’t work as simply as the .gif present in the Plugin page shows. Additionally, the Kubernetes plugin is not available in IntelliJ IDEA without an Ultimate subscription.

Debug the Right Way!

As an alternative to lacks a robust auto-completion, you can dump the entire Helm Chart context in JSON format to understand the Values references, Chart and Subchart metadata, and available static files.

Let’s get started. First, create a Helm Chart called foobar.

helm create foobar

Now, create a template named debug.yaml:

vim templates/debug.yaml

The debug.yaml file should contain:

{{ . | toPrettyJson }}

Now, render only the debug.yaml template:

helm template -s templates/debug.yaml . | tail -n +3 | jq

The output shows the Helm Chart context with all variables available in your templates.

{
  "Capabilities": {
    "KubeVersion": {
      "Version": "v1.34.0",
      "Major": "1",
      "Minor": "34"
    },
    "APIVersions": [
      "v1",
      "admissionregistration.k8s.io/v1",
      "admissionregistration.k8s.io/v1alpha1",
      "admissionregistration.k8s.io/v1beta1",
      "internal.apiserver.k8s.io/v1alpha1",
      "apps/v1",
      "apps/v1beta1",
      "apps/v1beta2",
      "authentication.k8s.io/v1",
      "authentication.k8s.io/v1alpha1",
      "authentication.k8s.io/v1beta1",
      "authorization.k8s.io/v1",
      "authorization.k8s.io/v1beta1",
      "autoscaling/v1",
      "autoscaling/v2",
      "autoscaling/v2beta1",
      "autoscaling/v2beta2",
      "batch/v1",
      "batch/v1beta1",
      "certificates.k8s.io/v1",
      "certificates.k8s.io/v1beta1",
      "certificates.k8s.io/v1alpha1",
      "coordination.k8s.io/v1alpha2",
      "coordination.k8s.io/v1beta1",
      "coordination.k8s.io/v1",
      "discovery.k8s.io/v1",
      "discovery.k8s.io/v1beta1",
      "events.k8s.io/v1",
      "events.k8s.io/v1beta1",
      "extensions/v1beta1",
      "flowcontrol.apiserver.k8s.io/v1",
      "flowcontrol.apiserver.k8s.io/v1beta1",
      "flowcontrol.apiserver.k8s.io/v1beta2",
      "flowcontrol.apiserver.k8s.io/v1beta3",
      "networking.k8s.io/v1",
      "networking.k8s.io/v1beta1",
      "node.k8s.io/v1",
      "node.k8s.io/v1alpha1",
      "node.k8s.io/v1beta1",
      "policy/v1",
      "policy/v1beta1",
      "rbac.authorization.k8s.io/v1",
      "rbac.authorization.k8s.io/v1beta1",
      "rbac.authorization.k8s.io/v1alpha1",
      "resource.k8s.io/v1",
      "resource.k8s.io/v1beta2",
      "resource.k8s.io/v1beta1",
      "resource.k8s.io/v1alpha3",
      "scheduling.k8s.io/v1alpha1",
      "scheduling.k8s.io/v1beta1",
      "scheduling.k8s.io/v1",
      "storage.k8s.io/v1beta1",
      "storage.k8s.io/v1",
      "storage.k8s.io/v1alpha1",
      "storagemigration.k8s.io/v1alpha1",
      "apiextensions.k8s.io/v1beta1",
      "apiextensions.k8s.io/v1"
    ],
    "HelmVersion": {
      "version": "v4.0.5",
      "git_commit": "1b6053d48b51673c5581973f5ae7e104f627fcf5",
      "git_tree_state": "clean",
      "go_version": "go1.25.5",
      "kube_client_version": "v1.34"
    }
  },
  "Chart": {
    "APIVersion": "v2",
    "Annotations": null,
    "AppVersion": "1.16.0",
    "Condition": "",
    "Dependencies": [],
    "Deprecated": false,
    "Description": "A Helm chart for Kubernetes",
    "Home": "",
    "Icon": "",
    "IsRoot": true,
    "Keywords": [],
    "KubeVersion": "",
    "Maintainers": [],
    "Name": "foobar",
    "Sources": [],
    "Tags": "",
    "Type": "application",
    "Version": "0.1.0"
  },
  "Files": {
    ".helmignore": "IyBQYXR0ZXJucyB0by...",
    "output.yaml": "LS0tCiMgU291cmNlOi..."
  },
  "Release": {
    "IsInstall": true,
    "IsUpgrade": false,
    "Name": "release-name",
    "Namespace": "default",
    "Revision": 1,
    "Service": "Helm"
  },
  "Subcharts": {},
  "Template": {
    "BasePath": "foobar/templates",
    "Name": "foobar/templates/debug.yaml"
  },
  "Values": {
    "affinity": {},
    "autoscaling": {
      "enabled": false,
      "maxReplicas": 100,
      "minReplicas": 1,
      "targetCPUUtilizationPercentage": 80
    },
    "fullnameOverride": "",
    "httpRoute": {
      "annotations": {},
      "enabled": false,
      "hostnames": [
        "chart-example.local"
      ],
      "parentRefs": [
        {
          "name": "gateway",
          "sectionName": "http"
        }
      ],
      "rules": [
        {
          "matches": [
            {
              "path": {
                "type": "PathPrefix",
                "value": "/headers"
              }
            }
          ]
        }
      ]
    },
    "image": {
      "pullPolicy": "IfNotPresent",
      "repository": "nginx",
      "tag": ""
    },
    "imagePullSecrets": [],
    "ingress": {
      "annotations": {},
      "className": "",
      "enabled": false,
      "hosts": [
        {
          "host": "chart-example.local",
          "paths": [
            {
              "path": "/",
              "pathType": "ImplementationSpecific"
            }
          ]
        }
      ],
      "tls": []
    },
    "livenessProbe": {
      "httpGet": {
        "path": "/",
        "port": "http"
      }
    },
    "nameOverride": "",
    "nodeSelector": {},
    "podAnnotations": {},
    "podLabels": {},
    "podSecurityContext": {},
    "readinessProbe": {
      "httpGet": {
        "path": "/",
        "port": "http"
      }
    },
    "replicaCount": 1,
    "resources": {},
    "securityContext": {},
    "service": {
      "port": 80,
      "type": "ClusterIP"
    },
    "serviceAccount": {
      "annotations": {},
      "automount": true,
      "create": true,
      "name": ""
    },
    "tolerations": [],
    "volumeMounts": [],
    "volumes": []
  }
}

groupby + attrgetter: The Pythonic Way to Group Objects

2025-06-22T00:00:00+00:00

Blog Post Publish Date: 2025/07/28

This blog post explores an approach to group objects by mulitple attribute name powered by capabilities of the built-in modules itertools.groupby and operator.attrgetter module.

To illustrate this scenario, let’s create a Band class, and a bands variable with a list of the instances. The objective is create groups by attributes category, genre, country and print the name.

from dataclasses import dataclass

@dataclass
class Band:
    name: str
    category: str
    genre: str
    country: str

bands = [
    Band(name="The Beatles", category="Rock", genre="Rock", country="UK"),
    Band(name="Queen", category="Rock", genre="Rock", country="UK"),
    Band(name="The Rolling Stones", category="Rock", genre="Rock", country="UK"),
    Band(name="Led Zeppelin", category="Rock", genre="Rock", country="UK"),
    Band(name="The Who", category="Rock", genre="Rock", country="UK"),
    Band(name="Deep Purple", category="Rock", genre="Rock", country="UK"),
    Band(name="Nirvana", category="Rock", genre="Grunge", country="USA"),
    Band(name="Temple of the Dog", category="Rock", genre="Grunge", country="USA"),
    Band(name="Pearl Jam", category="Rock", genre="Grunge", country="USA"),
    Band(name="Soundgarden", category="Rock", genre="Grunge", country="USA"),
    Band(name="Alice in Chains", category="Rock", genre="Grunge", country="USA"),
    Band(name="Stone Temple Pilots", category="Rock", genre="Grunge", country="USA"),
    Band(name="Pink Floyd", category="Rock", genre="Progressive Rock", country="UK"),
    Band(name="Genesis", category="Rock", genre="Progressive Rock", country="UK"),
    Band(name="Yes", category="Rock", genre="Progressive Rock", country="UK"),
    Band(name="King Crimson", category="Rock", genre="Progressive Rock", country="UK"),
    Band(name="The Smiths", category="Rock", genre="Indie Rock", country="UK"),
    Band(name="Arctic Monkeys", category="Rock", genre="Indie Rock", country="UK"),
    Band(name="Pet Shop Boys", category="Pop", genre="Synth-pop", country="UK"),
    Band(name="Depeche Mode", category="Pop", genre="Synth-pop", country="UK"),
    Band(name="Erasure", category="Pop", genre="Synth-pop", country="UK"),
    Band(name="Korine", category="Pop", genre="Synth-pop", country="USA"),
    Band(name="Oasis", category="Rock", genre="Britpop", country="UK"),
    Band(name="Blur", category="Rock", genre="Britpop", country="UK"),
    Band(name="Pulp", category="Rock", genre="Britpop", country="UK"),
    Band(name="Echo & the Bunnymen", category="Rock", genre="Post-Punk", country="UK"),
    Band(name="Joy Division", category="Rock", genre="Post-Punk", country="UK"),
    Band(name="The Cure", category="Rock", genre="Post-Punk", country="UK"),
    Band(name="The Killers", category="Rock", genre="Alternative Rock", country="USA"),
    Band(name="R.E.M.", category="Rock", genre="Alternative Rock", country="USA"),
    Band(name="Radiohead", category="Rock", genre="Alternative Rock", country="UK"),
    Band(name="U2", category="Rock", genre="Rock", country="Ireland"),
] 

The most common approach is to create a temporary dictionary where the key is a tuple of the attributes you want to group by, and the value is a list of objects that share those attributes:

bands_groupby_genre_country = {}

for band in bands:
    group_key = (band.genre, band.country, band.category)
    bands_groupby_genre_country.setdefault(group_key, []).append(band)

for (category, genre, country), band_group in bands_groupby_genre_country.items():
    print(f">>> {category=}, {genre=}, {country=}:")
    for band in band_group:
        print(f"- {band.name}")

This approach works, but the code readability can cause confusion. Therefore, I present a more pythonic way to perform this operation using groupby and attrgetter.

The groupby function takes an iterable as its first positional argument and has an optional named argument key. The key works exactly the same way as in sorted, accepting a function that is called to determine the grouping. In the example below, the function is created using a lambda, where the groups are formed based on the category, genre, and country attributes.

from itertools import groupby

attribute_group = lambda item: (item.category, item.genre, item.country)

for (category, genre, country), group in groupby(bands, key=attribute_group):
    print(f">>> {category=}, {genre=}, {country=}:")
    for band in group:
        print(f"  - {band.name}")

This approach offers better readability. However, by using the attrgetter function, we can eliminate the lambda and make the code more self-explanatory. The attrgetter returns a callable object that fetches the specified attribute from its operand. If more than one attribute is requested, it returns a tuple of attributes. For practical purposes, this solution works the same way as the lambda, but the code is clearer and easier to understand.

from operator import attrgetter
from itertools import groupby

attribute_group = attrgetter("category", "genre", "country",)

for (category, genre, country), group in groupby(bands, key=attribute_group):
    print(f">>> {category=}, {genre=}, {country=}:")
    for band in group:
        print(f"  - {band.name}")

The script execution output will be:

>>> category='Rock', genre='Rock', country='UK':
- The Beatles
- Queen
- The Rolling Stones
- Led Zeppelin
- The Who
- Deep Purple

>>> category='Rock', genre='Grunge', country='USA':
- Nirvana
- Temple of the Dog
- Pearl Jam
- Soundgarden
- Alice in Chains
- Stone Temple Pilots

>>> category='Rock', genre='Progressive Rock', country='UK':
- Pink Floyd
- Genesis
- Yes
- King Crimson

>>> category='Rock', genre='Indie Rock', country='UK':
- The Smiths
- Arctic Monkeys

>>> category='Pop', genre='Synth-pop', country='UK':
- Pet Shop Boys
- Depeche Mode
- Erasure

>>> category='Pop', genre='Synth-pop', country='USA':
- Korine

>>> category='Rock', genre='Britpop', country='UK':
- Oasis
- Blur
- Pulp

>>> category='Rock', genre='Post-Punk', country='UK':
- Echo & the Bunnymen
- Joy Division
- The Cure

>>> category='Rock', genre='Alternative Rock', country='USA':
- The Killers
- R.E.M.

>>> category='Rock', genre='Alternative Rock', country='UK':
- Radiohead

>>> category='Rock', genre='Rock', country='Ireland':
- U2
- Thin Lizzy

For the full working example, visit the GitHub repository. You can also run the code directly in your terminal:

curl -s https://raw.githubusercontent.com/c-neto/my-code-playground/refs/heads/main/blog-code-examples/2025-07-28-groupby/python-groupby-example.py | python -

Kubernetes Logs Simplified: Everything You Need to Know About Pod Logging

2025-06-22T00:00:00+00:00

Blog Post Publish Date: 2025/06/22

Have you ever wondered how $ kubectl logs retrieves logs, where they are stored, and how to access both current and past logs? Kubernetes logging can seem complex, but understanding it is key to troubleshooting and monitoring. This blog post outlines how Kubernetes organizes logs, how to send them to external systems, and tips to manage log growth effectively.

How Kubernetes Manages and Stores Logs

Kubernetes is essentially composed of containers, which are managed by container runtimes. These container runtimes must implement the Container Runtime Interface (CRI), which defines how Kubernetes interacts with the containers. One of the specifications in the CRI is container log handling, which standardizes details such as log names, directory locations, and formats.

CRI Logs Desing Proposal: kubernetes/design-proposals-archive
Kubernetes Logging Reference: https://kubernetes.io/docs/concepts/cluster-administration/logging/

Each container in a pod writes its logs to a separate file. These logs capture everything the application outputs to stdout and stderr (see standard streams). The log files are stored on the node where the pod is running at:

# pod container log file pattern
/var/log/pods/<namespace>_<podname>_<uid>/<container_name>/<execution-id>.log

# example
/var/log/pods/default_nginx-7f5c7d4f9b-abcde_12345/nginx-container/
  ├── 0.log   # current container live running log
  ├── 1.log   # last container log that exited on failure or was restarted
  ├── 2.log   # second to last container log that exited on failure or was restarted
  └── ...

The 0.log file contains the current container’s live running logs. If the container crashes and restarts, Kubernetes renames 0.log to 1.log and creates a new 0.log for the new run. This process continues with 2.log, 3.log, and so on. When you execute kubectl logs --previous <pod> -c <container>, the kubelet searches for logs in 1.log, 2.log, and subsequent files.

To make log ingestion easier for third-party tools, the kubelet creates symbolic links to each container’s current log file (0.log) in the /var/log/containers/ directory. Each symbolic link is named using the pod name, namespace, container name, and container ID, making it easy for log shippers to identify and process logs for each container.

# >>> symbolic link log file name pattern
/var/log/containers/<pod_name>_<namespace>_<container_name>-<container_id>.log

# >>> symbolic link log file name example
/var/log/containers/nginx-7f5c7d4f9b-abcde_default_nginx-container-12345.log

# >>> real file that the symbolic link points to
/var/log/pods/default_nginx-7f5c7d4f9b-abcde_12345/nginx-container/0.log

The $ kubectl logs command always uses the absolute path of the log file and never the symbolic link path. The symbolic link is a mechanism that the kubelet uses to simplify log ingestion for third-party log shipping tools like Fluent Bit, Vector, Filebeat, etc. It is important to understand that the files present in /var/log/containers/ point to the current container execution log (0.log) and do not concatenate the files 0.log, 1.log, 2.log, and so on. When a container crashes and a new 0.log is created, the symbolic link points to the new 0.log file, so the content is overwritten. The log shipping tool should be prepared to handle this behavior. Fluent Bit, for example, uses inode tracking of the symlinked log file. When the file that the symlink points to is rotated, the inode of the file changes as well, and Fluent Bit understands that it is a new log file and needs to read the log content from the start, even though the file name remains the same. For more details, see the Fluent Bit Input Plugin reference.

Understanding Kubernetes Log Format

The log file content is saved in a specific pattern defined by the CRI. This log line format is:

# >>> log line pattern
<timestamp> <stream> <flags> <message>

# >>> log line examples
2023-10-06T00:17:09.669794202Z stdout F Your log message here
2023-10-06T00:17:09.669794202Z stdout P Another log pt 1

timestamp: The date and time when the log line was written, formatted according to RFC3339 with nanosecond precision.
stream: Specifies whether the log line was written to stdout or stderr.
flags: F indicates a completed log line, delimited by \n, while P indicates a partial log line. More details will be explained below.
log message: The container’s log message in its raw format.

About flags: The F flag indicates a complete log line (ending with \n). The P flag signifies that the application has not finished writing the line yet, so the runtime is writing it in parts. When logs are retrieved using kubectl logs, the tool merges these parts together before displaying them to the user.

To understand this better, check the following Python application code:

#!/bin/python

print("part 1 - ", end="")
print("part 2 - ", end="")
print("part 3 - ", end="")
print("log line completed!", end="\n")

The logs are saved in the following format:

2025-06-21T22:15:10.123456789Z stdout P part 1 - 
2025-06-21T22:15:10.123456790Z stdout P part 2 - 
2025-06-21T22:15:10.123456791Z stdout P part 3 - 
2025-06-21T22:15:10.123456792Z stdout F log line completed!

When a pod log is retrieved using $ kubectl logs, the output will concatenate partial logs:

$ kubectl logs -f python-app

part 1 - part 2 - part 3 - log line completed!

Managing Log Growth with Log Rotation

Logs can grow uncontrollably, causing disk overflow, which can impact observability in your cluster and degrade application performance. Even if the application generates a low volume of logs, it is a best practice and highly recommended to enforce limits and implement log rotation routines. The best approach to control log size limits is to define the values using container runtime parameters. It is not recommended to use third-party tools like logrotate for this task, because file operations such as renaming, truncating, and closing file descriptors can corrupt Kubernetes’ log management state. Additionally, the container runtime’s log management architecture may change in future Kubernetes versions, requiring updates to third-party tools to align with the new behavior.

Thus, the configuration defines the Container Runtime’s log rotation process. The example below demonstrates how to configure containerd to manage the log rotation routine.

/etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  LogSizeMax = 104857600  # set log file size limit to 100MB
  LogFileMax = 5          # set limit 5 log files

Also, you can introduce this parameters via environment variables to kubelet service. Below as example using systemd, to define the log rotate parameters for kubelet process to inject log rotate parameters to containerd.

/etc/systemd/system/kubelet.service

Environment="KUBELET_EXTRA_ARGS=--container-log-max-size=100Mi --container-log-max-files=5"

Strategy to Ingest Kubernetes Logs to External Thirdy Party System

For better observability and log search capabilities, it is highly recommended to ingest Kubernetes workload logs into a dedicated log analytics system such as OpenSearch, Elasticsearch, Grafana Loki, Splunk, or Datadog. Within the Kubernetes cluster, specialized tools are optimized for reading, processing, and shipping logs to external systems, such as Fluent Bit, Filebeat, and Vector.

Regardless of which log shipping tool you choose, the deployment pattern remains the same. The log ingestion tool is deployed as a DaemonSet, which schedules one pod per node in the cluster. The DaemonSet is configured with a hostPath volume mount, allowing the pod to access files stored on the node’s disk. This enables the log ingestion tool to read log files for all pod containers scheduled on the node at /var/log/containers/*.log.

Below is an example DaemonSet configuration for deploying Fluent Bit:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: default
  labels:
    app: fluent-bit
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      containers:
        - name: fluent-bit
          image: docker.io/bitnami/fluent-bit:4.0.3-debian-12-r0
          resources:
            limits:
              memory: "200Mi"
              cpu: "100m"
            requests:
              memory: "100Mi"
              cpu: "50m"
          volumeMounts: 
            - name: config
              mountPath: /opt/bitnami/fluent-bit/conf/fluent-bit.conf
              subPath: fluent-bit.conf
            # >>> THIS LINE MOUNTS THE NODE'S POD CONTAINERS DIRECTORY INSIDE THE CONTAINER.
            - name: varlog
              mountPath: /var/log/containers
      volumes:
        - name: config
          configMap:
            name: fluent-bit-config
        # >>> THIS LINE ALLOWS THE FLUENT BIT POD TO ACCESS LOG FILES OF PODS SCHEDULED ON THE NODE. 
        - name: varlog
          hostPath:
            path: /var/log/containers
            type: Directory

Many log collection tools can add extra information to your logs, like pod labels and annotations. To do this, the log collector needs permission to talk to the Kubernetes API. This is usually done by giving the tool a ClusterRole with the right permissions and linking it to the service account the tool uses. For example, you can look at this Fluent Bit ClusterRole to see how it works. With these permissions, the tool can get metadata from Kubernetes and add it to each log before sending it to your log system. Setting up metadata enrichment with Fluent Bit will be explained in a future post.

References

Fluent Bit: Generating Log IDs with SHA-256 Hashes for Log Deduplication

2025-06-19T00:00:00+00:00

Blog Post Publish Date: 2025/06/19

This post explores the root causes of log duplication, the necessity of generating unique log identifiers, and how to leverage Fluent Bit’s native capabilities to create SHA-256 hashes for effective deduplication without relying on external scripts or plugins.

Why Do Duplicate Logs Occur in Fluent Bit?

I encountered issues with duplicated logs being indexed in OpenSearch by Fluent Bit. After some research, I concluded that currently, Fluent Bit, when using the Tail input, offers features designed for the At Least Once delivery strategy. This means that a message may be delivered one or more times, potentially causing duplication. It does not support Exactly Once or At Most Once strategies, where a message is delivered only once or at most once, respectively.

There are several variables that can contribute to this behavior. For example, when using the Tail input plugin, even if you use an SQL database with DB On parameter, you may still experience duplicate logs. This is because the database only persists the file reading offset, not the delivery status for each configured Output.

I plan to write a separate blog post with more details about how Fluent Bit manages log delivery state (chunk, buffering, SQL Lite, Storage Type). For now, it’s important to understand that in certain scenarios such as output unavailability and Fluent Bit restarts—duplication can occur. This is expected behavior and not necessarily a problem, especially when multiple Output configurations are used.

The Importance of Unique Log IDs

If the delivery strategy is At Least Once, generating a unique ID is a hard requirement to avoid duplication at the Output destination. This unique ID should be created based on the log line content. If you are using Logstash and OpenSearch in your log analytics stack, you can generate IDs using the Logstash Fingerprint filter plugin or the OpenSearch Fingerprint Ingest Pipeline Processor, both of which create hashes based on specified log field values. Unfortunately, Fluent Bit does not have a specific native filter plugin to create unique IDs. You can use a Lua script or bash script callback for it, but in both cases, the complexity increases and you become dependent on external crypto libraries. However, don’t worry, I will show you a workaround to generate a hash from the log line content using only Fluent Bit’s native features.

If you are using Fluent Bit to send logs directly to OpenSearch, you can use the OpenSearch Output Plugin with the Generate_ID On parameter enabled to generate a unique ID for logs. However, note that the ID generated by this option will be different even if the one ore more log line content is the same. The Generate_ID On option generates an ID within Fluent Bit only; it does not create a unique ID based on the log content. To avoid duplicate logs, generate a hash-based ID derived from the log content itself.

Generating SHA-256 Hashes for Log Deduplication

I reviewed the available Fluent Bit filter plugins and did not find any that generate hash values based on a log key. However, I discovered a Processor Content Modifier Action called hash. This action takes a log event key as an argument and replaces its value with a SHA-256 hash. While this feature is primarily intended for masking sensitive data such as passwords and PII, it can also be used to generate unique identifiers for log events.

The key insight is to copy the log line content to a new key and hash this new field, keeping the original log line content intact and using the hashed field as the log ID. Check example:

/fluent-bit/fluent-bit.yaml

# The `pipeline.inputs.processors` feature is only available when using the `yaml` configuration format.  
# It is not supported in the legacy `.conf` configuration format.

pipeline:
  inputs:
    - name: dummy
      dummy: |
        {
          "message": "2025-06-19T17:02:35.123456789Z stdout F This is a Foobar application log sample running in kubernetes persisted on /var/log/containers/*.log"
        }
      tag: dummy.foobar
      processors:
        logs:
          # Copy the 'message' field to a new field called '_id'.
          - name: modify
            copy: message _id

          # Hash the '_id' field to anonymize or deduplicate.
          - name: content_modifier
            action: hash
            key: _id

    - name: dummy
      dummy: |
        {
          "message": "2025-06-19T17:02:35.123456789Z stdout F This is a Greeting application log sample running in kubernetes persisted on /var/log/containers/*.log"
        }
      tag: dummy.greeting 
      processors:
        logs:
          # Copy the 'message' field to a new field called '_id'.
          - name: modify
            copy: message _id

          # Hash the '_id' field to anonymize or deduplicate.
          - name: content_modifier
            action: hash
            key: _id

  outputs:
    - name: stdout
      format: json_lines
      match: "dummy.*"

The output is like:

{
  "date": 1750366003.411804,
  "message": "2025-06-19T17:02:35.123456789Z stdout F This is a Foobar application log sample running in kubernetes persisted on /var/log/containers/*.log",
  "_id": "19a93f55808eb5478c65813c028045f1b354abe12790eb8aee0dd825697aa93e"
}
{
  "date": 1750366003.415653,
  "message": "2025-06-19T17:02:35.123456789Z stdout F This is a Greeting application log sample running in kubernetes persisted on /var/log/containers/*.log",
  "_id": "148917c2efe0114da7f7cef6327bde63f5c9ec5cac5cf05d4a73acefaa69a55c"
}
{
  "date": 1750366004.412541,
  "message": "2025-06-19T17:02:35.123456789Z stdout F This is a Foobar application log sample running in kubernetes persisted on /var/log/containers/*.log",
  "_id": "19a93f55808eb5478c65813c028045f1b354abe12790eb8aee0dd825697aa93e"
}
{
  "date": 1750366004.413101,
  "message": "2025-06-19T17:02:35.123456789Z stdout F This is a Greeting application log sample running in kubernetes persisted on /var/log/containers/*.log",
  "_id": "148917c2efe0114da7f7cef6327bde63f5c9ec5cac5cf05d4a73acefaa69a55c"
}

The hash action takes the binary value of the log line, applies the SHA-256 algorithm, and outputs the result as a hexadecimal string. Below is a Python script that performs the same hashing process as Fluent Bit.

import hashlib

log_line_in_string = "2025-06-19T17:02:35.123456789Z stdout F This is a Foobar application log sample running in kubernetes persisted on /var/log/containers/*.log"

log_line_in_bytes = log_line_in_string.encode("utf-8")
log_line_in_sha256_hexa_decimal = hashlib.sha256(log_line_in_bytes).hexdigest()

print(log_line_in_sha256_hexa_decimal)
>>> "19a93f55808eb5478c65813c028045f1b354abe12790eb8aee0dd825697aa93e"

Now, you can use the _id as log unique ID field!

Note

You can find a complete, working example of this configuration in my Docker Compose lab repository: github.com/c-neto/my-devops-labs/blog/2025-06-19/docker-compose.yml

References

Docker-Compose Lab: c-neto/my-devops-labs
Logstash Fingerprint Filter Plugin: https://www.elastic.co/docs/reference/logstash/plugins/plugins-filters-fingerprint
OpenSearch Fingerprint Ingest Pipeline Processor: https://docs.opensearch.org/docs/latest/ingest-pipelines/processors/fingerprint/
Fluent Bit Processors Reference: https://docs.fluentbit.io/manual/pipeline/processors/content-modifier
Fluent Bit Hash Example: https://docs.fluentbit.io/manual/pipeline/processors/content-modifier#hash-example

Helm Charts: Development Practices from a Programmer’s Perspective

2025-02-25T00:00:00+00:00

Blog Post Publish Date: 2025/02/25

In the last quarter of 2024, I delved deep into the world of Helm Charts and encountered several challenges. Initially, creating a new Helm Chart seemed to introduce complexity, and I didn’t see significant advantages over using Kustomize. However, after spending days reflecting, reading more about Helm, and examining existing projects, I gained insights that helped me view Helm as an ally rather than an unnecessarily complex tool. So, I compiled the main insights in this blog post to present Helm chart development tips from a programmer’s perspective.

Why Helm Charts Are a Powerful Tool for IaC

Before discussing best practices for Helm chart development, it is important to understand why Helm Charts are a powerful tool for Infrastructure as Code (IaC). Helm Charts enable the adoption of IaC by providing a robust templating engine for rendering Kubernetes manifests (powered by Go Templates). They allow the creation of reusable components, abstracting Kubernetes workloads, and offering a high-level parameterization interface.

One of Helm’s key advantages is its ability to distribute charts as artifacts, similar to programming language packages or system distributions like RPM and DEB. This allows the creation of new charts that use existing ones as dependencies, which increases agility in developing new solutions based on proven components.

Helm simplifies Kubernetes workload management, streamlining configuration, deployment, and updates. It also supports storing charts as OCI artifacts in registries like AWS Elastic Container Registry (ECR) or as tarball files, ensuring reproducible and seamless deployments. CI/CD tools like Spinnaker and ArgoCD natively support Helm for deployment automation.

Helm Charts are particularly effective for creating reusable, deployable infrastructure components, especially when managing multiple clusters with environment-specific configurations. This approach reduces manifest duplication in Git repositories, improving maintainability and operational efficiency.

Simple Made Easy - Is Helm Chart Overengineering?

“Complexity has to pay for itself. It has to buy you something.”
— Simple Made Easy (Rich Hickey, 2011)

When I first started working with Helm Charts, my initial thought was: “Oh my God, this is so complex, am I just introducing unnecessary complexity?” However, like any new tool, Helm has a learning curve that takes time to overcome. The key to embracing Helm lies in understanding the problem it solves. If you think Helm is complex, try maintaining multiple nearly identical Kubernetes manifests, where the only differences are a few parameters per cluster it’s a recipe for chaos.

In reality, Helm Charts are overengineering only if you’re solving a problem you don’t have. The complexity introduced by Helm justifies itself when it eliminates a bigger problem such as duplicated code across multiple environments. If you’re managing workloads across several clusters, the complexity of Helm is not an overhead but a necessary trade-off to improve maintainability.

I’ll explore this concept further in the next section.

DRY - Eliminating Redundancy in Kubernetes Manifests with Helm

“DRY (Don’t Repeat Yourself) is about the duplication of knowledge, of intent. It’s about expressing the same thing in two different places, possibly in two totally different ways.”
— The Pragmatic Programmer (Andrew Hunt & David Thomas, 1999)

To illustrate the DRY (Don’t Repeat Yourself) principle, let’s consider a common scenario in IT corporations: maintaining isolated environments for different stages of development. Typically, companies have a development environment used internally for testing, a staging environment for validation, and a production environment serving real users.

If you work in this type of setup, you likely manage workloads across multiple Kubernetes clusters, running the same applications with slight variations, such as Ingress settings, resource allocations, and environment-specific variables. A straightforward approach to handling these differences is to create separate Kubernetes manifest files for each environment, maintaining the same structure but modifying specific values. While this method works, it introduces a significant drawback: code duplication.

When managing duplicated manifests, you increase the risk of errors and inconsistencies. This approach may seem manageable with one or two environments, but as the number of environments grows (e.g., 3 or more), it quickly becomes unscalable and difficult to maintain. For instance, applying a patch to one environment requires updating several separate files, increasing operational overhead.

This is where Helm Charts excel. Instead of maintaining multiple duplicate manifests, Helm enables template-driven configuration. The base structure of your Kubernetes manifests resides within the Helm Chart, while environment-specific values are dynamically injected through Go Templates.

Helm achieves this through the Values.yaml file, where each environment has its own configuration file defining runtime parameters. Instead of maintaining duplicated YAML files, you have a single chart that dynamically renders manifests based on input values. This approach ensures consistency, maintainability, and scalability.

As you can see, templatizing Kubernetes manifests is another way Helm follows the DRY principle. By reducing redundancy, Helm minimizes errors and simplifies the management of multiple environments, making Kubernetes deployments more efficient and maintainable.

YAGNI - Avoid Overengineering in Helm Templates

“Always implement things when you actually need them, never when you just foresee that you will need them.”
— Ron Jeffries about XP principle: “You ain’t gonna need it (YAGNI).”

Templates should only be created when they represent a well-defined and reproducible structure. A common mistake when designing Helm Charts is over-templatization, introducing excessive parameterization that makes templates overly flexible but harder to maintain.

If you find yourself defining numerous template functions to handle many variations in your Values.yaml, it’s likely a sign that your template lacks a clear structure. Instead of simplifying deployment, you’re introducing unnecessary complexity that can lead to confusion, harder debugging, and increased maintenance effort.

Even worse than not having a template is maintaining template functions that will never be used. Every extra function, conditional statement, or parameter adds cognitive overhead. When a team inherits a Helm Chart filled with unused or rarely used logic, they are forced to understand and maintain complexity that provides no real value.

This follows the YAGNI (You Ain’t Gonna Need It) principle, a fundamental concept in software development that warns against building features or abstractions that are not immediately necessary. Instead of preparing for hypothetical future scenarios, focus on solving real, present problems efficiently.

Library Charts - Ensuring Reusable Patterns

“The design of reusable libraries should provide clear and simple interfaces, allowing users to interact with them without needing to understand their internal details.” — Design Patterns: Elements of Reusable Object-Oriented Software (paraphrased from the sections on encapsulation and reusability)

There are two types of Helm Charts: Application and Library. Application charts are complete solutions, specifically designed for deployment. They define all the necessary Kubernetes resources required to deploy an application. On the other hand, Library charts contain reusable resources such as templates and functions, which are shared across other charts but are not directly deployed. These library charts act as an abstraction layer, providing common components that can be reused in multiple contexts.

A prime example of a well-designed library chart is the Bitnami Common Chart, which consists of reusable functions and patterns that ensure consistency across all Bitnami charts. These patterns are adopted across the entire Bitnami Charts repository, making it easier to maintain and extend their Kubernetes applications. You can find more about the Bitnami Common Library Chart here, and an example of how the common chart is used in practice within Bitnami Fluentbit can be seen here:

Conclusion

Incorporating best practices from programming concepts such as DRY, YAGNI, and modular design can significantly improve Helm chart development. By focusing on maintainability, scalability, and reducing unnecessary complexity, you can leverage Helm to create more efficient, reusable Kubernetes components. Helm is not just a tool for templating Kubernetes manifests; it’s a framework for thinking about infrastructure in a modular, reusable way—something that every developer can appreciate.

These practices allow teams to scale their Kubernetes deployments more efficiently while minimizing overhead and ensuring that the system is easier to maintain in the long run.

References

Logstash DB Enrichment - Tips and Traps!

2025-02-16T00:00:00+00:00

Blog Post Publish Date: 2025/02/16

In this blog post, I share my experiences with Logstash log enrichment using a database. I’ll cover some hidden behaviors and the importance of configuring the JDBC connection string parameters to avoid mysterious problems that are hard to replicate and not clearly documented. These tips come from real-world scenarios and aren’t always obvious in the official documentation. I hope they help you save hours of troubleshooting and searching through forums.

The examples assume you’re running Logstash OSS 8.17.2 in a Kubernetes cluster with a PostgreSQL database. However, the key considerations apply even if you’re using a single instance or a different relational database. I’ve also created a docker-compose lab for you to explore and test all the tips discussed in this blog post: github.com/c-neto/my-devops-labs/logstash/db-enrichment

I reserve special thanks to PostgreSQL Database Specialist Lucio Chiessi for helping me explore the JDBC connection string options that can be beneficial for Logstash.

Use Prepared Statements

The first tip is to use the use_prepared_statements option in the jdbc_streaming filter plugin. A Prepared Statement in PostgreSQL is a pre-compiled SQL query that improves performance and security by allowing repeated execution with different parameters while preventing SQL injection. When a query is prepared, PostgreSQL analyzes and optimizes the execution plan once. In subsequent calls, only the parameters are replaced, avoiding recompilation and improving efficiency. Normally, the query in the log ingestion pipeline has a defined structure, changing only the parameters, and it is frequently executed. It is an ideal use case for log enrichment.

Example of use_prepared_statements:

jdbc_streaming {
  jdbc_driver_library => "/usr/share/logstash/postgresql.jar"
  jdbc_driver_class => "org.postgresql.Driver"
  jdbc_connection_string => "${DB_JDBC_CONNECTION_STRING}"
  jdbc_user => "${DB_USER}"
  jdbc_password => "${DB_PASSWORD}"
  use_prepared_statements => true
  prepared_statement_name => "logstash_enrich_query"
  prepared_statement_bind_values => [
    "[document][user_id]"
  ]
  statement => "
  SELECT
    user_name,
    user_email,
    user_group
  FROM table_users
  WHERE id = ?
  "
  target => "sql"
}

When the Database is Out, Your Pipeline Won’t Start!

When you configure a Logstash pipeline that uses log enrichment by jdbc_streaming, Logstash verifies the connection with the database before starting to receive the logs. If the database is out, the pipeline will not be registered, and log ingestion will be removed. Pay attention, Logstash verifies the database connection before starting to receive logs, not when receiving logs that will use the jdbc_streaming. In other words, if your database is out, the log ingestion pipeline that has jdbc_streaming will not start, even if all logs don’t satisfy if conditions to be enriched.

However, there is one detail that causes confusion in the behavior mentioned above: If the database is running when the Logstash pipeline starts, and the database connection is broken, the pipeline will continue working with log enrichment errors only. When the database connection is live again, the enrichment will be re-established and work.

It is important to understand because when you configure a database log enrichment, indirectly, you are configuring a hard dependency between Logstash new instances and database health.

It is a problem because if the database is out, it invalidates scaling the Logstash pods, and if Logstash restarts, the log ingestion will be blocked because of the hard dependency on the database.

Ok, but… How to fix this behavior? I didn’t find any parameter in the Logstash documentation to prevent this behavior… In this case, you can organize distinct pipelines in a way that one pipeline is only for logs that will always be enriched, and another pipeline for logs that will not be enriched. This approach can avoid stopping log ingestion of logs that don’t need to be enriched when the database is out. But, this approach can increase the duplicate code. Thus, my suggestion is to use templates of your log pipeline to have a fallback pipeline without log enrichment to avoid rendering the jdbc_streaming statements. If you are working in Kubernetes, I suggest templating the pipeline using Helm; if not, you can use another template engine like Jinja.

Lock Table == Lock Log Ingestion

This is the worst problem that can occur: when issues arise, and service logs are frozen without an explicit indication of the root cause.

When a log is performing the jdbc_streaming filter plugin, and at that moment, the table defined in the query is locked, all logs will be stopped, even logs that will not perform jdbc_streaming. The worst thing is that Logstash service logs don’t generate logs to explicitly indicate this behavior (at least in the default log level).

I explored the documentation of the jdbc_streaming to check parameters that can help avoid this behavior, and I didn’t find any. At this moment, I realized that I needed to explore the PostgreSQL JDBC connection string. I explored https://jdbc.postgresql.org/ and found important things that explain the behavior. By default, if a query execution is on a table that is locked, the timeout for query execution is infinite!

To solve this problem, it is necessary to explicitly set the parameter options=-c lock_timeout=1000 in the JDBC connection string to set the maximum time in milliseconds to wait for a lock on a table. If the lock cannot be acquired within 1 second, an error is thrown in Logstash service explicitly indicating that the log was not enriched due to lock timeout, and the log pipeline will continue, and the log will proceed to the next statement.

To simulate this case, you can execute the following command in PostgreSQL:

LOCK TABLE table_used_by_logstash IN ACCESS EXCLUSIVE MODE;

At this moment, I paid attention to reviewing all JDBC connection parameters with infinite values by default that could cause problems like the one mentioned, and I will present them in the following topics in this blog post.

Slow Query == Slow Log Ingestion

Another JDBC connection parameter that defaults to infinite is the query execution timeout. If the database is overloaded and facing throttling, your log ingestion will be affected. If an enrichment query execution takes a long time, all logs in this pipeline will be queued, similar to the behavior explained in the previous topic. The default connection parameter in PostgreSQL for query execution timeout is infinite, thus, database overload will impact log ingestion.

To solve this case, you can explicitly set the JDBC connection parameter options=-c statement_timeout=5000 to set the maximum time in milliseconds that a query can run before being terminated. In this case, queries running longer than 5 seconds will be aborted, and it will be explicitly logged in Logstash service logs, the pipeline will continue working, and log enrichment will be skipped, adding the tag.

To simulate a slow query, you can add the function pg_sleep. In the example below, pg_sleep(10)::text is introduced to artificially simulate a slow query:

jdbc_streaming {
  jdbc_driver_library => "/usr/share/logstash/postgresql.jar"
  jdbc_driver_class => "org.postgresql.Driver"
  jdbc_connection_string => "${DB_JDBC_CONNECTION_STRING}"
  jdbc_user => "${DB_USER}"
  jdbc_password => "${DB_PASSWORD}"
  use_prepared_statements => true
  prepared_statement_name => "logstash_enrich_query"
  prepared_statement_bind_values => ["[document][user_id]"]
  statement => "
    SELECT
      pg_sleep(10)::text,    
      user_name,
      user_email,
      user_group
    FROM table_users
    WHERE id = ?
  "
  target => "sql"
}

Define Timeouts for Database Login and Socket

In the same context as the previously presented timeout parameters, it is important to define loginTimeout=10 to specify the maximum time in seconds to wait for a connection to be established. If the connection cannot be established within this time, an error is thrown, and socketTimeout=10 to set the maximum time in seconds for reading data from the database. If the database does not respond within this time, the connection is closed.

Ensure DB Connection Identification

The Logstash enrichment queries can be offensive to the database. In this case, it is necessary to tag the connections to make it easier to identify in the aspect of database administration. To define the connection identification, use the JDBC Connection String parameter ApplicationName=logstash to set the application name to logstash for easier identification in PostgreSQL logs.

Thus, you can identify the connection from Logstash by the following PostgreSQL query:

SELECT pid, application_name, usename, datname, client_addr, state
FROM pg_stat_activity
WHERE application_name = 'logstash';

Sequel Opts != Database Connection Parameters

Don’t be confused, the jdbc_streaming::sequel_opts parameters and JDBC Connection String performing distinct roles. Logstash connects to databases using JDBC, but it internally manages the connection through the Sequel library in JRuby, allowing for additional configuration via sequel_opts. Thus, the JDBC connection string defines behavior with the database, and Sequel defines connection characteristics to manage this connection.

The example below is ideal to explain it. The sequel_opts::pool_timeout defines a timeout for Logstash to wait for an available pool connection to use, which is not related to the timeout of query execution, which is the parameter related to the database, defined in the jdbc_connection_string.

jdbc_streaming {
  sequel_opts => {
    max_connections => 4
    pool_timeout => 5
  }
  jdbc_connection_string => "jdbc:postgresql://postgres:5432/db?options=-c%20statement_timeout=1000%20"
}

Conclusion

When working with Logstash, I initially assumed that its behaviors were straightforward and default. However, Logstash has many nuances that can impact log ingestion and even halt it.

As requirements grow, so do the challenges. Nothing should be taken for granted, and everything needs to be validated. I highly recommend setting up a docker-compose lab to test features in various contexts. This ensures reliability, helps plan workarounds, and prevents unexpected issues, especially late on a Friday night.

Don’t forget to check out the docker-compose lab: github.com/c-neto/my-devops-labs/logstash/db-enrichment

References

OpenSearch Concepts - Index and Shards

2025-01-05T00:00:00+00:00

Blog Post Publish Date: 2025/01/05

OpenSearch is a powerful platform, but having a solid understanding of its basic concepts is essential for a seamless experience with the stack. In this blog post, we’ll cover one of the most fundamental aspects of OpenSearch administration: Indices and Shards.

This blog post provides a high-level explanation of these concepts and includes useful commands for retrieving information about indices and shards helpful during troubleshooting.

Understanding Indices and Shards

An index is a logical namespace that organizes a collection of documents along with their associated characteristics, such as field types, life cycle policies, and storage layer state. These documents are stored on disk within logical partitions called shards. Shards can be classified as primary or replica:

Primary shards handle write operations and can also serve read requests when needed;
Replica shards are exact copies of primary shards, designed to ensure fault tolerance and improve query performance by handling read operations.

An index can consist of one or more shards distributed across multiple nodes. This distributed nature enables OpenSearch to form clusters, allowing indices to span across different machines for improved scalability and reliability.

Increasing the number of primary shards improves throughput and indexing performance by distributing data across multiple partitions. While increasing the number of replica shards enhances fault tolerance and boosts query performance by providing additional nodes to handle read operations.

The number of primary shards can only be defined during the creation of an index. Carefully consider the number of shards needed to meet your current requirements, factoring in query demands and, most importantly, write performance. Replica shards, however, can be adjusted on demand.

Note

A primary shard and its corresponding replica cannot reside on the same node within a cluster, ensuring fault tolerance in case of node failure.

Useful Commands for Managing Indices and Shards Using the OpenSearch API

The following commands are especially useful for troubleshooting and monitoring the status of indices and shards. They can be executed in OpenSearch Dashboards » Dev Tools or via an HTTP client like cURL or Postman.

List all indices, sorted by name:

GET _cat/indices?v&s=index

List the shards composing a specific index:

GET <INDEX-NAME>/_search_shards

Get index metadata (e.g., number of shards, replicas, creation time):

GET <INDEX-NAME>/_settings

Retrieve field types defined in the index mapping:

GET <INDEX-NAME>/_mapping

List the ISM Policy attached to an index during creation:

GET _plugins/ism/explain/<INDEX-NAME>?show_policy=true

List shards not currently allocated to any node (status: UNASSIGNED):

GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason&s=state

Check shard allocation across cluster nodes:

GET _cat/allocation?v

Monitor shard recovery or relocation progress:

GET _cat/recovery/<INDEX-NAME>?v&s=stage:desc

List all shards, sorted by index:

GET _cat/shards?v&s=index

List indices, sorted by size:

GET _cat/indices?v&h=index,store.size&s=store.size:desc

With these commands you can efficiently get useful information about your OpenSearch clusters and streamline troubleshooting processes. If you have any questions or need further insights into OpenSearch, feel free to leave a comment below!

Dynaconf: The Python Config Silver Bullet!

2024-07-21T00:00:00+00:00

Blog Post Publish Date: 2024/07/21

This blog post outlines a brief overview of best practices for application configuration, explains the main features of Dynaconf, and provides a hands-on example of how to extend it to retrieve parameters from AWS Parameter Store.

Twelve-Factor App: Good Practices in Application Config

Before starting to talk about Dynaconf or any code implementation for configuration, it’s necessary to understand the best practices used behind the scenes in systems with portable running requirements.

One of the main best practices references is the Twelve-Factor App, which is a methodology with a bundle of good practices for building modern applications with portable requirements, such as applications running on SaaS platforms and Kubernetes clusters. There is a chapter dedicated to application configuration: 12 Factor » Config.

In summary, the config chapter explains what (and what not) categorizes an application parameter, and emphasizes the separation of configuration from code.

Python Config Libraries

Like almost all recurring generic needs and well-defined best practices, there are modules and frameworks to help us implement them. In Python, several tools can help achieve these best practices. I have used some nice tools like python-dotenv, which focuses on reading simple key-value pairs from a .env file and can set them as environment variables, and python-decouple, which provides a dynamic settings interface to help change parameters without having to redeploy your app (useful for web applications). Additionally, there are modules for parsing structured files that can be used as config files, such as PyYAML, json, tomllib, shlex, and configparser.

These modules are really nice and have specific use cases, especially when you can use a native approach and simple programs like routine scripts. But, with the provocative blog post title, I will present Dynaconf.

Dynaconf!

Dynaconf is a robust configuration management for Python, designed to follow Twelve-Factor App config principles. It serves as an abstraction layer between code and settings, providing a simple interface to retrieve parameters and secrets from files or external system sources.

The highlight features include support for multiple config files in various formats (toml, yaml, json, ini), parameters via environment variables, and multi-hierarchical parameter set profiles support (e.g., default, development, production). Additionally, there is a CLI that is useful for debugging.

A notable feature is the capability to retrieve parameters from external parameter/secrets storage systems. There is built-in support for HashiCorp Vault and Redis, but it is also possible to extend it to retrieve parameters from any source based on your needs. This feature is really awesome! I will show an example of how to use it to retrieve parameters from AWS Parameter Store.

Talk is cheap, Show me the code!

To exemplify how to use Dynaconf, I will show a step-by-step guide on how to configure and use it from scratch. The example covers how to set up multiple settings files and how to extend the parameter source to retrieve parameters from the AWS Parameter Store service (simulated by LocalStack in a Docker container).

Note

All code is in the following GitHub repository: github.com/c-neto/my-code-playground/blog-code-examples/2024-07-21-dynaconf-loaders

Step-by-step

Create a virtual environment and install dependencies:

python3 -m venv venv
venv/bin/pip install dynaconf boto3 
source venv/bin/activate

Use the dynaconf-cli to speed up the settings and config.py files creation:

dynaconf init \
    -v PARAMETER_FROM_SETTINGS_TOML_FILE="foobar from settings.toml" \
    -s PARAMETER_FROM_SECRETS_TOML_FILE="foobar from .secrets.toml"

# » output
#
# ├── config.py       » python module with dynaconf instance
# ├── .secrets.toml   » settings files with: PARAMETER_FROM_SECRETS_TOML_FILE
# └── settings.toml   » settings files with: PARAMETER_FROM_SETTINGS_TOML_FILE

The config.py source code created by dynaconf-cli looks like this:

config.py

from dynaconf import Dynaconf

settings = Dynaconf(
    envvar_prefix="DYNACONF",
    settings_files=['settings.toml', '.secrets.toml'],
)

You can use the dynaconf-cli to list parameters based on the configuration defined in config.py:

dynaconf -i config.settings list 

# » output
#
# PARAMETER_FROM_SETTINGS_TOML_FILE<str> 'foobar from settings.toml'
# PARAMETER_FROM_SECRETS_TOML_FILE<str> 'foobar from .secrets.toml'

Run LocalStack in a container to simulate the AWS Parameter Store service:

docker run --rm -it --detach \
    --name localstack \
    -p 4566:4566 \
    -e SERVICES=ssm,sts \
    -e DEFAULT_REGION=us-east-1 \
    -e AWS_ACCESS_KEY_ID=test \
    -e AWS_SECRET_ACCESS_KEY=test \
    localstack/localstack:latest

With LocalStack running, export the AWS environment variables to access the LocalStack API.

export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_ENDPOINT_URL=http://localhost:4566
export AWS_DEFAULT_REGION=us-east-1

Create two parameters prefixed by /my-application/ named PARAMETER_FROM_AWS_PARAMETER_STORE_1 and PARAMETER_FROM_AWS_PARAMETER_STORE_2.

aws ssm put-parameter \
    --name "/my-application/PARAMETER_FROM_AWS_PARAMETER_STORE_1" \
    --value "foo" \
    --type "SecureString" \
    --endpoint-url http://localhost:4566

aws ssm put-parameter \
    --name "/my-application/PARAMETER_FROM_AWS_PARAMETER_STORE_2" \
    --value "bar" \
    --type "SecureString" \
    --endpoint-url http://localhost:4566

Edit the conf.py to add the custom loaders to retrieve the parameters from AWS Parameter Store. All parameters within the /my-application/ namespace will be retrieved

config.py

import logging

import boto3
from botocore.exceptions import ClientError, BotoCoreError
from dynaconf import Dynaconf, LazySettings
from dynaconf.loaders.base import SourceMetadata

logger = logging.getLogger(__name__)
PARAMETERS_PREFIX = "/my-application/"


# Retrieve parameters from AWS Parameter Store recursively by prefix
def _aws_parameter_store_loader(parameters_prefix: str) -> dict:
    ssm_client = boto3.client('ssm')

    paginator = ssm_client.get_paginator("get_parameters_by_path")
    pages = paginator.paginate(Path=parameters_prefix, Recursive=True, WithDecryption=True)

    parameters_from_aws_raw = {}

    for page in pages:
        for param in page["Parameters"]:
            name_full = param["Name"]
            value = param["Value"]
            parameters_from_aws_raw[name_full] = value

    parameters_from_aws = {}

    for name_full, value in parameters_from_aws_raw.items():
        try:
            *_, name = name_full.split("/")
        except ValueError:
            logger.warning(f'Parameter name pattern outside of expected format, skipping: "{name_full}"')
        else:
            parameters_from_aws[name] = value

    return parameters_from_aws


# Dynaconf callback - used in this case for retrieve parameters from AWS Parameter Store
def load(obj: LazySettings, env: str, silent: bool = True, key: str = None, filename: str = None) -> None:
    global parameters_from_aws

    source_metadata = SourceMetadata('aws-parameter-store', PARAMETERS_PREFIX)
    obj.update(parameters_from_aws, loader_identifier=source_metadata)
    logger.debug(f"Parameters loaded from AWS Parameter Store: {list(parameters_from_aws)}")


# Attempt to load parameters from AWS Parameter Store and update Dynaconf loaders
try:
    parameters_from_aws = _aws_parameter_store_loader(PARAMETERS_PREFIX)
except (BotoCoreError, ClientError) as e:
    LOADERS_FOR_DYNACONF = ["dynaconf.loaders.env_loader"]
    logger.warning("AWS active session not detected, skipping load parameters from AWS Parameter Store")
else:
    # __name__ is module name that have "load()" callback. It could be another module...
    LOADERS_FOR_DYNACONF = [__name__, "dynaconf.loaders.env_loader"]


# Initialize Dynaconf settings with specified files and loaders
settings = Dynaconf(
    settings_files=["settings.toml", ".secrets.toml"],
    envvar_prefix="MY_APP",
    LOADERS_FOR_DYNACONF=LOADERS_FOR_DYNACONF,
)

Check the variables and note the PARAMETER_FROM_AWS_PARAMETER_STORE_1 and PARAMETER_FROM_AWS_PARAMETER_STORE_2 that were retrieved from AWS Parameter Store.

dynaconf -i config.settings list

# » output
#
# PARAMETER_FROM_SETTINGS_TOML_FILE<str> 'foobar from settings.toml'
# PARAMETER_FROM_SECRETS_TOML_FILE<str> 'foobar from .secrets.toml'
# PARAMETER_FROM_AWS_PARAMETER_STORE_1<str> 'foo'
# PARAMETER_FROM_AWS_PARAMETER_STORE_2<str> 'bar'

The environment variables take precedence over other settings sources, following the Unix philosophy. To test this behavior, export MY_APP_PARAMETER_FROM_AWS_PARAMETER_STORE_2.

export MY_APP_PARAMETER_FROM_AWS_PARAMETER_STORE_2="param from environment variable"

Dynaconf will retrieve only variables that are prefixed by the value defined in the envvar_prefix instance argument; in this case, MY_APP.

Now, when PARAMETER_FROM_AWS_PARAMETER_STORE_2 is accessed, the value will be the one defined in the environment variable.

dynaconf -i config.settings list

# » output
# 
# PARAMETER_FROM_SETTINGS_TOML_FILE<str> 'foobar from settings.toml'
# PARAMETER_FROM_SECRETS_TOML_FILE<str> 'foobar from .secrets.toml'
# PARAMETER_FROM_AWS_PARAMETER_STORE_1<str> 'foo'
# PARAMETER_FROM_AWS_PARAMETER_STORE_2<str> 'param from environment variable'

In addition, you can peform inspect action to check the parameters to ensure source of each them.

dynaconf -i config.settings inspect

# » output
# 
# {
#   "header": {
#     "env_filter": "None",
#     "key_filter": "None",
#     "new_first": "True",
#     "history_limit": "None",
#     "include_internal": "False"
#   },
#   "current": {
#     "PARAMETER_FROM_SETTINGS_TOML_FILE": "foobar from settings.toml",
#     "PARAMETER_FROM_SECRETS_TOML_FILE": "foobar from .secrets.toml",
#     "PARAMETER_FROM_AWS_PARAMETER_STORE_1": "foo",
#     "PARAMETER_FROM_AWS_PARAMETER_STORE_2": "bar"
#   },
#   "history": [
#     {
#       "loader": "aws-parameter-store",
#       "identifier": "/my-application/",
#       "env": "global",
#       "merged": false,
#       "value": {
#         "PARAMETER_FROM_AWS_PARAMETER_STORE_1": "foo",
#         "PARAMETER_FROM_AWS_PARAMETER_STORE_2": "bar"
#       }
#     },
#     {
#       "loader": "toml",
#       "identifier": ".secrets.toml",
#       "env": "default",
#       "merged": false,
#       "value": {
#         "PARAMETER_FROM_SECRETS_TOML_FILE": "foobar from .secrets.toml"
#       }
#     },
#     {
#       "loader": "toml",
#       "identifier": "settings.toml",
#       "env": "default",
#       "merged": false,
#       "value": {
#         "PARAMETER_FROM_SETTINGS_TOML_FILE": "foobar from settings.toml"
#       }
#     },
#     {
#       "loader": "set_method",
#       "identifier": "settings_module_method",
#       "env": "global",
#       "merged": false,
#       "value": {
#         "SETTINGS_MODULE": [
#           "settings.toml",
#           ".secrets.toml"
#         ]
#       }
#     },
#     {
#       "loader": "set_method",
#       "identifier": "init_kwargs",
#       "env": "global",
#       "merged": false,
#       "value": {
#         "LOADERS_FOR_DYNACONF": [
#           "config",
#           "dynaconf.loaders.env_loader"
#         ],
#         "SETTINGS_FILE_FOR_DYNACONF": [
#           "settings.toml",
#           ".secrets.toml"
#         ],
#         "ENVVAR_PREFIX_FOR_DYNACONF": "MY_APP"
#       }
#     }
#   ]
# }

Conclusion (Author Opinion)

I have been using Dynaconf around 4 years, and I really appreciate how the module is designed to address recurring challenges in a simple and straightforward manner. Previously, I used PyYAML, but I frequently created repetitive code when reading configurations and secrets from distinct files. When a mix of input parameter sources was required, such as YAML and environment variables, the code complexity increased, diverting attention from the core problem my systems aimed to solve.

When I first tried Dynaconf, I was impressed by its ability to manage multiple config files while merging the parameters and sub-parameters structure. This feature is very useful to me, especially for separating sensitive information into specific files.

Now, I only avoid using Dynaconf in cases where there is a Software Requirement to use only built-in modules. Otherwise, I don’t see a reason not to use it, especially in applications designed to run in Kubernetes.

I really like the module and suggest you give it a try!

Logstash Sniffing: Improving Log Ingestion in OpenSearch Scalable Clusters

2024-06-08T00:00:00+00:00

Blog Post Publish Date: 2024/06/08

This blog post outlines an optimize log ingestion in OpenSearch using Logstash. It will address a common issue related to shipping logs to scalable OpenSearch clusters and provide an effective solution to enhance resource utilization.

The Problem - Unbalanced Node Utilization in OpenSearch Cluster

When managing an OpenSearch cluster with scalable Ingest Nodes accessed through a load balancer and using Logstash to ingest logs, the Elasticsearch/OpenSearch output plugin requires customization to optimally utilize the available OpenSearch resources.

Although log shipping to OpenSearch is stateless over the HTTP protocol, Logstash establishes a fixed connection with a specific OpenSearch ingest node. When a load balancer selects a node, Logstash will keep to send all logs to this node as long as the connection remains active. This can overwhelm the selected node, leaving other Ingest Nodes underutilized. Additionally, if you use average resource metrics for scaling Ingest Nodes with Horizontal Pod Autoscaling (HPA), this imbalance can impact performance.

The Solution - Utilizing OpenSearch API for Dynamic Hosts Registration

The hosts parameter in the OpenSearch output plugin can be configured with a single address or multiple addresses. Using multiple addresses allows Logstash to manage and balance log shipping across declared hosts. For non-scalable Ingest Nodes, specifying multiple hosts is a solution of the issue described above. However, for scalable Ingest Nodes, this static approach is insufficient due to the dynamic nature of node scaling.

To address this, the OpenSearch output plugin offers sniffing parameters to dynamically retrieve the current OpenSearch nodes addresses from the OpenSearch API and update the hosts configuration accordingly.

Consider the following example:

output {
  opensearch {
    hosts           => "https://opensearch-load-balancer:9200"
    sniffing        => true
    sniffing_delay  => 30
    sniffing_path   => "/_nodes/ingest:true"
    ...
  }
}

When Logstash starts, it sends a request to the OpenSearch API endpoint /_nodes/ingest:true to retrieve the addresses of all Ingest Nodes. These addresses are dynamically added and removed from the hosts list. Every 30 seconds, the node pool addresses are updated. The Load Balancer address https://opensearch-load-balancer:9200 is used only to fetch the IP addresses.

You can specify other OpenSearch node types using the sniffing_path parameter. For instance, to use Data nodes, set the parameter to /_nodes/data:true.

Conclusion (Author Opinion)

This approach ensures better traffic distribution to OpenSearch, significantly reduces load balancer traffic, eliminates an extra network hop, and enhances resource usage, thereby improving Horizontal Pod Autoscaling in OpenSearch.

This solution is so effective and straightforward that I’m beginning to miss its presence in other log shipping tools like Fluentbit.

I have been using this approach and the results are really nice!

fzf: Life is Too Short for Pipe Grep

2024-04-21T00:00:00+00:00

Blog Post Publish Date: 2024/04/28

This blog post outlines the advantages of the fzf (Fuzzy Finder CLI) and explains how to configure its Zsh widgets.

Searching in the Terminal - Tedious Task

Searching for items in the terminal is a trivial activity, but can be a boring and tedious task. I will list three cases to exemplify it: searching command history; files and directories; and command output.

Usually, users press the arrow up/down to recall last commands executed. The most used shells implemented an interactive search tool called by CTRL + R shortcut. In Zsh, it is implemented as reverse-i-search, which works well, but you need to press the shortcut again to see the next results. This behavior can be a problem if you need to recall commands executed multiple times with different arguments.

When it is necessary to find some file or directory, the most common method adopted is to use $ find. It works well when you know exactly the item that you need to find, but not so good when you only know some partials of the path.

Lastly, I can mention searching elements in command outputs. The most adopted method for this case is to use matching expressions within a command output using $ command | grep. When the same problem of finding files and directories previously described is true for this. Depending on the case, it is necessary to execute grep multiple times until you find the elements that you need.

This tedious task prompted me to research better alternatives for interactive searching.

Fuzzy Finders - The Solution

After a few days of research, I found an amazing project that solves my problems: fzf — command-line fuzzy finder. It is a cross-platform Fuzzy Finder command-line written in Go. But, what is Fuzzy Finder?

Fuzzy Finder is a search tool category for providing a quickly and flexibly way to find files, directories, or other elements, even when they don’t remember the precise names. It employs fuzzy matching to find results based on partial words, erratic characters, or typographical errors.

fzf filters the list of items as you type. This makes your searching much more efficient, avoiding $ command | grep execution. It can receive output of the other commands for interative searching, for instance you can execute $ kgp -A | fzf to find pods in a Kubernetes cluster. In addition, it has some predefined shells widgets to search files, directories, and search history commands.

The next section will explain how to configure these widgets in Zsh.

There are widgets available for Fish, Bash, Zsh. You can check the available implementations in github.com/junegunn/fzf/shell

How to Configure fzf Widgets in Zsh

The first step is to install fzf. There are some distinct ways to install it described in fzf — installation section. Depending on the package manager on your workstation, the version available can be older. Thus, I will download and install the latest version available in GitHub Releases section.

setopt INTERACTIVE_COMMENTS

# download the latest version (in a moment of the blog publish date, the latest version is 0.50.0)
curl -L https://github.com/junegunn/fzf/releases/latest/download/fzf-0.50.0-linux_amd64.tar.gz > fzf-0.50.0-linux_amd64.tar.gz

# uncompress the donwloaded file to access the fzf binary
tar -xzvf fzf-0.50.0-linux_amd64.tar.gz

# check fzf binary execution
./fzf --help

# move the binary to a folder present in your path
mv fzf /usr/local/bin/

The fzf use the command $ find behind the scenes as engine to search directories and files. You can change it. In this example, I will install and configure the fd-find, a fast and user-friendly alternative to $ find.

# In Linux workstation with dnf package manager
dnf install fd-find

# In MacOs workstation with homebrew package manager
brew install fd

For file content preview, I will install and configure bat, a $ cat clone with syntax highlighting.

# In Linux workstation with dnf package manager
dnf install bat

# In MacOs workstation with homebrew package manager
brew install bat

For directory and sub directories preview, I will install and configure tree, a utility which recursively displays the contents of directories in a tree-like format.

# In Linux workstation with dnf package manager
dnf install tree

# In MacOs workstation with homebrew package manager
brew install tree

Now, it is necessary load the Shell widgets. For this, you can get the widget setup source-code given the shell name as argument (--zsh, --fish, --bash). In this case, I will setup the widgets for Zsh.

# Set up fzf key bindings and fuzzy completion (only available in 0.48.0 or later)
eval "$(fzf --zsh)"

Run the following command to check the keybindings are correctly loaded.

bindkey -a | grep fzf

# output expected:
#
# "^R" fzf-history-widget  # CTRL + R: reverse history search
# "^T" fzf-file-widget     # CTRL + T: search files
# "^[c" fzf-cd-widget      # ALT  + C: search directories

You can customize the widgets fzf parameters through environment variables described in fzf — environment variables. Each widget has its own environments variables you can customize based on your needs. Take a look at following code block for an example of the how to parametrize fzf widgets.

# fzf parameters used in all widgets - configure layout and wrapped the preview results (useful in large command rendering)
export FZF_DEFAULT_OPTS="--height 100% --layout reverse --preview-window=wrap"

# CTRL + R: put the selected history command in the preview window - "{}" will be replaced by item selected in fzf execution runtime
export FZF_CTRL_R_OPTS="--preview 'echo {}'"

# ALT + C: set "fd-find" as directory search engine instead of "find" and exclude "venv|virtualenv|.git" of the results during searching
export FZF_ALT_C_COMMAND="fd --type directory --exclude venv --exclude virtualenv --exclude .git"

# ALT + C: put the tree command output based on item selected
export FZF_ALT_C_OPTS="--preview 'tree -C {}'"

# CTRL + T: set "fd-find" as search engine instead of "find" and exclude "venv|virtualenv|.git" for the results
export FZF_CTRL_T_COMMAND="fd --exclude venv --exclude virtualenv --exclude .git"

# CTRL + T: put the file content if item select is a file, or put tree command output if item selected is directory
export FZF_CTRL_T_OPTS="--preview '[ -d {} ] && tree -C {} || bat --color=always --style=numbers {}'"

The next section will present widgets executions preview.

You can check my ~/.zshrc file content in the following link. It contains my personal fzf parametrization: c-neto/ansible-configure-fedora

Results

fzf-history-widget executed by CTRL + R.

fzf-cd-widget executed by ALT + C.

fzf-file-widget executed by CTRL + T.

Searching pods in Kubernetes Cluster with $ kgp -A | fzf execution.

Conclusion (Author Opinion)

My research for improvements in searching history commands has yielded results that exceed my expectations. fzf opened my mind to understand what is Fuzzy Finder and it purpose. It not limited to be command history searching only, it is a tool that you can search anything. It is really nice!

Certainly, I think it is a much better alternative to command history search widget default in Zsh, Bash, and Fish (which already has a good history search widget).

The interactive searching are blazingly fast and customization expanding the possibilities based on your needs. The productivity which fzf provides is really awesome! In a fact, that its justifies more than 59K stars in your GitHub repo.

I approve and recommend!

ZSH + Starship: A Productivity Masterpiece

2024-02-08T00:00:00+00:00

Blog Post Publish Date: 2024/02/08

This blog post covers my prompt customization experience, favorite ZSH Plugins, ZSH options, and Starship configuration. Finally, I introduce a simple guide to configure my custom theme from scratch.

My Prompts Customization Experiences

Productivity is a topic that I definitely like. For long time, I used the raw Terminal over Bash, It is force me to memorize the commands, but the productivity is not so good. I felt that I need to improve my Terminal setup to focus in productivity.

xonsh: Python + Bash Advantages (and Disadvantages)

Due to my expertise in Python, I have been testing xonsh for approximately six months. The xonsh is a superset of Python that includes additional shell primitives familiar to Bash users. The combination of Bash and Python enhances productivity by blending the simplicity of Python syntax with the versatility of Bash. However, debugging can be challenging (imagine a list comprehension in Python syntax combined with dictionaries and environment variables in Bash syntax).

zsh: Extensible and Product Ready (If Customized)

I needed come back the real world again, then I gave a chance for ZSH, a advanced and highly customizable command-line interface with enhanced productivity features and a large Plugins. It greatly appealed to me, mainly when I tested the Spaceship a “minimalistic, powerful and extremely customizable ZSH prompt”, and the oh-my-zsh framework that provides a simple way to manage Plugins and Themes. I believed I had reached a stable terminal configuration with a good balance between productivity and simplicity, but in the MacOs workstation, the input and startup delay of the commands got on my nerves.

I searched improvements tips in the Blog Posts and Youtube videos, but the result don’t satisfied myself. Then, I opted to install the Plugins manually avoiding the oh-my-zsh, and I tested the Starship (in my words: “a Spaceship alternative, in Rust, blazing-fast, with many stars in the GitHub”).

The results were astonishing. I didn’t have to put in any effort to transition my theme configuration from Spaceship to Starship, and the performance is exactly as I expected.

fish: Ready to Use (But non-POSIX Compliant)

In the past few days, I tried out Fish, described as “a smart and user-friendly command line shell”. I was impressed by its performance and the built-in features in the default setup. Starship also supports Fish, earning some positive points from me. I noticed better performance compared to ZSH, and the default configuration already includes features like dynamic syntax checking with color indicators, dynamic autocompletion based on the history file, and menu selection for command options and arguments. It’s simple and Ready to Use.

However, there’s a drawback: Fish is not compliant with the POSIX sh definition. For instance, declaring variables using $ FOOBAR=123 won’t work, instead, you should use the correct syntax like $ set FOOBAR 123. This lack of compliance with POSIX is a downside for me, as it rules out the possibility of using Fish in my daily work. Nevertheless, if ZSH is not an option, I’d still be willing to use Fish without any issues.

My ZSH Favorite Plugins

There are numerous plugins available for ZSH. Typically, these plugins are installed and managed by oh-my-zsh, but you can install them directly, once these plugins are essentially script files with a predefined set of functions and routines.

I will present in the following sub-topics, my indispensable ZSH Plugins.

zsh-autosuggestions: Dynamic Assistant to Remember Commands

No doubt, it is my favorite plugin. This plugin emulates the behavior of the Fish shell, suggesting command completions dynamically based on history.

More Details: zsh-users/zsh-autosuggestions

zsh-syntax-highlighting: Valid Commands by Color

Another Plugin which emulate the Fish shell. When a command is correctly typed, the color turns green. Otherwise, it changes to red. This feature is handy for detecting typos in a dynamic way

More Details: zsh-users/zsh-syntax-highlighting

kubectl: k alias for kubectl

This plugin adds completion for the Kubernetes, as well as some aliases for common kubectl commands.

For example:

# shortcut for: kubectl get pods
$ kgp

# shortcut for: kubectl delete cm foobar-config-map
$ kdelcm foobar-config-map

# shortcut for: kubectl get cronjob
$ k get cj

More Details: ohmyzsh/ohmyzsh/plugins/kubectl

My Starship Configuration

The simplicity is elegance. I customized my Starship prompt with only attributes of the tools important for me.

aws: Only the profile name of the my AWS session;
git: Only the Branch, no more info;
python: The current activated virtualenv;
kubernetes: The Kubernetes context name. Based on the name, I applied the different styles to explicit the study, tests, and production workloads.

Check my ~/.config/starship.toml entire content: c-neto/ansible-configure-fedora/files/dotfiles/starship.toml

My favorite ZSH options (setopt)

Some usually features (like comments, history file) from other shells like Bash or Fish must be explicit enabled in ZSH. These features are enabled by $ setopt command in ~/.zshrc.

You can get all options available in the $ zle -al command execution or in the following ZSH Options Reference.

Check the ZSH Options I have enabled:

INTERACTIVE_COMMENTS: Enable comments “#” expressions in the prompt shell;
APPEND_HISTORY: Append new history entries to the history file;
INC_APPEND_HISTORY: Save each command to the history file as soon as it is executed;
HIST_IGNORE_DUPS: Ignore recording duplicate consecutive commands in the history (improve the performance of the zsh-autosuggestions);
HIST_IGNORE_SPACE: Ignore commands that start with a space in the history;

Check my ~/.zshrc entire content: c-neto/ansible-configure-fedora/files/dotfiles/.zshrc

How to Setup My ZSH + Starship From Scratch

You can check my dotfiles in my GitHub: c-neto/ansible-configure-fedora/files/dotfiles/

1 optional: Save a backup of your current dotfiles (~/.zshrc and ~/.zsh_history):

$ cp "$HOME/.zshrc" "$HOME/.zshrc.backup"
$ cp "$HOME/.zsh_history" "$HOME/.zsh_history.backup"

2: Install the ZSH with your package manager.

# if Linux (Fedora)
$ dnf install zsh

# if MacOS
$ brew install zsh

3: Install the Starship:

$ curl -sS https://starship.rs/install.sh | sh

About Nerd Fonts:

The ~/.config/starship.toml file (which will be downloaded in step 5) does not necessitate Nerd Fonts. However, if you encounter issues with emoji rendering, consider the installation.

4: Create a directory in your home to save the ZSH plugins and the Starship configuration:

$ mkdir "$HOME/.config/"
$ mkdir "$HOME/.my-custom-zsh/"

5: Download my ~/.config/starship.toml:

$ curl 'https://raw.githubusercontent.com/c-neto/ansible-configure-fedora/main/files/dotfiles/starship.toml' > "$HOME/.config/starship.toml"

6: Install the plugins manually:

# k alias
$ curl 'https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/plugins/kubectl/kubectl.plugin.zsh' > "$HOME/.my-custom-zsh/kubectl.plugin.zsh"

# zsh-syntax-highlighting
$ git clone --depth 1 'https://github.com/zsh-users/zsh-syntax-highlighting.git' "$HOME/.my-custom-zsh/zsh-syntax-highlighting"

# zsh-autosuggestions
$ git clone --depth 1 'https://github.com/zsh-users/zsh-autosuggestions' "$HOME/.my-custom-zsh/zsh-autosuggestions"

7: Download my ~/.zshrc:

$ curl 'https://raw.githubusercontent.com/c-neto/ansible-configure-fedora/main/files/dotfiles/.zshrc' > "$HOME/.zshrc"

Key Binding Tip

You can customize the bindkey values according to your preferences. To find the code for your keys, run $ cat -v and press the desired key; the code will be displayed in your shell. This way, you can modify the functions like forward-word, backward-word based on key code displayed.

More details about bindkey options (Standard Widgets): https://zsh.sourceforge.io/Doc/Release/Zsh-Line-Editor.html#Zle-Widgets

8: Restart your terminal, the results will be like this:

Conclusion (Author Opinion)

Productivity is a dynamic goal, and tools must fit your current needs and continually may be changed to enhance processes based on demands.

Currently, for my DevOps routine, the combination of ZSH and Starship proves to be a balanced cross-platform approach, offering a blend of performance, productivity, and extendability.

Ultimately, I reserve an honorable mention for Fish which your default configuration is a source of inspiration for the plugins I use in ZSH.

Properties Overriding: A Smart Way to Customize Systemd Unit Properties

2023-11-19T00:00:00+00:00

Blog Post Publish Date: 2023/11/19

This blog post covers an smart alternative to customize systemd unit process parameters.

This blog post delivers a explanation of the systemd override.conf file. It elucidates the drawbacks of directly altering unit service configurations, introduces the overriding approach as a solution, presents a practical, hands-on example, and culminates with my perspective on the effectiveness of this structural methodology.

Drawbacks of Direct Unit Service Editing

At times, there’s a need to modify certain properties within a particular systemd unit service, such as adjusting the value of an environment variable to enhance debug levels for troubleshooting. Traditionally, this involves directly altering the unit service file. For instance, if you wish to modify properties of Docker, you’d edit /etc/systemd/system/docker.service, implement the changes, refresh systemd with $ systemctl daemon-reload, and then restart the Docker process using $ systemctl restart docker.

While this approach functions adequately, it introduces certain behaviors that may pose issues depending on your use case.

Modifying default unit services elevates the risk of configuring settings incorrectly.
Custom changes can be challenging to discern if they lack proper comments.
Unit service files might be overridden based on the installation method. For instance, the unit service could be replaced according to package update definitions of a package manager like rpm or apt.

In the following section, I will elucidate an elegant workaround for these behaviors.

The Solution: Parameters Overriding

A sophisticated approach to customizing unit service parameters involves overriding them.

This method enables you to make changes to systemd unit properties while preserving the default file without any modifications. This is possible because the method reads custom properties from another file and merges them with the default properties.

Override parameters are declared in the following file pattern:

/etc/systemd/system/<UNIT-SERVICE-NAME>.service.d/override.conf

<UNIT-SERVICE-NAME> is a placeholder.

All unit service properties defined in override.conf will take precedence over those defined in <UNIT-SERVICE-NAME>.service.

Consider the following example, where both default and override unit properties are illustrated.

Default Properties

/etc/systemd/system/my-script.service

...
[Description]
# collapsed parameters...

[Service]
ExecStart=my-script.sh --prod

[Install]
# collapsed parameters...

Override Properties

/etc/systemd/system/my-script.service.d/override.conf

[Service]
ExecStart=my-script.sh --dev

When the my-script.service be executed, the ExecStart command will be my-script.sh --dev because it were declared in the override.conf that replace the ExecStart defined in the my-script.service.

An efficient shortcut to execute this action is through the $ systemctl edit <UNIT-SERVICE-NAME> command.

This command creates a directory and file at /etc/systemd/system/<UNIT-SERVICE-NAME>.service.d/override.conf, opens your text editor for making the necessary changes, and upon closing it, automatically executes $ systemctl daemon-reload behind the scenes. The only additional step is to execute $ systemctl restart <UNIT-SERVICE-NAME> when necessary to apply the changes.

Hands-On

Let’s get hands-on! This section will delve into a detailed example of the override method.

We will create a systemd unit service named my-script. To achieve this, we need to create the following files:

/tmp/my-script/env_custom.env           # env vars source of the my-script process
/tmp/my-script/env_default.env          # other env vars source of the my-script process
/tmp/my-script/my-script.sh             # script that my-script process run
/etc/systemd/system/my-script.service   # unit service file definition

The files can be found on my GitHub: c-neto/my-devops-labs/blog/2023-11-19

Initially, let’s create the working directory for the script.

$ mkdir /tmp/my-script/

Following that, we’ll generate the source code for the script.

$ vim /tmp/my-script/my-script.sh

#!/bin/bash

while true; do
  echo ">>> $APP_NAME - $MY_CUSTOM_ENV_VAR"
  sleep 1
done

Assign the source code to the execution file parameter.

$ chmod +x /tmp/my-script/my-script.sh

We will create two files that will serve as the source for environment variables in the my-script unit service.

$ vim /tmp/my-script/env_default.env

APP_NAME="My Awesome Script"
MY_CUSTOM_ENV_VAR="Irish Blood"

$ vim /tmp/my-script/env_custom.env

MY_CUSTOM_ENV_VAR="English Heart"

Next, we will create the systemd unit service file.

$ vim /etc/systemd/system/my-script.service

[Service]
WorkingDirectory=/tmp/my-script/
ExecStart=my-script.sh
EnvironmentFile=/tmp/my-script/env_default.env

[Install]
WantedBy=multi-user.target

Now, let’s reload the systemd daemon so that my-script can be managed as a service by systemd, and then start it.

$ systemctl daemon-reload
$ systemctl start my-script

To inspect the log of my-script, you can verify the value of the /tmp/my-script/env_default.env file.

$ journalctl -u my-script -f

Nov 19 16:11:29 fedora my-script.sh[33631]: >>> My Awesome Script - Irish Blood
Nov 19 16:11:30 fedora my-script.sh[33631]: >>> My Awesome Script - Irish Blood

Next, we will override the source file for environment variables.

$ mkdir /etc/systemd/system/my-script.service.d/

We will override the EnvironmentFile parameter.

$ vim /etc/systemd/system/my-script.service.d/override.conf

[Service]
EnvironmentFile=/tmp/my-script/env_custom.env

This time, variables from the env_custom.env file will replace those present in the env_default.env file.

$ journalctl -u my-script -f

Nov 19 16:21:42 fedora my-script.sh[58573]: >>> My Awesome Script - English Heart
Nov 19 16:21:43 fedora my-script.sh[58573]: >>> My Awesome Script - English Heart

Conclusion (Author Opinion)

In conclusion, leveraging systemd override parameters emerges as an invaluable tool in proactively addressing and preventing recurring issues. By preserving the integrity of the original unit service file while explicitly specifying custom parameters, this method significantly enhances troubleshooting capabilities. Another notable advantage lies in safeguarding against the loss of custom configurations that may be tied to the installation manager used to create a unit service (dnf, apt, brew).

The seamless flexibility provided by the $ systemctl edit command empowers users to swiftly and efficiently modify process properties, ensuring an agile and responsive approach to system management. In essence, incorporating systemd override parameters not only fortifies system stability but also streamlines the process of adapting and optimizing configurations to meet evolving needs. As a dynamic solution in the realm of systemd, it stands as an essential practice for maintaining a resilient and adaptable system environment.

Logstash Modular Pipelines: An Elegant Structure for Code Reusability and Duplication Avoiding

2023-11-12T00:00:00+00:00

Blog Post Publish Date: 2023/11/12

This blog post explores a Logstash pipelines structure to mitigate code duplicated and presents an elegant method for reusing code section across multiple pipelines.

The post provides a clear explanation of the Logstash configuration structure, outlining the problem addressed by modular pipelines. It includes an example configuration and a Docker-compose lab for hands-on testing and exploration of the possibilities. Finally, the post concludes with my opinion on the effectiveness of this structure.

A Little About Logstash and Pipelines Structure

The Logstash is an amazing tool for crafting robust log pipelines. Several plugins empower the ingestion, process, enrich, and output integration with external stacks. The pipelines are created using Logstash Configuration DSL (Domain-Specific Language), a high-level configuration language designed to be efficient and flexible, and focused on Log Pipeline needs.

The Logstash pipeline configuration is composed of three main sections:

input: Define the log ingestion source.
filter: Define the process, parser, and enrich routines.
output: Define the forward routines with external stacks.

More Details: Logstash - Configuration File Structure

The Problem: Scaling Up Pipelines == Growing Code Duplication

As the number of log pipelines increases, so does the complexity and tendency for code duplication. For straightforward application observability needs, having pipelines with input, filter, and output configured together in the same file might suffice. However, in complex scenarios involving large applications, such as distributed applications deployed on Kubernetes, various challenges emerge. Among these challenges, one of the most significant is avoiding code duplication.

When dealing with multiple applications requiring distinct logic for log processing, one approach is to create a single pipeline with logical conditions determining the specific processing statements based on fields such as Tags, which identify the module generating the event. While this method is advantageous in avoiding duplication of input and output configurations, it has drawbacks. It introduces overhead in the filter section and complicates troubleshooting in the Logstash API metrics for identifying application process issues.

Alternatively, another approach is to establish separate pipelines, each with its own dedicated configuration files for individual applications. While this resolves the filter overhead issue, it comes at the cost of duplicating both input and output configurations.

The Solution: Modularize the Pipeline

A good way to solve the problem mentioned before is modularize the Logstash Pipelines sections (input, filter, output) in isolated files, and load them in dedicated Pipelines based on their needs.

For this, it is necessary to create separate files with only one Pipeline section definition (input, filter, output), create dedicated pipelines for each Application, and configure them using Glob Pattern Support and Environment Variable notation.

The following sub topic will explain with more details about how to do it.

Configuration Example

First step, is to create the modules, that is, the code of the input, filter, and output in separate files, each one focused on their specifics and atomic needs.

$ tree /usr/share/logstash/pipeline/

├── filter-app-a.cfg        # filter of the application-a logic parser
├── filter-app-b.cfg        # filter of the application-b logic parser
├── input-http.cfg          # input to receive logs from http protocol
├── input-rabbitmq.cfg      # input to receive logs from rabbitmq queue
├── output-opensearch.cfg   # output to forward logs processed to opensearch server 
└── output-http.cfg         # output to forward logs processed to http server

Next, for each application, configure them with Environment Variable Logstash notation:

$ cat /usr/share/logstash/config/pipeline.yml

- pipeline.id: application-a-pipeline
  path.config: "/usr/share/logstash/pipeline/${LOGSTASH_PIPELINE_APPLICATION_A}.cfg"

- pipeline.id: application-b-pipeline
  path.config: "/usr/share/logstash/pipeline/${LOGSTASH_PIPELINE_APPLICATION_B}.cfg"

Finally, configure the composition of the pipeline with Glob Pattern Support notation. Consider the following example:

$ env

LOGSTASH_PIPELINE_APPLICATION_A="{input-http,input-rabbitmq,filter-app-a,output-opensearch,output-http}"
LOGSTASH_PIPELINE_APPLICATION_B="{input-http,filter-app-b,output-opensearch}"

LOGSTASH_PIPELINE_APPLICATION_A: Composes the Logstash pipeline application-a-pipeline to receive logs from HTTP and RabbitMQ, processed them with parse logic defined in the filter-app-a.cfg and forwarded to OpenSearch and HTTP server.
LOGSTASH_PIPELINE_APPLICATION_B: Composes the Logstash pipeline application-b-pipeline to receive logs from HTTP, process them with the logic defined in the filter-app-b.cfg and forwarded to OpenSearch only.

Main Advantages:

This approach brings the following advantages:

Avoid Code Duplication: Allows to use modules across multiple pipelines (for example, configure the same input and output module for all pipelines).
Reusable Code: Allows a simple configurable way to use more than one module of the same type in specific pipelines (for example, two output, and three input in one specific pipeline).
Decrease Complexity: The filter code is composed of only the application parser logic, avoiding conditional logical based on source event tags to identify how the logic need to be used.
Troubleshooting: Preserve the traceable of the pipelines in the logstash metric API.
Tests: Make it easier to test filter code because the input and output modules is simple to change (environment variables), allowing a configuration of the mock log source as input and stdout as a output to check if the logic is working as expected.

Docker Compose Lab

I created a docker-compose lab to test and explore the possibilities of the content presented in this post.

Check in my GitHub: c-neto/my-devops-lab/blog/2023-11-12/docker-compose.yaml.

Conclusion (Author Opinion)

Logstash is a versatile tool for log processing, providing well-configured solutions for both small and large workloads to meet Observability needs. While the Logstash Module structure offers a straightforward approach to mitigate recurring issues in pipeline creation, it is not the exclusive remedy for this challenge.

Understanding Observability requirements is essential for effectively harnessing the tool’s features. Occasionally, the Logstash Module Structure may introduce unnecessary complexity, especially in scenarios involving a limited number of applications.

I have implemented this structure and witnessed significant improvements in maintainability, particularly in the Tests advantages mentioned, where I can easily test my filter logics with a simple a environment variable value changing.

I genuinely appreciate this structure because my primary goal is to simplify and make it easier the maintainability, but it’s important to recognize that every case is unique, and each context presents its own complexity and challenges. Therefore, I recommend adopting this structure when the maintenance of pipelines becomes complex, code duplication is on the rise, and the configuration of Logstash becomes prone to errors, ultimately rendering it difficult to maintain.

References

Cloud Computing: Overview and Concepts

2023-10-21T00:00:00+00:00

Blog Post Publish Date: 2023/10/21

This blog post explores the impact of Cloud Computing, essential concepts, primary Cloud models, and my opinion about its adoption.

The Impact of Cloud Computing

Historically, Development teams emphasized agility in contributions, while Operations/Infrastructure teams focused more on stability due to hardware interactions. However, with the advent of technologies like virtualization, manual infrastructure tasks became obsolete, and infrastructure components started being provided as a service by Cloud providers like AWS, Google, Microsoft, and others.

Now, agility has become a crucial aspect for Operations teams as well. The maintenance of infrastructure as a service by Cloud providers alleviates concerns about electricity, hardware health, and other infrastructure requirements. This shift allows businesses to concentrate on their core purposes rather than infrastructure necessities. Outsourcing maintenance tasks to specialized teams enhances reliability and security, with various services tailored for infrastructure contexts and business cases. Additionally, hardware security is vital to address concerns like natural disasters and physical threats.

A significant keyword in the Cloud landscape is possibilities. The Cloud offers possibilities to access high computing performance components as needed, scale components on demand, and pay only for the time used. These possibilities can lead to reduced costs if your application or infrastructure is scalable, allowing you to pay only for the resources needed for application functionality.

However, it’s important to note that Cloud Computing isn’t a one-size-fits-all solution for every IT infrastructure case. It is one of the options to deploy your infrastructure or application. Before adopting the Cloud for your infrastructure, it’s crucial to ask yourself, “What problem am I trying to solve?” If your application aligns with the Cloud characteristics mentioned above, it might be the best choice for your needs.

Concepts and Characteristics

Cloud Computing is a model for delivering computing services over the internet, providing a platform to manage, process, and store data on remote servers.

The Cloud Computing main characteristics are:

Maintenance

Less concern about energy and hardware maintenance processes;
Business focuses more on IT technologies.

Reliability

Specialization in determining tools;
High availability possibilities.

Security

Utilization of specific security tools;
Software and hardware security solutions.

Performance

Hardware components available as a service;
Performance enhancements possible when needed.

Scalability

Scalability on demand;
Minimal IT resource requirements to run applications.

Cost and Elasticity

Cost management options;
Both soft and hard cost considerations with available possibilities.

See also: AWS - What is Cloud Computing

Cloud Main Models

The cloud is not limited to AWS, Google Cloud Platform, or Oracle Cloud. These mentioned cloud providers belong to the category of Public Cloud. However, it’s essential to understand that there are other ways to implement cloud solutions.

Public Cloud

Resources are shared and can be used by anyone (examples: AWS, GCP, Oracle Cloud).

Private Cloud

Often referred to as Bare Metal Cloud owned by a single entity, and resources are not shared with other corporations (examples: OpenStack, VMWare, and XenServer).

Hybrid Cloud

A combination of both Private Cloud and Public Cloud components.

MultiCloud

Cloud architecture is not dependent on a specific Cloud model and can run in both Private and Public Cloud environments.

Community Cloud

Custom Cloud architecture created by corporations to provide clients with the flexibility of multiple Public and Private Cloud features.

More Details: NIST - Cloud Computing Definition

Conclusion (Author Opinion)

Cloud computing is a powerful tool that can greatly benefit businesses. Cloud providers offer fantastic solutions and services to address the challenges and product requirements, mainly to large enterprises. Undoubtedly, Cloud computing has contributed to improvements in the IT product experience and the creation of new products, primarily due to the Business focusing more on IT technologies characteristic, which allows engineers to concentrate their efforts on the product rather than infrastructure challenges.

However, it is important to understand that it is not a one-size-fits-all solution. Its effectiveness depends on your specific needs and use cases. While it offers scalability and flexibility, it can also be more expensive, especially if not optimized for your requirements. Migrating to the Cloud requires specialized expertise to ensure a smooth transition and efficient management. Additionally, it’s important to note that the Cloud is not a guarantee of security; it requires a diligent implementation of security measures to safeguard your data and applications, further emphasizing the need for specialized expertise. Ultimately, the decision to embrace Cloud computing should be carefully evaluated, taking into account the unique demands and goals of your organization.

Git Clients Benchmarking: CLI vs. Graphical Interfaces

2023-10-19T00:00:00+00:00

Blog Post Publish Date: 2023/10/19

This blog post is aimed at helping you understand the pros and cons of the most popular Git repository clients.

The content was created with a focus on everyday use for fundamental tasks.

I presented the solutions pros/cons and my particular conclusion and suggestions.

1. Command Line Interface (CLI)

It is the official method for interacting with your repository. You can perform all the functions that the Git system provides using the Git CLI. It also offers comprehensive documentation, and official resources typically use Git CLI in their examples.

Key Pros:

Full Functionality: Git CLI offers all functionalities and capabilities that Git provides, ensuring nothing is compromised.

Scripting and Automation: Easily integrate Git commands into scripts and automate repetitive tasks.

Key Cons:

Learning Curve: For those unfamiliar with command-line interfaces, there may be a learning curve to grasp the commands and their usage effectively.

Less Visual Representation: Git CLI provides data primarily through text output, which can be less intuitive for some users.

Productivity: Repetitive and extensive use of fundamental operations can become unproductive.

More Details: Git Official Reference

2. GitKraken

GitKraken is a comprehensive graphical enterprise suite focusing on enhancing Git functionalities.

Key Pros:

Intuitive Interface: GitKraken offers an easy-to-understand graphical interface, making it accessible for both beginners and experienced users.

Visualization: Users can visualize their Git repository and workflow, aiding in a better understanding of branching, merging, and commits.

Key Cons:

Cost: While GitKraken offers a free version, more advanced features require a paid subscription.

More Details: GitKraken Official Reference

3. SourceTree

SourceTree is a freely available Git GUI developed and maintained by Atlassian, offering an intuitive interface for Git repository management.

Key Pros:

User-Friendly: SourceTree offers an intuitive, user-friendly interface, making it easy to understand for users new to Git.

Visual Representation: Users can visualize the Git workflow, including branches, commits, and merges, aiding in understanding repository history.

Key Cons:

Atlassian Like: Limited compatibility with non-Atlassian tools.

Features: Limited advanced features.

Resources Usage: Resource-intensive, slowing down your computer.

UI/UX: Complex interface, overwhelming for beginners.

OS Supports Limited: Available only for Windows and macOS.

More Details: SourceTree Official Reference

4. VSCode

VSCode, a popular text editor, inherently provides basic Git functionalities in a visual manner.

Key Pros:

Integration: Git features are seamlessly integrated into the editor, allowing for efficient version control directly from the coding environment.

UI/UX: Straight and Forward and intuitive interface to perform basic operations.

Customization: Users can enhance Git capabilities through various extensions available for VSCode.

Key Cons:

Not Dedicated: While it provides Git features, it’s primarily a text editor, so it might lack some advanced Git-specific features.

More Details: Introduction to Git in VS Code

5. JetBrains IDEs

JetBrains offers a range of Integrated Development Environments (IDEs) for various programming languages, all of which integrate Git functionalities.

Key Pros: Pros:

Comprehensive: JetBrains IDEs provide a full suite of development tools along with Git integration.

Language Support: Each IDE is tailored to specific languages, ensuring a seamless Git experience for the corresponding programming language.

Key Cons:

Resource Intensive: IDEs can be resource-heavy, especially for larger projects or on less powerful machines.

Cost: Depends on IDE programming language, it’s only paid version available.

More Details: JetBrains IDEA Using Git Integration

Conclusion (Author Opinion)

The Git CLI mastering is fundamental before to use the Graphical solutions. The GUI solutions are made to provides agility in the recurring operations. Only use the Git GUI solutions if you understand that what’s happening behind the scenes. The GUI solutions helps to make the changes, but when a big problems occurs, probably you will needs to use the Git CLI to solve it.

The choice between Git CLI and graphical interfaces depends on your preference, familiarity with command-line tools, and the specific needs of your project.

If you works a Software Developer, my suggestion is mastering the use of the tool your IDE provides.
If you work several stacks, like me as a DevOps Engineer, I suggest using VSCode Git integration. It provides a simple UX/UI for a quick way to check the commit differences in one place, and the free version of the extension GitLens provides a simple way to perform more advanced operations like squash and rebase.
For Release Engineer enterprise workloads, the GitKraken sounds like the better chosen because your UI/UX brings pretty historical commits graphics. If you work an Atlassian stack, and Linux workstations are not used, the SourceTree can be a good idea because integrations with Atlassian Stack are available.

Carlos Neto

Vim for Kubernetes Certification Exams (CKA, CKAD & CKS)

Vim: Less Mouse, More Efficiency

Vim Configuration

Essential Editing Commands

Cursor Movement

YAML Indentation

Commenting Multiple Lines

Search and Replace

Working with YAML Blocks

Working with Numbers

Running Shell Commands in Vim

Learn More

Workstation Setup for Kubernetes Exams (CKA, CKAD & CKS)

Configure the U.S. Keyboard Layout (MacBooks)

Disable Mission Control Shortcuts (MacBooks)

Disable Window Tiling Apps (MacBook)

History Search by Prefix

Setting Up Vim

Kubectl Aliases and Shortcuts

Copy Settings to Exam Question Node

References

Helm Tips: Inspect Your Chart Context with JSON

Are There IDEs for Helm Charts?

Debug the Right Way!

groupby + attrgetter: The Pythonic Way to Group Objects

Kubernetes Logs Simplified: Everything You Need to Know About Pod Logging

How Kubernetes Manages and Stores Logs

Understanding Kubernetes Log Format

Managing Log Growth with Log Rotation

Strategy to Ingest Kubernetes Logs to External Thirdy Party System

References

Fluent Bit: Generating Log IDs with SHA-256 Hashes for Log Deduplication

Why Do Duplicate Logs Occur in Fluent Bit?

The Importance of Unique Log IDs

Generating SHA-256 Hashes for Log Deduplication

References

Helm Charts: Development Practices from a Programmer’s Perspective

Why Helm Charts Are a Powerful Tool for IaC

Simple Made Easy - Is Helm Chart Overengineering?

DRY - Eliminating Redundancy in Kubernetes Manifests with Helm

YAGNI - Avoid Overengineering in Helm Templates

Library Charts - Ensuring Reusable Patterns

Conclusion

References

Logstash DB Enrichment - Tips and Traps!

Use Prepared Statements

When the Database is Out, Your Pipeline Won’t Start!

Lock Table == Lock Log Ingestion

Slow Query == Slow Log Ingestion

Define Timeouts for Database Login and Socket

Ensure DB Connection Identification

Sequel Opts != Database Connection Parameters

Conclusion

References

OpenSearch Concepts - Index and Shards

Understanding Indices and Shards

Useful Commands for Managing Indices and Shards Using the OpenSearch API

Dynaconf: The Python Config Silver Bullet!

Twelve-Factor App: Good Practices in Application Config

Python Config Libraries

Dynaconf!

Talk is cheap, Show me the code!

Step-by-step

Conclusion (Author Opinion)

Links

Logstash Sniffing: Improving Log Ingestion in OpenSearch Scalable Clusters

The Problem - Unbalanced Node Utilization in OpenSearch Cluster

The Solution - Utilizing OpenSearch API for Dynamic Hosts Registration

Conclusion (Author Opinion)

Links

fzf: Life is Too Short for Pipe Grep

Searching in the Terminal - Tedious Task

Fuzzy Finders - The Solution

How to Configure fzf Widgets in Zsh

Results

Conclusion (Author Opinion)

Links

ZSH + Starship: A Productivity Masterpiece

My Prompts Customization Experiences