Blog Post Publish Date: 2024/07/21

Dynaconf: The Python Config Silver Bullet!#

This blog post outlines a brief overview of best practices for application configuration, explains the main features of Dynaconf, and provides a hands-on example of how to extend it to retrieve parameters from AWS Parameter Store.

Twelve-Factor App: Good Practices in Application Config#

Before starting to talk about Dynaconf or any code implementation for configuration, it’s necessary to understand the best practices used behind the scenes in systems with portable running requirements.

One of the main best practices references is the Twelve-Factor App, which is a methodology with a bundle of good practices for building modern applications with portable requirements, such as applications running on SaaS platforms and Kubernetes clusters. There is a chapter dedicated to application configuration: 12 Factor » Config.

In summary, the config chapter explains what (and what not) categorizes an application parameter, and emphasizes the separation of configuration from code.

Python Config Libraries#

Like almost all recurring generic needs and well-defined best practices, there are modules and frameworks to help us implement them. In Python, several tools can help achieve these best practices. I have used some nice tools like python-dotenv, which focuses on reading simple key-value pairs from a .env file and can set them as environment variables, and python-decouple, which provides a dynamic settings interface to help change parameters without having to redeploy your app (useful for web applications). Additionally, there are modules for parsing structured files that can be used as config files, such as PyYAML, json, tomllib, shlex, and configparser.

These modules are really nice and have specific use cases, especially when you can use a native approach and simple programs like routine scripts. But, with the provocative blog post title, I will present Dynaconf.

Dynaconf!#

Dynaconf is a robust configuration management for Python, designed to follow Twelve-Factor App config principles. It serves as an abstraction layer between code and settings, providing a simple interface to retrieve parameters and secrets from files or external system sources.

The highlight features include support for multiple config files in various formats (toml, yaml, json, ini), parameters via environment variables, and multi-hierarchical parameter set profiles support (e.g., default, development, production). Additionally, there is a CLI that is useful for debugging.

A notable feature is the capability to retrieve parameters from external parameter/secrets storage systems. There is built-in support for HashiCorp Vault and Redis, but it is also possible to extend it to retrieve parameters from any source based on your needs. This feature is really awesome! I will show an example of how to use it to retrieve parameters from AWS Parameter Store.

Talk is cheap, Show me the code!#

To exemplify how to use Dynaconf, I will show a step-by-step guide on how to configure and use it from scratch. The example covers how to set up multiple settings files and how to extend the parameter source to retrieve parameters from the AWS Parameter Store service (simulated by LocalStack in a Docker container).

Note

All code is in the following GitHub repository: github.com/c-neto/my-code-playground/blog-code-examples/2024-07-21-dynaconf-loaders

Step-by-step#

  1. Create a virtual environment and install dependencies:

python3 -m venv venv
venv/bin/pip install dynaconf boto3 
source venv/bin/activate

  1. Use the dynaconf-cli to speed up the settings and config.py files creation:

dynaconf init \
    -v PARAMETER_FROM_SETTINGS_TOML_FILE="foobar from settings.toml" \
    -s PARAMETER_FROM_SECRETS_TOML_FILE="foobar from .secrets.toml"

# » output
#
# ├── config.py       » python module with dynaconf instance
# ├── .secrets.toml   » settings files with: PARAMETER_FROM_SECRETS_TOML_FILE
# └── settings.toml   » settings files with: PARAMETER_FROM_SETTINGS_TOML_FILE

The config.py source code created by dynaconf-cli looks like this:

config.py#
from dynaconf import Dynaconf

settings = Dynaconf(
    envvar_prefix="DYNACONF",
    settings_files=['settings.toml', '.secrets.toml'],
)

You can use the dynaconf-cli to list parameters based on the configuration defined in config.py:

dynaconf -i config.settings list 

# » output
#
# PARAMETER_FROM_SETTINGS_TOML_FILE<str> 'foobar from settings.toml'
# PARAMETER_FROM_SECRETS_TOML_FILE<str> 'foobar from .secrets.toml'

  1. Run LocalStack in a container to simulate the AWS Parameter Store service:

docker run --rm -it --detach \
    --name localstack \
    -p 4566:4566 \
    -e SERVICES=ssm,sts \
    -e DEFAULT_REGION=us-east-1 \
    -e AWS_ACCESS_KEY_ID=test \
    -e AWS_SECRET_ACCESS_KEY=test \
    localstack/localstack:latest

  1. With LocalStack running, export the AWS environment variables to access the LocalStack API.

export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_ENDPOINT_URL=http://localhost:4566
export AWS_DEFAULT_REGION=us-east-1

  1. Create two parameters prefixed by /my-application/ named PARAMETER_FROM_AWS_PARAMETER_STORE_1 and PARAMETER_FROM_AWS_PARAMETER_STORE_2.

aws ssm put-parameter \
    --name "/my-application/PARAMETER_FROM_AWS_PARAMETER_STORE_1" \
    --value "foo" \
    --type "SecureString" \
    --endpoint-url http://localhost:4566

aws ssm put-parameter \
    --name "/my-application/PARAMETER_FROM_AWS_PARAMETER_STORE_2" \
    --value "bar" \
    --type "SecureString" \
    --endpoint-url http://localhost:4566

  1. Edit the conf.py to add the custom loaders to retrieve the parameters from AWS Parameter Store. All parameters within the /my-application/ namespace will be retrieved

config.py#
import logging

import boto3
from botocore.exceptions import ClientError, BotoCoreError
from dynaconf import Dynaconf, LazySettings
from dynaconf.loaders.base import SourceMetadata

logger = logging.getLogger(__name__)
PARAMETERS_PREFIX = "/my-application/"


# Retrieve parameters from AWS Parameter Store recursively by prefix
def _aws_parameter_store_loader(parameters_prefix: str) -> dict:
    ssm_client = boto3.client('ssm')

    paginator = ssm_client.get_paginator("get_parameters_by_path")
    pages = paginator.paginate(Path=parameters_prefix, Recursive=True, WithDecryption=True)

    parameters_from_aws_raw = {}

    for page in pages:
        for param in page["Parameters"]:
            name_full = param["Name"]
            value = param["Value"]
            parameters_from_aws_raw[name_full] = value

    parameters_from_aws = {}

    for name_full, value in parameters_from_aws_raw.items():
        try:
            *_, name = name_full.split("/")
        except ValueError:
            logger.warning(f'Parameter name pattern outside of expected format, skipping: "{name_full}"')
        else:
            parameters_from_aws[name] = value

    return parameters_from_aws


# Dynaconf callback - used in this case for retrieve parameters from AWS Parameter Store
def load(obj: LazySettings, env: str, silent: bool = True, key: str = None, filename: str = None) -> None:
    global parameters_from_aws

    source_metadata = SourceMetadata('aws-parameter-store', PARAMETERS_PREFIX)
    obj.update(parameters_from_aws, loader_identifier=source_metadata)
    logger.debug(f"Parameters loaded from AWS Parameter Store: {list(parameters_from_aws)}")


# Attempt to load parameters from AWS Parameter Store and update Dynaconf loaders
try:
    parameters_from_aws = _aws_parameter_store_loader(PARAMETERS_PREFIX)
except (BotoCoreError, ClientError) as e:
    LOADERS_FOR_DYNACONF = ["dynaconf.loaders.env_loader"]
    logger.warning("AWS active session not detected, skipping load parameters from AWS Parameter Store")
else:
    # __name__ is module name that have "load()" callback. It could be another module...
    LOADERS_FOR_DYNACONF = [__name__, "dynaconf.loaders.env_loader"]


# Initialize Dynaconf settings with specified files and loaders
settings = Dynaconf(
    settings_files=["settings.toml", ".secrets.toml"],
    envvar_prefix="MY_APP",
    LOADERS_FOR_DYNACONF=LOADERS_FOR_DYNACONF,
)

  1. Check the variables and note the PARAMETER_FROM_AWS_PARAMETER_STORE_1 and PARAMETER_FROM_AWS_PARAMETER_STORE_2 that were retrieved from AWS Parameter Store.

dynaconf -i config.settings list

# » output
#
# PARAMETER_FROM_SETTINGS_TOML_FILE<str> 'foobar from settings.toml'
# PARAMETER_FROM_SECRETS_TOML_FILE<str> 'foobar from .secrets.toml'
# PARAMETER_FROM_AWS_PARAMETER_STORE_1<str> 'foo'
# PARAMETER_FROM_AWS_PARAMETER_STORE_2<str> 'bar'

  1. The environment variables take precedence over other settings sources, following the Unix philosophy. To test this behavior, export MY_APP_PARAMETER_FROM_AWS_PARAMETER_STORE_2.

export MY_APP_PARAMETER_FROM_AWS_PARAMETER_STORE_2="param from environment variable"

Dynaconf will retrieve only variables that are prefixed by the value defined in the envvar_prefix instance argument; in this case, MY_APP.

Now, when PARAMETER_FROM_AWS_PARAMETER_STORE_2 is accessed, the value will be the one defined in the environment variable.

dynaconf -i config.settings list

# » output
# 
# PARAMETER_FROM_SETTINGS_TOML_FILE<str> 'foobar from settings.toml'
# PARAMETER_FROM_SECRETS_TOML_FILE<str> 'foobar from .secrets.toml'
# PARAMETER_FROM_AWS_PARAMETER_STORE_1<str> 'foo'
# PARAMETER_FROM_AWS_PARAMETER_STORE_2<str> 'param from environment variable'

In addition, you can peform inspect action to check the parameters to ensure source of each them.

dynaconf -i config.settings inspect

# » output
# 
# {
#   "header": {
#     "env_filter": "None",
#     "key_filter": "None",
#     "new_first": "True",
#     "history_limit": "None",
#     "include_internal": "False"
#   },
#   "current": {
#     "PARAMETER_FROM_SETTINGS_TOML_FILE": "foobar from settings.toml",
#     "PARAMETER_FROM_SECRETS_TOML_FILE": "foobar from .secrets.toml",
#     "PARAMETER_FROM_AWS_PARAMETER_STORE_1": "foo",
#     "PARAMETER_FROM_AWS_PARAMETER_STORE_2": "bar"
#   },
#   "history": [
#     {
#       "loader": "aws-parameter-store",
#       "identifier": "/my-application/",
#       "env": "global",
#       "merged": false,
#       "value": {
#         "PARAMETER_FROM_AWS_PARAMETER_STORE_1": "foo",
#         "PARAMETER_FROM_AWS_PARAMETER_STORE_2": "bar"
#       }
#     },
#     {
#       "loader": "toml",
#       "identifier": ".secrets.toml",
#       "env": "default",
#       "merged": false,
#       "value": {
#         "PARAMETER_FROM_SECRETS_TOML_FILE": "foobar from .secrets.toml"
#       }
#     },
#     {
#       "loader": "toml",
#       "identifier": "settings.toml",
#       "env": "default",
#       "merged": false,
#       "value": {
#         "PARAMETER_FROM_SETTINGS_TOML_FILE": "foobar from settings.toml"
#       }
#     },
#     {
#       "loader": "set_method",
#       "identifier": "settings_module_method",
#       "env": "global",
#       "merged": false,
#       "value": {
#         "SETTINGS_MODULE": [
#           "settings.toml",
#           ".secrets.toml"
#         ]
#       }
#     },
#     {
#       "loader": "set_method",
#       "identifier": "init_kwargs",
#       "env": "global",
#       "merged": false,
#       "value": {
#         "LOADERS_FOR_DYNACONF": [
#           "config",
#           "dynaconf.loaders.env_loader"
#         ],
#         "SETTINGS_FILE_FOR_DYNACONF": [
#           "settings.toml",
#           ".secrets.toml"
#         ],
#         "ENVVAR_PREFIX_FOR_DYNACONF": "MY_APP"
#       }
#     }
#   ]
# }

Conclusion (Author Opinion)#

I have been using Dynaconf around 4 years, and I really appreciate how the module is designed to address recurring challenges in a simple and straightforward manner. Previously, I used PyYAML, but I frequently created repetitive code when reading configurations and secrets from distinct files. When a mix of input parameter sources was required, such as YAML and environment variables, the code complexity increased, diverting attention from the core problem my systems aimed to solve.

When I first tried Dynaconf, I was impressed by its ability to manage multiple config files while merging the parameters and sub-parameters structure. This feature is very useful to me, especially for separating sensitive information into specific files.

Now, I only avoid using Dynaconf in cases where there is a Software Requirement to use only built-in modules. Otherwise, I don’t see a reason not to use it, especially in applications designed to run in Kubernetes.

I really like the module and suggest you give it a try!

Logstash Sniffing: Improving Log Ingestion in OpenSearch Scalable Clusters