AI TL;DR
Too lazy to read? Summarize this post with:
Blog Post Publish Date: 2024/07/21
Dynaconf: The Python Config Silver Bullet!#
This blog post gives a short overview of good practices for application configuration, explains the main features of Dynaconf, and walks through a hands-on example that extends it to read parameters from AWS Systems Manager Parameter Store.
Twelve-Factor App: Good Practices for Application Config#
Before talking about Dynaconf or any code, it helps to know the practices behind portable configuration.
The Twelve-Factor App is a methodology with a set of good practices for building modern applications that run anywhere, such as on SaaS platforms or Kubernetes clusters. It has a chapter dedicated to configuration, which defines what counts as configuration (anything that changes between deploys, such as credentials and resource addresses) and insists on keeping it strictly separate from code.
Python Config Libraries#
Like most common needs with well-known practices, configuration has libraries to help. I have used a few good ones in Python:
python-dotenv reads simple key-value pairs from a
.envfile and can export them as environment variables.python-decouple gives you one settings interface so you can change parameters without redeploying the app, which is useful for web applications.
Structured file parsers can also serve as config readers, such as PyYAML, json, tomllib, shlex, and configparser.
These modules have their use cases, especially for simple scripts where the standard library is enough. But, as the provocative title suggests, this post is about Dynaconf.
Dynaconf#
Dynaconf is a configuration management library for Python, designed around the Twelve-Factor config principles. It sits between your code and your settings and gives you one interface to read parameters and secrets from files or external systems.
Its main features are:
Multiple config files in several formats (toml, yaml, json, ini), merged into one settings object.
Parameters from environment variables, which take precedence over files.
Layered environments (profiles) such as default, development, production.
A CLI that helps with debugging.
The feature I like most is reading parameters from external parameter and secret stores. Dynaconf has built-in support for HashiCorp Vault and Redis, and you can write your own loader for any other source. The example below does exactly that for AWS Parameter Store.
Talk Is Cheap, Show Me the Code!#
This step-by-step guide sets up Dynaconf from scratch with two settings files, then adds a custom loader for the AWS Parameter Store service. LocalStack simulates AWS in a Docker container, so you don't need an AWS account.
Note
All code is in this GitHub repository: github.com/c-neto/my-code-playground/blog-code-examples/2024-07-21-dynaconf-loaders
Step-by-step#
Create a virtual environment and install dependencies:
python3 -m venv venv
venv/bin/pip install dynaconf boto3
source venv/bin/activate
Use the Dynaconf CLI to create the settings files and
config.py:
dynaconf init \
-v PARAMETER_FROM_SETTINGS_TOML_FILE="foobar from settings.toml" \
-s PARAMETER_FROM_SECRETS_TOML_FILE="foobar from .secrets.toml"
# » output
#
# ├── config.py » python module with dynaconf instance
# ├── .secrets.toml » settings files with: PARAMETER_FROM_SECRETS_TOML_FILE
# └── settings.toml » settings files with: PARAMETER_FROM_SETTINGS_TOML_FILE
The config.py that the CLI creates looks like this:
from dynaconf import Dynaconf
settings = Dynaconf(
envvar_prefix="DYNACONF",
settings_files=['settings.toml', '.secrets.toml'],
)
The CLI can list the parameters that config.py loads:
dynaconf -i config.settings list
# » output
#
# PARAMETER_FROM_SETTINGS_TOML_FILE<str> 'foobar from settings.toml'
# PARAMETER_FROM_SECRETS_TOML_FILE<str> 'foobar from .secrets.toml'
Run LocalStack in a container to simulate the AWS Parameter Store service:
docker run --rm -it --detach \
--name localstack \
-p 4566:4566 \
-e SERVICES=ssm,sts \
-e DEFAULT_REGION=us-east-1 \
-e AWS_ACCESS_KEY_ID=test \
-e AWS_SECRET_ACCESS_KEY=test \
localstack/localstack:latest
With LocalStack running, export the AWS environment variables that point to its API:
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_ENDPOINT_URL=http://localhost:4566
export AWS_DEFAULT_REGION=us-east-1
Create two parameters under the
/my-application/prefix, namedPARAMETER_FROM_AWS_PARAMETER_STORE_1andPARAMETER_FROM_AWS_PARAMETER_STORE_2:
aws ssm put-parameter \
--name "/my-application/PARAMETER_FROM_AWS_PARAMETER_STORE_1" \
--value "foo" \
--type "SecureString" \
--endpoint-url http://localhost:4566
aws ssm put-parameter \
--name "/my-application/PARAMETER_FROM_AWS_PARAMETER_STORE_2" \
--value "bar" \
--type "SecureString" \
--endpoint-url http://localhost:4566
Edit
config.pyto add a custom loader that reads every parameter under/my-application/from AWS Parameter Store. A Dynaconf loader is a module with aload()function that Dynaconf calls to add values to the settings. If there is no AWS session, the code falls back to the environment loader only:
import logging
import boto3
from botocore.exceptions import ClientError, BotoCoreError
from dynaconf import Dynaconf, LazySettings
from dynaconf.loaders.base import SourceMetadata
logger = logging.getLogger(__name__)
PARAMETERS_PREFIX = "/my-application/"
# Retrieve parameters from AWS Parameter Store recursively by prefix
def _aws_parameter_store_loader(parameters_prefix: str) -> dict:
ssm_client = boto3.client('ssm')
paginator = ssm_client.get_paginator("get_parameters_by_path")
pages = paginator.paginate(Path=parameters_prefix, Recursive=True, WithDecryption=True)
parameters_from_aws_raw = {}
for page in pages:
for param in page["Parameters"]:
name_full = param["Name"]
value = param["Value"]
parameters_from_aws_raw[name_full] = value
parameters_from_aws = {}
for name_full, value in parameters_from_aws_raw.items():
try:
*_, name = name_full.split("/")
except ValueError:
logger.warning(f'Parameter name pattern outside of expected format, skipping: "{name_full}"')
else:
parameters_from_aws[name] = value
return parameters_from_aws
# Dynaconf callback - used in this case for retrieve parameters from AWS Parameter Store
def load(obj: LazySettings, env: str, silent: bool = True, key: str = None, filename: str = None) -> None:
global parameters_from_aws
source_metadata = SourceMetadata('aws-parameter-store', PARAMETERS_PREFIX)
obj.update(parameters_from_aws, loader_identifier=source_metadata)
logger.debug(f"Parameters loaded from AWS Parameter Store: {list(parameters_from_aws)}")
# Attempt to load parameters from AWS Parameter Store and update Dynaconf loaders
try:
parameters_from_aws = _aws_parameter_store_loader(PARAMETERS_PREFIX)
except (BotoCoreError, ClientError) as e:
LOADERS_FOR_DYNACONF = ["dynaconf.loaders.env_loader"]
logger.warning("AWS active session not detected, skipping load parameters from AWS Parameter Store")
else:
# __name__ is module name that have "load()" callback. It could be another module...
LOADERS_FOR_DYNACONF = [__name__, "dynaconf.loaders.env_loader"]
# Initialize Dynaconf settings with specified files and loaders
settings = Dynaconf(
settings_files=["settings.toml", ".secrets.toml"],
envvar_prefix="MY_APP",
LOADERS_FOR_DYNACONF=LOADERS_FOR_DYNACONF,
)
List the settings again.
PARAMETER_FROM_AWS_PARAMETER_STORE_1andPARAMETER_FROM_AWS_PARAMETER_STORE_2now come from AWS Parameter Store:
dynaconf -i config.settings list
# » output
#
# PARAMETER_FROM_SETTINGS_TOML_FILE<str> 'foobar from settings.toml'
# PARAMETER_FROM_SECRETS_TOML_FILE<str> 'foobar from .secrets.toml'
# PARAMETER_FROM_AWS_PARAMETER_STORE_1<str> 'foo'
# PARAMETER_FROM_AWS_PARAMETER_STORE_2<str> 'bar'
Environment variables take precedence over every other source. To see it, export
MY_APP_PARAMETER_FROM_AWS_PARAMETER_STORE_2:
export MY_APP_PARAMETER_FROM_AWS_PARAMETER_STORE_2="param from environment variable"
Dynaconf only reads environment variables that start with the
envvar_prefixvalue,MY_APPin this case.
Now PARAMETER_FROM_AWS_PARAMETER_STORE_2 returns the value from the environment variable:
dynaconf -i config.settings list
# » output
#
# PARAMETER_FROM_SETTINGS_TOML_FILE<str> 'foobar from settings.toml'
# PARAMETER_FROM_SECRETS_TOML_FILE<str> 'foobar from .secrets.toml'
# PARAMETER_FROM_AWS_PARAMETER_STORE_1<str> 'foo'
# PARAMETER_FROM_AWS_PARAMETER_STORE_2<str> 'param from environment variable'
The inspect command shows where each value came from:
dynaconf -i config.settings inspect
# » output
#
# {
# "header": {
# "env_filter": "None",
# "key_filter": "None",
# "new_first": "True",
# "history_limit": "None",
# "include_internal": "False"
# },
# "current": {
# "PARAMETER_FROM_SETTINGS_TOML_FILE": "foobar from settings.toml",
# "PARAMETER_FROM_SECRETS_TOML_FILE": "foobar from .secrets.toml",
# "PARAMETER_FROM_AWS_PARAMETER_STORE_1": "foo",
# "PARAMETER_FROM_AWS_PARAMETER_STORE_2": "bar"
# },
# "history": [
# {
# "loader": "aws-parameter-store",
# "identifier": "/my-application/",
# "env": "global",
# "merged": false,
# "value": {
# "PARAMETER_FROM_AWS_PARAMETER_STORE_1": "foo",
# "PARAMETER_FROM_AWS_PARAMETER_STORE_2": "bar"
# }
# },
# {
# "loader": "toml",
# "identifier": ".secrets.toml",
# "env": "default",
# "merged": false,
# "value": {
# "PARAMETER_FROM_SECRETS_TOML_FILE": "foobar from .secrets.toml"
# }
# },
# {
# "loader": "toml",
# "identifier": "settings.toml",
# "env": "default",
# "merged": false,
# "value": {
# "PARAMETER_FROM_SETTINGS_TOML_FILE": "foobar from settings.toml"
# }
# },
# {
# "loader": "set_method",
# "identifier": "settings_module_method",
# "env": "global",
# "merged": false,
# "value": {
# "SETTINGS_MODULE": [
# "settings.toml",
# ".secrets.toml"
# ]
# }
# },
# {
# "loader": "set_method",
# "identifier": "init_kwargs",
# "env": "global",
# "merged": false,
# "value": {
# "LOADERS_FOR_DYNACONF": [
# "config",
# "dynaconf.loaders.env_loader"
# ],
# "SETTINGS_FILE_FOR_DYNACONF": [
# "settings.toml",
# ".secrets.toml"
# ],
# "ENVVAR_PREFIX_FOR_DYNACONF": "MY_APP"
# }
# }
# ]
# }
Why I Use Dynaconf#
I have been using Dynaconf for about 4 years, and I like how it solves common configuration problems in a simple, direct way. Before, I used PyYAML, but I kept writing the same code to read configuration and secrets from different files. When a mix of sources was needed, such as YAML and environment variables, the code grew more complex and pulled attention away from the problem the system was supposed to solve.
The first time I tried Dynaconf, its ability to merge several config files, including nested parameters, impressed me. It's especially useful for keeping sensitive values in their own file.
Today, I only skip Dynaconf when a project requires the standard library only. Otherwise, I don't see a reason not to use it, especially in applications that run on Kubernetes. I really like the module and suggest you give it a try!
If you write Python for operations work, two other posts may interest you: groupby + attrgetter for grouping objects without loops, and OpenTelemetry for instrumenting a Python application.
Key Takeaways#
The Twelve-Factor App says configuration is anything that changes between deploys, and it must live outside the code.
python-dotenv, python-decouple, and file parsers such as PyYAML cover simple cases, but mixing sources with them means writing glue code.
Dynaconf merges several config files and formats, environment variables, and environment profiles into one
settingsobject.Environment variables with the configured prefix (
MY_APP_in the example) override every other source.A custom loader is a module with a
load()function, added toLOADERS_FOR_DYNACONF. The example reads every parameter under/my-application/from AWS Parameter Store.dynaconf listanddynaconf inspectshow the final values and which loader set each one.
References#
I'm Carlos Neto, and this blog is a brain dump of my career experience. More about me on my home page. If you have questions or a different approach, contact me on LinkedIn or leave a comment below.