Simplifying Configuration Management in Pure Python

Photo by Hans-Peter Gauster on Unsplash

Hey! Alejandro here, from Mostly Harmless Ideas. Welcome to Coding Lessons, a section of MHI where I sparingly post short, coding-focused articles. This one came out after a couple of days struggling to find the right balance for managing complexity in a recent project. Hope you enjoy it.

If you’ve ever worked on any application with a development and a production environment, you’ve dealt with this problem: managing configurations and feature flags. A centralized, easy-to-modify location for configurable options that vary across different deployment environments is crucial for maintaining flexibility and enabling rapid iterations. I’ve been recently working on a mid-sized chatbot project that requires a way to toggle specific features in different environments (think enabling different kinds of tools or skills for the chatbot).

After struggling with environment variables and looking over some really over-engineered solutions, I found a straightforward, pure Python approach that works perfectly for small teams that want to iterate fast. This article is a short presentation of this approach. We will explore the motivation behind this method, provide implementation details, and highlight potential pitfalls to avoid.

Motivation

As applications grow, so do their configurations. Development, staging, and production environments often require distinct settings. If not appropriately managed, managing these configurations can become cumbersome. Traditional approaches involve plain text files like a .env or config.toml file—that are not expressive enough—, using structured formats like a config.json or config.yaml file—which are more expressive but harder to read and maintain—or worse, relying on external libraries to manage configurations, leading to complexity and confusion.

The proposed method simplifies this by utilizing separate Python files for each environment. By dynamically loading the appropriate configuration based on a single environment variable, we can keep our code clean, organized, and easily managed.

An example of a well-established framework that follows this practice is Django. Django projects often separate environment-specific configurations into distinct Python files, such as development.py and production.py, to manage settings for different environments effectively. This approach is widely recommended in the Django community as it simplifies transitions between environments and allows for better organization of configuration settings.

Implementation Details

The core idea of this method is to put configuration options and feature flags in a Python file so that, once imported, you can access all those variables as module members. The only real trick is to dynamically import the correct file based on an environment variable, so you get development, staging, testing, or production configuration (or whatever contrived scenario you have) without the rest of your code knowing what environment is running on.

Here’s a plain step-by-step process.

Step 1: Create Environment-Specific Configuration Files

We will create separate configuration files for each environment (e.g., production.config.py, staging.config.py, development.config.py). Each file will define global variables for feature flags and other configuration options.

Example: production.config.py

NEW_FEATURE = False 
EXPERIMENTAL_FEATURE = False 
DATABASE_URL = "https://prod.database.url"`

Example: staging.config.py

NEW_FEATURE = True 
EXPERIMENTAL_FEATURE = False 
DATABASE_URL = "https://staging.database.url"`

Example: development.config.py

NEW_FEATURE = True 
EXPERIMENTAL_FEATURE = True 
DATABASE_URL = "https://dev.database.url"`

Step 2: Create a Loader Function

Next, we will create a config.py file that contains a function to load the appropriate configuration based on the ENVIRONMENT variable.

import os
import importlib

def load_config():
    environment = os.getenv("ENVIRONMENT", "DEVELOPMENT").upper()

    try:
        return importlib.import_module(f"{environment.lower()}.config")
    except ImportError:
        raise Exception(f"Configuration for environment '{environment}' not found.")

Step 3: Access Configuration in Your Application

Finally, we can access the loaded configuration in our application by calling the load_config function.

from config import load_config

def main():
    config = load_config()

    if config.NEW_FEATURE:
        print("New Feature is enabled!")
    else:
        print("New Feature is disabled.")

    print(f"Database URL: {config.DATABASE_URL}")

if __name__ == "__main__":
    main()

Step 4: Set the Environment Variable

Before running your application, set the ENVIRONMENT variable in your terminal (or in a .env file if you are already using it):

export ENVIRONMENT=production python main.py
# or staging or development

Advantages and Limitations

When managing application configurations and feature flags, developers often face a choice between using environment-specific Python configuration files and traditional .env files that are most commonplace. Each approach has its own set of advantages and limitations. Here, we will explore these aspects in detail.

Advantages of Python Config Files

1. Version Control: Python configuration files can be checked into version control systems (like Git) without exposing sensitive information. This allows teams to track changes, collaborate effectively, and maintain a history of configuration adjustments across different environments.

2. Expressivity: Python is far more expressive than any plain-text or structured format. You can have options that are lists, dictionaries, or even instances of custom classes. You shouldn’t abuse this power, though.

3. Computed Values: Python files allow for the inclusion of computed values and logic. This means you can dynamically generate configurations based on other parameters or conditions, providing greater flexibility than static key-value pairs in .env files.

4. Modular Design: Common configuration values can be refactored into separate modules imported by relevant configuration files. This modularity promotes code reuse and reduces redundancy, making maintaining configurations across multiple environments easier.

Limitations of Python Config Files

1. Security Concerns: Unlike .env files, which are often excluded from version control (ensuring sensitive information remains private), Python configuration files may inadvertently expose sensitive data if not managed properly. Developers must exercise caution to avoid committing sensitive information to the repository.

2. Risk of Misconfiguration: The flexibility of Python configuration files can lead to developers unintentionally modifying production settings during development or testing phases, potentially causing disruptions in live environments.

3. Learning Curve: This approach may not be as familiar to new developers as the widely adopted .env file convention. As a result, onboarding new team members may require additional training or documentation to understand the custom setup.

Advantages of .env Files

• Simplicity: .env files are straightforward text files that store key-value pairs, making them easy to read and edit.

• Common Practice: The use of .env files is a well-established practice in many development communities, making it easier for new developers to adapt.

Limitations of .env Files

• No Security by Default: While .env files can be excluded from version control, they are still plain text files that can be easily accessed if not adequately secured.

• Limited Functionality: .env files do not support computed values or complex logic, which can limit their usability in more dynamic applications. However, this may be considered a strength in reducing configuration complexity.

Striking a Sane Balance

There is no free lunch, or so the saying goes. However, you may strike a sane balance in this case by externalizing sensitive and static configuration options to .env files and keeping more dynamic options (e.g., feature flags, option lists, or computable values) in Python files. This way, you maintain flexibility for rapidly changing or very development-specific configuration variables and leave the most sensitive, project-wide configurations for maintainers or DevOps engineers to handle.

Obvious Pitfalls to Avoid

While this approach offers a clean and organized way to manage configurations, there are some pitfalls to be aware of:

1. Security Risks: I already said it, but to reiterate, DO NOT STORE sensitive information (e.g., API keys and database credentials) in plain text within your configuration files. Instead, consider using environment variables or, even better, secret management tools.

2. Version Control: As a follow-up to the above, ensure that sensitive information (e.g., what’s included in .env files) is excluded from version control (e.g., using .gitignore). This helps prevent accidental exposure of secrets.

3. Error Handling: Implement robust error handling when loading configurations. If a specified configuration file does not exist or fails to import, your application should gracefully handle this in upstream code rather than crash unexpectedly.

4. Documentation: Clearly document each configuration file and its purpose. You have Python comments, for Turing’s sake! This will help team members understand the differences between environments and make necessary modifications.

5. Testing: Regularly test your application in all environments to ensure that the configurations are correctly applied and that features behave as expected.

Conclusion

Managing feature flags and configurations in Python can be streamlined using environment-specific configuration files loaded dynamically based on a ENVIRONMENT variable. This approach enhances organization, simplifies edits, and provides clarity when dealing with multiple environments. By being mindful of security risks and pitfalls, you can implement this strategy effectively in your projects, leading to smoother development cycles and more robust applications.

Choosing between Python configuration files and .env files depends on your project's needs and the team's familiarity with each approach. While Python configuration files offer greater flexibility and modularity, they come with risks that require careful management. On the other hand, .env files provide simplicity and are widely recognized but lack built-in security features and advanced capabilities.

Ultimately, understanding the advantages and limitations of each method will help you make an informed decision that aligns with your development practices and security requirements.

Now, I’m interested in reading your thoughts. Have you ever used Python configuration files? If not, why not? If yes, what was your experience?