Writing

Mastering Python Context Managers: from with Statements to contextlib

System resources—such as open file descriptors, network sockets, database sessions, and locks—are finite. When an application allocates a resource but fails to release it, it causes resource leaks, eventually degrading performance or exhausting operating system handles.

The Problem with Manual Cleanup

Consider opening a file and reading its content:

f = open('memo.txt', 'r')
print(f.read())
f.close()

While clean on the surface, this pattern is brittle. If an unexpected exception occurs between open() and f.close(), the file handle remains unclosed until the garbage collector eventually runs:

f = open('memo.txt', 'r')
data = f.read()
process_data(data)  # Suppose an unhandled exception occurs here
f.close()           # Never executed!

Using try...finally guarantees cleanup:

f = open('memo.txt', 'r')
try:
    data = f.read()
    process_data(data)
finally:
    f.close()  # Guaranteed to execute regardless of errors

While effective, repeating try...finally everywhere introduces visual noise and error-prone boilerplate.

The Idiomatic Solution: The with Statement

Python's with statement encapsulates acquisition and guaranteed release into a concise, readable block:

with open('memo.txt', 'r') as f:
    data = f.read()
    process_data(data)
# File is automatically closed upon leaving this block

Even if process_data() raises an unhandled exception, Python guarantees the file is safely closed before the exception bubbles up.

Building Custom Context Managers (Class Protocol)

Any object can participate in a with statement by implementing two dunder methods:

  • __enter__(self): Invoked upon entering the with block. Its return value is bound to the identifier specified after as.
  • __exit__(self, exc_type, exc_val, exc_tb): Invoked when leaving the block. If an exception occurred, its type, value, and traceback are passed as arguments. Returning True suppresses the exception; returning None or False allows it to propagate.

Here is an example managing a database session with SQLAlchemy:

from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker


class DatabaseSession:
    """Manages transactional database session lifecycle."""

    def __init__(self, connection_string: str):
        self.engine = create_engine(connection_string)
        self.SessionFactory = sessionmaker(bind=self.engine)
        self.session = None

    def __enter__(self):
        self.session = self.SessionFactory()
        return self.session

    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_type is not None:
            self.session.rollback()
        self.session.close()

Usage:

db = DatabaseSession("sqlite:///app.db")

with db as session:
    user = session.query(User).filter_by(id=1).one_or_none()
    print(user.name)

The Lightweight Alternative: @contextmanager

When building a full class feels unnecessarily heavy, Python's standard library provides contextlib.contextmanager. This decorator converts a generator containing a single yield statement into a context manager:

from contextlib import contextmanager


@contextmanager
def managed_resource(*args, **kwargs):
    # Setup / Acquisition
    resource = acquire_resource(*args, **kwargs)
    try:
        yield resource
    finally:
        # Teardown / Release
        release_resource(resource)

Everything before yield runs on __enter__, the yielded value is bound to as, and everything in finally runs on __exit__.

Asynchronous Context Managers (async with)

Modern asynchronous code using asyncio often requires managing connections or locks asynchronously. Python provides:

  1. __aenter__ and __aexit__ for class-based async contexts.
  2. contextlib.asynccontextmanager for generator-based async contexts.
from contextlib import asynccontextmanager


@asynccontextmanager
async def get_db_connection():
    conn = await acquire_db_connection()
    try:
        yield conn
    finally:
        await release_db_connection(conn)


async def get_all_users():
    async with get_db_connection() as conn:
        return await conn.query("SELECT * FROM users")

Best Practices

  1. Keep context blocks narrow: Only enclose lines that strictly depend on the scoped resource.
  2. Never swallow unexpected exceptions: In __exit__, only return True if you deliberately intended to catch and handle the specific error.
  3. Favor @contextmanager for simple state swaps: Temporarily modifying environment variables or redirecting stdout can be done in 5 lines with contextlib.

References