Writing

Preventing Python Memory Leaks with weakref

"The application ran smoothly for two days, then memory usage steadily climbed until the OS terminated the worker."

Python manages memory automatically via reference counting combined with a cyclic garbage collector. In theory, once an object is no longer in use, it is freed. In practice, long-lived collections, caching layers, and circular dependencies retain strong references, preventing the garbage collector from reclaiming memory. This pattern is one of the most common causes of silent Python memory leaks.

Python's built-in weakref module solves this by allowing you to reference objects without increasing their reference count.

Strong vs. Weak References

1. weakref.ref() and weakref.proxy()

A standard assignment creates a strong reference:

import weakref


class ExpensiveTask:
    def __init__(self, name: str):
        self.name = name

    def __del__(self):
        print(f"'{self.name}' destroyed!")


task = ExpensiveTask("TaskA")
ref = weakref.ref(task)

print(ref())  # <ExpensiveTask object at 0x...>

del task  # Strong reference count drops to zero!
# Output: 'TaskA' destroyed!

print(ref())  # Returns None because the target no longer exists

With weakref.ref(), you dereference the object by calling ref(). If the underlying instance was reclaimed, it returns None.

Alternatively, weakref.proxy() creates a transparent surrogate:

proxy = weakref.proxy(task)
proxy.do_something()  # Behaves like task, but raises ReferenceError if task is collected

Specialized Collections: WeakKeyDictionary and WeakValueDictionary

The most common real-world culprit for memory leaks is module-level caching dictionaries. Placing instances into a standard dict keeps them alive forever unless manually removed.

WeakKeyDictionary

Keys are weakly held. When the key object has no other strong references outside the dict, the entire key-value pair is automatically purged:

import weakref
import gc


class Session:
    def __init__(self, user_id: int):
        self.user_id = user_id


cache = weakref.WeakKeyDictionary()

s = Session(101)
cache[s] = {"ip": "127.0.0.1"}
print(len(cache))  # 1

del s
gc.collect()
print(len(cache))  # 0: Automatically cleared!

WeakValueDictionary

Values are weakly held. Ideal for caching expensive objects (such as database entities or images) that should only remain in memory as long as active callers are holding them.

Real-World Open Source Implementations

1. Event Dispatchers (blinker and Django Signals)

Publish-subscribe systems register observer callbacks. If the dispatcher maintains strong references to observer instances, subscribers can never be garbage collected, even if the surrounding page or connection was closed.

Django and Blinker use weakref.ref and weakref.WeakMethod so subscribers are freed naturally when their owner scope terminates:

# Conceptual pattern in event dispatchers:
def connect(self, receiver):
    if hasattr(receiver, "__self__"):
        # Store weak reference to the bound method's instance
        self.receivers.append(weakref.WeakMethod(receiver))
    else:
        self.receivers.append(weakref.ref(receiver))

2. Method Caching (cachetools)

The @cachedmethod decorator caches function results on instances. Using a regular dict keyed by self would prevent self from ever being destroyed. cachetools leverages WeakKeyDictionary to associate per-instance cache tables without interfering with lifecycle teardown.

3. Cleanup Hooks with weakref.finalize

When wrapping C-extensions or external OS handles, weakref.finalize(obj, cleanup_func) provides a much safer alternative to __del__() (which notoriously interferes with cycle detection and exception logging):

class TempDirectoryManager:
    def __init__(self, path: str):
        self.path = path
        self._finalizer = weakref.finalize(self, self._cleanup, path)

    @staticmethod
    def _cleanup(path: str):
        print(f"Removing temporary files at {path}")

When to Reach for weakref

  • You are building a cache of objects whose lifetimes are determined elsewhere.
  • You are implementing observer or signal listeners.
  • You need back-pointers in tree, graph, or doubly-linked list structures without forming reference cycles.

References