"The application ran smoothly for two days, then memory usage steadily climbed until the OS terminated the worker."
Python manages memory automatically via reference counting combined with a cyclic garbage collector. In theory, once an object is no longer in use, it is freed. In practice, long-lived collections, caching layers, and circular dependencies retain strong references, preventing the garbage collector from reclaiming memory. This pattern is one of the most common causes of silent Python memory leaks.
Python's built-in weakref module solves this by allowing you to reference objects without increasing their reference count.
Strong vs. Weak References
1. weakref.ref() and weakref.proxy()
A standard assignment creates a strong reference:
import weakref
class ExpensiveTask:
def __init__(self, name: str):
self.name = name
def __del__(self):
print(f"'{self.name}' destroyed!")
task = ExpensiveTask("TaskA")
ref = weakref.ref(task)
print(ref()) # <ExpensiveTask object at 0x...>
del task # Strong reference count drops to zero!
# Output: 'TaskA' destroyed!
print(ref()) # Returns None because the target no longer exists
With weakref.ref(), you dereference the object by calling ref(). If the underlying instance was reclaimed, it returns None.
Alternatively, weakref.proxy() creates a transparent surrogate:
proxy = weakref.proxy(task)
proxy.do_something() # Behaves like task, but raises ReferenceError if task is collected
Specialized Collections: WeakKeyDictionary and WeakValueDictionary
The most common real-world culprit for memory leaks is module-level caching dictionaries. Placing instances into a standard dict keeps them alive forever unless manually removed.
WeakKeyDictionary
Keys are weakly held. When the key object has no other strong references outside the dict, the entire key-value pair is automatically purged:
import weakref
import gc
class Session:
def __init__(self, user_id: int):
self.user_id = user_id
cache = weakref.WeakKeyDictionary()
s = Session(101)
cache[s] = {"ip": "127.0.0.1"}
print(len(cache)) # 1
del s
gc.collect()
print(len(cache)) # 0: Automatically cleared!
WeakValueDictionary
Values are weakly held. Ideal for caching expensive objects (such as database entities or images) that should only remain in memory as long as active callers are holding them.
Real-World Open Source Implementations
1. Event Dispatchers (blinker and Django Signals)
Publish-subscribe systems register observer callbacks. If the dispatcher maintains strong references to observer instances, subscribers can never be garbage collected, even if the surrounding page or connection was closed.
Django and Blinker use weakref.ref and weakref.WeakMethod so subscribers are freed naturally when their owner scope terminates:
# Conceptual pattern in event dispatchers:
def connect(self, receiver):
if hasattr(receiver, "__self__"):
# Store weak reference to the bound method's instance
self.receivers.append(weakref.WeakMethod(receiver))
else:
self.receivers.append(weakref.ref(receiver))
2. Method Caching (cachetools)
The @cachedmethod decorator caches function results on instances. Using a regular dict keyed by self would prevent self from ever being destroyed. cachetools leverages WeakKeyDictionary to associate per-instance cache tables without interfering with lifecycle teardown.
3. Cleanup Hooks with weakref.finalize
When wrapping C-extensions or external OS handles, weakref.finalize(obj, cleanup_func) provides a much safer alternative to __del__() (which notoriously interferes with cycle detection and exception logging):
class TempDirectoryManager:
def __init__(self, path: str):
self.path = path
self._finalizer = weakref.finalize(self, self._cleanup, path)
@staticmethod
def _cleanup(path: str):
print(f"Removing temporary files at {path}")
When to Reach for weakref
- You are building a cache of objects whose lifetimes are determined elsewhere.
- You are implementing observer or signal listeners.
- You need back-pointers in tree, graph, or doubly-linked list structures without forming reference cycles.