Chapter 15: Dunder Methods and Protocols
Section 1
Dunder methods, short for double-underscore methods, are special methods whose names begin and end with two underscores, such as __init__, __str__, and __len__. Python calls them automatically in response to built-in operations: __init__ runs when an object is created, __str__ runs when it is printed, and __len__ runs when len() is called on it. You rarely call a dunder method directly; instead, you define it in a class, and Python's built-in functions and operators dispatch to it for you. That dispatch is the heart of Python's data model, and it is what makes custom classes feel like first-class citizens of the language, rather than foreign objects that require special handling.
A protocol, in Python, is a set of dunder methods that together define a kind of behaviour, such as being a container, being iterable, being comparable, or being usable as a context manager. Unlike a formal interface in some other languages, a Python protocol is usually informal: a class supports a protocol simply by defining its methods, without needing to declare that it does so or to inherit from a specific parent. That is duck typing applied to protocols, where what matters is what an object can do, not what it claims to be. That flexibility allows any class to support a protocol by defining a few methods, without fitting into a hierarchy.
That flexibility also requires care, because a class that defines only part of a protocol may behave inconsistently when used in a context that expects the whole protocol. A collection that supports len() but not iteration, for example, may surprise a caller who reasonably expects both, based on experience with built-in collections. To avoid that surprise, implement a protocol completely, or document clearly which parts are implemented and which are not, so callers know what to rely on and do not discover a missing method only when they try to use it.
The most important dunder methods for beginners are those for creation, representation, and comparison, which were introduced in earlier chapters and are revisited here as a system. Creation methods set up an object, representation methods control how it appears as text, and comparison methods control how it compares to other objects. Those operations are fundamental, because almost every class will be printed, logged, compared, or sorted at some point, and they therefore deserve careful design as part of a class's interface, not as afterthoughts added hastily when a specific operation happens to be needed.
Consistency across those methods matters as much as each method individually. An object that prints one way should not compare in a way that contradicts its printed form, and equality and hashing should agree, as should ordering and equality. Treating dunder methods as a system of protocols, rather than as a list of isolated special methods, helps you design classes whose behaviour is coherent across many different contexts, which is the ultimate goal of protocol implementation.
class Book:
def __init__(self, title):
self.title = title
def __repr__(self):
return f"Book({self.title!r})"
print(Book("Dune"))Check: What is a protocol in Python, and how does defining dunder methods let a class support one without inheriting from a parent?
Section 2
Object creation in Python is a two-step process, involving __new__ and __init__. The __new__ method creates a new, empty object, and __init__ then initialises it by setting its attributes. Most classes need only __init__, and understanding both helps you see creation as stages rather than as a single, indivisible event. When a class is called, Python first creates the object through __new__, then passes that object to __init__ along with the arguments given at creation, so __init__ receives an already-created object rather than creating it itself.
__new__ is implicitly a static method that receives the class as its first argument and returns a new instance, usually by calling the parent's __new__. You will rarely need to define it yourself, except for immutable types, or for other advanced cases where creation itself must be customised rather than just initialisation. Knowing that it exists, and that it runs before __init__, helps you understand why __init__ works the way it does, and why customisation for immutable objects must happen in __new__ rather than in __init__.
That timing matters for immutable objects in particular. By the time __init__ runs, an immutable object is already immutable, and its attributes cannot be changed, so setting them in __init__ would fail. For such types, attributes must be set during creation, through __new__, which is a different pattern from the ordinary mutable case. For most mutable classes, however, __init__ alone is sufficient, and __new__ can safely be left to its default implementation, which creates an empty object ready for initialisation.
Overriding __new__ incorrectly can break object creation in subtle ways, such as returning an object of the wrong class or failing to call the parent's __new__, which would leave the object uncreated or incorrectly created. Failures at creation time can be hard to diagnose, because they occur before any of the object's methods have been called, and the traceback may not clearly show that creation itself was the problem. For that reason, leave __new__ alone unless you have a specific, advanced need for customising creation itself, and trust its default for ordinary mutable classes, where it works reliably without intervention.
class Point:
def __init__(self, x, y):
self.x, self.y = x, y
print(Point(1, 2).x)Understanding creation as stages also helps when debugging a failure that occurs before an object is fully initialised, because you can ask whether the failure was in creation, in __new__, or in initialisation, in __init__, and that distinction narrows the search. A failure in creation often involves the class itself, or its parents, while a failure in initialisation often involves the arguments passed, or the validation performed on them. That diagnostic distinction, between stages, is a practical benefit of knowing that creation is not a single event, even when you never define __new__ yourself.
Check: What is the difference between __new__ and __init__ in object creation?
Section 3
Representation methods, __str__ and __repr__, control how an object appears as text, and they serve different audiences. __str__ is for end users, giving a readable, friendly form, while __repr__ is for developers, giving a detailed form that ideally shows how the object could be recreated. If __str__ is not defined, str() will fall back to __repr__, which means defining __repr__ alone gives a reasonable representation for both audiences, though not necessarily an ideal one for users, who may prefer a simpler form without class names, quotes, or technical details.
The guideline is to define __repr__ always, for debugging, and to define __str__ additionally when the class will be shown to end users in a context where a readable form matters, such as a user interface or a report. In those contexts, the technical __repr__ form would be distracting, while a simpler __str__ improves readability without sacrificing debuggability, because __repr__ remains available for developers who need the technical form. That dual representation serves both audiences well, without forcing either to use a form designed for the other, which is common for domain entities such as books, tasks, or users that appear both in user-facing output and in developer-facing logs.
Representation methods should be reliable and free of side effects. They may be called in contexts where an exception would be confusing, such as during logging or debugging when a program is already in an unusual state, and a representation that raises an exception would obscure the original problem with a new one, making debugging harder rather than easier. Similarly, printing an object should not change it, because callers who think they are merely printing or logging an object would be surprised by a hidden change to its state, which violates the principle of least surprise.
That safety requirement applies to all dunder methods that Python calls implicitly, not just to representation methods. Implicit calls happen in many contexts, including sorting, iteration, and context management, where the class author cannot control when or how their method will be called. Protocol methods must therefore be robust and predictable in all contexts, not just in the ones the author anticipated, because generic code may call them in ways the author did not foresee, and a method that fails or changes state unexpectedly in such a context will cause bugs that are hard to trace to their source.
class Book:
def __init__(self, title):
self.title = title
def __str__(self):
return self.title
def __repr__(self):
return f"Book({self.title!r})"
print(str(Book("Dune")), repr(Book("Dune")))A good representation also helps when objects are stored in logs that are read later, by people who were not present when the program ran, because a clear __repr__ lets them reconstruct what the object was, without needing to rerun the program or to guess from a cryptic address. That reconstructive value is why __repr__ should include the class name and the key attributes, in a form resembling a constructor call, so that a reader can see both the type and the state at a glance, and can even copy the representation to recreate a similar object for testing.
Check: Why should representation methods be free of side effects, given that Python calls them implicitly in many contexts?
Section 4
Comparison methods define how objects compare, for equality and for ordering, and they form a protocol where several methods work together to provide consistent behaviour across different operations. Equality uses __eq__ and __ne__, while ordering uses __lt__, __le__, __gt__, and __ge__, each corresponding to an operator. Defining only some of them can lead to inconsistencies, such as an object that compares equal with == but not with !=, or that sorts correctly with < but not with <=. Those inconsistencies arise when comparison methods are not defined consistently as a group, and they should be avoided by defining the full group or by using a helper to generate related methods from a single definition.
Python's functools.total_ordering decorator is such a helper: it generates the other ordering methods from __eq__ and one ordering method, such as __lt__, ensuring consistency among them automatically, without needing to write each one manually and without risking inconsistency through manual duplication of logic across several methods. That benefit grows with the number of comparison methods and with the frequency of changes to comparison logic, because when a new attribute is added that should affect equality or ordering, only the defined method needs to change, and the generated methods will follow automatically.
Consistency also extends to hashing. Objects that compare equal should have the same hash, or set and dictionary operations will behave incorrectly, finding one equal object but not another. That relationship between equality and hashing is a protocol-level concern, not just a method-level one, and it should be addressed in class design by defining comparison as a group, with a single source of truth, whether through a helper or through careful manual implementation with tests that check consistency across operations.
Testing comparison methods should therefore check logical relationships, not just individual operations in isolation. A test should verify that a == b implies not (a != b), that a < b implies a <= b and not (a == b), and similar relationships that should hold for any well-defined comparison. Those grouped tests verify the protocol as a whole, rather than just its individual methods, which is a testing discipline that applies to all protocols where related methods must work together consistently to provide a coherent behaviour.
from functools import total_ordering
@total_ordering
class Book:
def __init__(self, title):
self.title = title
def __eq__(self, other):
return isinstance(other, Book) and self.title == other.title
def __lt__(self, other):
return isinstance(other, Book) and self.title < other.title
print(Book("A") < Book("B"))Comparison consistency also matters for sorting stability and for deduplication, because a list that is sorted and then deduplicated relies on equality and ordering agreeing about which objects are the same. If equality says two objects are the same but ordering says one is less than the other, then sorting and deduplication may produce different results depending on the order in which they are applied, which is confusing and hard to debug. That agreement, between equality and ordering, is therefore not just a theoretical nicety, but a practical requirement for programs that sort, search, and deduplicate collections of custom objects.
Check: Why should comparison methods be defined and tested as a consistent group, rather than individually?
Section 5
The container protocol defines how an object behaves like a collection, through methods such as __len__, which returns the number of items, __contains__, which implements the in operator for membership testing, __getitem__, which implements indexing and slicing for accessing items, and __iter__, which implements iteration for looping over items. Together, those methods allow a custom class to be used like a built-in collection, with familiar syntax such as len(), in, obj[i], and for item in obj, without needing to learn collection-specific method names for those basic operations.
That familiarity is a major benefit, because it allows a custom collection to integrate with Python's language and with generic functions that expect collections, such as sorting, filtering, and summarising tools, which work with any object that supports the container protocol, or a subset of it, without needing to know its specific class. That is polymorphism through protocols rather than through inheritance: the protocol is the interface, and any class that implements it can be used interchangeably with others that implement it, in contexts that rely only on the protocol, not on the specific class.
Implementing the container protocol completely, or at least the subset relevant to a class's domain, is important, because partial implementation can mislead callers. A collection that supports len() but not iteration may surprise a caller who reasonably expects both, based on experience with built-in collections where all container operations work together as a coherent set. Generic code often assumes a protocol based on one operation succeeding, and then uses other operations from the same protocol without checking whether they are implemented, which is a reasonable assumption for built-ins where protocols are complete, but which can fail for custom classes with partial support.
To avoid that surprise, either implement the full container protocol or document clearly which parts are implemented and which are not, and why. That documentation responsibility applies to all protocol implementations, but it is especially relevant for containers, because users have strong expectations about which operations should work together, based on their experience with built-ins, and a custom collection that violates those expectations without warning will cause bugs that are hard to trace to their source in partial protocol support.
class Shelf:
def __init__(self, books):
self.books = books
def __len__(self):
return len(self.books)
def __contains__(self, book):
return book in self.books
def __getitem__(self, index):
return self.books[index]
print(len(Shelf(["A", "B"])), "A" in Shelf(["A", "B"]))Container methods also interact with each other in useful ways: defining __getitem__ with integer indices can make an object iterable by default in older Python code, and defining __len__ and __getitem__ together can support slicing and other sequence operations automatically. Those interactions are convenient, but they should not be relied on as a substitute for explicitly defining __iter__, because explicit iteration is clearer, more predictable, and less likely to surprise callers who expect a specific traversal order. Relying on implicit iteration through indexing, for example, can be slow for large collections, because it accesses items one by one through indexing, rather than through an efficient iterator.
Check: What methods make up the container protocol, and why should a custom collection implement them consistently?
Section 6
The iteration protocol defines how an object produces a sequence of values, one by one, through an iterator. An iterable is an object with a __iter__ method that returns an iterator, while an iterator is an object with a __next__ method that returns the next value, or raises StopIteration when there are no more values. That distinction matters for design, and it will be covered in depth in the next chapter, but it is introduced here as part of protocols, because iteration is a protocol that many classes should support to be usable in for loops and in other iteration contexts.
Iteration contexts in Python are numerous: for loops, comprehensions, unpacking, and built-in functions like sum and list all rely on the iteration protocol to obtain values from an object, one by one, without needing to know its internal structure or its specific class. That is polymorphism through iteration, where any iterable can be used interchangeably in an iteration context, regardless of its class, as long as it provides __iter__ correctly. That abstraction allows generic code to work with many kinds of collections and sequences without special-casing each one.
For custom collections, iteration should traverse the collection's internal structure in a defined order, without exposing that structure itself. The traversal logic lives inside the class, in its __iter__ and __next__ methods, while callers simply use a for loop, without needing to know how traversal works internally. That separation between interface and implementation is central to protocol design: the interface is simple, a for loop, while the implementation can be as complex as needed to traverse a custom structure, such as a tree or a linked list, correctly and efficiently.
A key design decision is whether an iterable is also an iterator, or whether it returns a new iterator each time. An iterable that returns a new iterator can be iterated multiple times, independently, which is usually what callers expect from a collection. An iterator, by contrast, can typically be used only once, because its state advances with each __next__ call and is not reset automatically. Mistaking a one-shot iterator for a reusable iterable is a common confusion, and it should be avoided by designing collections as iterables that return fresh iterators, and by documenting clearly when a class is a one-shot iterator rather than a reusable collection.
That decision, between combined and separated iterable and iterator designs, will be revisited in the next chapter with more detail and examples, including the trade-offs between them and how to avoid common pitfalls, such as an iterator that can only be used once being mistaken for an iterable that can be used multiple times.
class Countdown:
def __init__(self, start):
self.start = start
def __iter__(self):
return iter(range(self.start, 0, -1))
print(list(Countdown(3)))Check: What is the difference between an iterable and an iterator in the iteration protocol?
Section 7
Numeric protocols define how objects behave in arithmetic, through dunder methods such as __add__, __sub__, __mul__, and __truediv__, for addition, subtraction, multiplication, and division respectively. They also include reflected methods, such as __radd__, which are called when the left operand does not know how to handle the operation with the right operand's type, allowing the right operand to handle it instead. That mechanism supports operations between different types, such as a custom measurement object and a number, where either operand may appear on either side of the operator.
Reflected methods are essential for smooth integration with built-in numbers. An expression such as measurement + 5 uses the measurement's __add__, while 5 + measurement uses the measurement's __radd__, because the number's own addition does not know how to handle a measurement. If only __add__ is defined, the second expression will fail, even though the first succeeds, which is a surprising inconsistency that is easy to overlook when testing only one order of operands. For that reason, both orders should be tested explicitly for every numeric operation a class claims to support.
That completeness requirement is similar to the completeness requirement for container and iteration protocols, where partial implementation can lead to surprising failures. A numeric class that supports addition in one order but not the other is only partially implementing the numeric protocol, and callers who write a natural expression with the operands in the other order will encounter a failure they did not expect, based on the success of the first order. Documentation should therefore state which operations are supported, and tests should cover both operand orders, to ensure that the documented support is real in both directions.
Numeric protocols also include methods for other operations, such as comparison, conversion to numbers, and rounding, each with its own dunder method, and each contributing to a class feeling like a number in Python expressions. A well-designed numeric class implements the operations that make sense for its domain, documents which ones it supports, and tests them in both operand orders where applicable, so that callers can use it naturally in arithmetic without needing to know about reflected methods or about the internal details of numeric dispatch.
class Money:
def __init__(self, amount):
self.amount = amount
def __add__(self, other):
if isinstance(other, Money):
return Money(self.amount + other.amount)
return NotImplemented
def __repr__(self):
return f"Money({self.amount})"
print(Money(5) + Money(3))Numeric classes should also consider how they handle operations with incompatible types, by returning NotImplemented rather than raising an exception immediately, so that Python can try the other operand's reflected method before giving up. Returning NotImplemented is a signal to Python's dispatch machinery, not a final failure, and it allows a well-designed interaction between two custom types, each of which may know how to handle the operation, even if the first one tried does not. That cooperative dispatch is part of what makes numeric protocols work smoothly between different types.
Check: Why does a numeric class need reflected methods like __radd__ in addition to normal methods like __add__?
Section 8
The context manager protocol, which will be covered in depth in a later chapter, is introduced here as a protocol because it illustrates how dunder methods can define a protocol for resource management. It uses __enter__, which runs when entering a with block, and __exit__, which runs when exiting it, whether normally or through an exception. Together, those two methods allow a custom class to be used in a with statement for managing a resource, such as a file, a lock, a connection, or a temporary state change, where acquisition happens in __enter__ and release happens in __exit__, guaranteed even if an exception occurs inside the block.
That guarantee is the same one the with statement provides for built-in resources such as files, and custom context managers extend it to custom resources, allowing them to be used with the same clean syntax and the same guarantee of cleanup. That guarantee is powerful, because manual acquisition and release, without a with statement, requires explicit try and finally blocks in every place where the resource is used, which is verbose and easy to forget in one place, leading to a leaked resource. A context manager centralises acquisition and release in the class itself, rather than in every caller, which is encapsulation applied to resource lifecycle.
The __exit__ method receives information about any exception that occurred inside the block, and it can return a truthy value to suppress that exception, or a falsy value to let it propagate. Suppression should be rare and documented clearly when it is intended, because suppressing an exception silently can hide a failure from callers who expect it to propagate, which is surprising behaviour that should be avoided unless suppression is the documented purpose of the context manager, such as a manager that intentionally ignores a specific, expected, harmless exception type in a specific context.
For most context managers, __exit__ should simply clean up and return a falsy value, letting any exception propagate normally, while still guaranteeing cleanup. That behaviour, cleanup without suppression, is the common case, and it should be the default design, with suppression reserved for the rare cases where it is genuinely needed and clearly documented, so callers are not surprised by an exception they expected to see being hidden by a context manager they used without understanding its suppression behaviour.
class Timer:
def __enter__(self):
print("enter")
return self
def __exit__(self, exc_type, exc, tb):
print("exit")
return False
with Timer():
print("inside")Context managers should also be tested with exceptions, not just with normal exits, because their most important behaviour, guaranteed cleanup, only becomes visible when an exception occurs. A test that enters a context manager, raises an exception inside it, and then checks that cleanup happened and that the exception propagated correctly, verifies the protocol's core guarantee in the situation where it matters most, which is during failure rather than during success.
Check: What do __enter__ and __exit__ do in the context manager protocol, and why must __exit__ run even on exception?
Section 9
Callable objects are objects that can be called like functions, through the __call__ method, which is invoked when an object is used with parentheses, as in obj(). That allows a class to define objects that behave like functions but also carry state between calls, such as a counter that increments on each call, a formatter that remembers its configuration across calls, or a validator that counts how many values it has checked. Those uses are cases where a plain function, with no state, would be insufficient, because it could not remember information between calls without using global variables or closures.
Closures are an alternative way to carry state, by capturing variables from an enclosing scope, but they hide that state inside the function object, rather than exposing it as attributes that can be inspected or modified by callers who may need to check or reset the state between uses. A callable object, by contrast, makes its state visible as attributes, and its behaviour explicit as a method invoked through call syntax, which is a clean way to combine function-like invocation with object-like state in a single entity. That entity can be passed to code expecting a function, such as a callback or a sorting key, while also maintaining state across calls, which is a flexibility that plain functions do not provide without additional machinery.
That dual interface, callable as a function and manageable as an object, is a benefit of callable objects that should be considered when designing a function-like entity that needs to remember state and to expose that state for inspection or management. A callable object can be used interchangeably with a plain function in a context expecting a callable, which is a protocol in itself, the callable protocol, defined by __call__ and by Python's built-in callable() function, which checks whether an object can be called regardless of whether it is a function or an object with __call__.
Generic code that accepts callbacks should check, where appropriate, that a callback it receives is indeed callable before trying to call it, and should give a clear error at acceptance time if it is not, rather than a confusing error at call time far from where the callback was provided. That defensive check, like other protocol checks, can prevent confusing failures later, when the protocol is actually used and the cause, a missing method, may not be obvious from the failure itself, which may appear as a generic TypeError rather than as a clear message about a missing protocol method.
class Counter:
def __init__(self):
self.count = 0
def __call__(self):
self.count += 1
return self.count
c = Counter()
print(c(), c())A callable object can also be reset or inspected between uses, which is useful in testing, where a test may need to check how many times a callback was called, or to reset its count before a new test run.
Check: When is a callable object, with state and a __call__ method, a better choice than a plain function?
Section 10
The first ten sections of this chapter have introduced dunder methods and protocols as a system, and it is worth pausing to see how the pieces fit together as a coherent view of Python's data model. Built-in operations dispatch to dunder methods, protocols group those methods into kinds of behaviour, such as containers, iterables, numerics, callables, and context managers, and custom classes participate in the language by defining those methods, rather than by inheriting from specific parents. That flexible, duck-typed approach to polymorphism allows any class to work with generic code, as long as it provides the methods that generic code expects, which is a powerful idea that underlies much of Python's design.
Many of Python's features are protocol-based in that sense: for loops, with statements, len(), in, indexing, arithmetic, and calling all dispatch to dunder methods, and work with any object that implements the corresponding protocol, regardless of its class. That unifying principle explains why Python feels consistent across its built-in types and custom types, and why learning protocols, rather than just learning individual dunder methods, gives a deeper understanding of the language as a whole, rather than a list of special cases each to be memorised separately without seeing how they relate to each other and to the language's design.
The remaining sections of this chapter will cover additional protocols, such as attribute access, descriptor, and hashing protocols, and will explore how protocols interact, such as how iteration relates to containers, or how comparison relates to hashing, with examples showing how protocols work together in a single class to provide coherent, complete behaviour in many different contexts. A class that is printed, compared, iterated, and used as a context manager, all through different protocols, each implemented by a few dunder methods, can feel like a natural part of Python, rather than a foreign object requiring special handling in each context, which is the ultimate goal of protocol implementation.
That goal, making custom classes indistinguishable in usability from built-in types in the contexts where they are meant to be used, is a high standard, but it is achievable for well-designed classes that implement protocols completely, consistently, and with clear documentation of which protocols they support and how they support them. That documentation is information callers need to use a class effectively in generic contexts, where its protocol support is assumed rather than checked explicitly for each operation, which is a common pattern in Python where generic code assumes a protocol based on one operation succeeding and then uses other operations from the same protocol without checking whether they are implemented.
class Book:
def __init__(self, title):
self.title = title
def __len__(self):
return len(self.title)
print(len(Book("Dune")))That assumption is reasonable for built-ins, where protocols are complete.
Check: How does viewing dunder methods as protocols, rather than isolated special methods, change how you design a class?
Section 11
Attribute access in Python is itself a protocol, governed by dunder methods such as __getattr__, __getattribute__, and __setattr__. Normally, attribute access is simple: reading obj.name looks up name in the object's instance dictionary, then in its class, and writing obj.name = value stores the value in the instance dictionary. That default behaviour is sufficient for most classes, and you should not override it without a clear reason, because custom attribute access can make a class harder to understand, harder to debug, and slower, as every attribute access must go through your custom logic, even for attributes that do not need it. Understanding the default, before customising it, is therefore essential, because it tells you what you are changing and why. __getattr__ is called only when normal lookup fails, that is, when an attribute is not found through the usual mechanisms. That makes it useful for delegation, as discussed in the previous chapter, where a wrapper forwards unknown attributes to a contained object, or for lazy loading, where an attribute is computed on first access and then cached, rather than computed eagerly at creation. Because __getattr__ is only called on failure, it does not interfere with normal attributes, which continue to work as usual, and it is therefore safer and clearer than overriding all attribute access, which affects every attribute, whether it needs custom handling or not. That safety, through fallback-only invocation, is why __getattr__ is the preferred tool for dynamic attributes, when they are genuinely needed. __getattribute__, by contrast, is called for every attribute access, whether the attribute exists or not, and overriding it requires great care, because it is easy to cause infinite recursion, by accessing an attribute inside __getattribute__ itself, which calls __getattribute__ again, and again, without end. For beginners, the guideline is simple: use __getattr__ for fallback, use properties for controlled access to specific attributes, and avoid __getattribute__ unless you have a compelling reason and understand its recursion risks. __setattr__ is called for every attribute assignment, and it can be used to validate values, to transform them, or to trigger side effects on assignment, similar to property setters, but applied to all attributes rather than to a specific one. Like __getattribute__, it requires care to avoid recursion, because assigning to an attribute inside __setattr__ will call __setattr__ again, unless the assignment is done through the parent's method or through direct dictionary manipulation. For most classes, property setters are a clearer way to control assignment for specific attributes, because they are explicit, per-attribute, and easier to understand than a single __setattr__ that handles all attributes with branching logic. Reserve __setattr__ for cases where many attributes need the same handling, such as a class that must log all assignments, or that must validate all values against a common rule, and document that handling clearly, so readers understand why assignment behaves unusually for that class.
class Lazy:
def __getattr__(self, name):
if name == "value":
return 42
raise AttributeError(name)
print(Lazy().value)Check: What is the difference between __getattr__, which is called only on lookup failure, and __getattribute__, which is called for every access?
Section 12
Descriptors are objects that define how attribute access works for a specific attribute, through methods such as __get__, __set__, and __delete__, and they are the mechanism behind properties, which are themselves descriptors, as well as behind other Python features, such as methods, which are descriptors that bind a function to an instance, and class methods and static methods, which are descriptors with different binding behaviour. Understanding descriptors, at a basic level, helps you understand why properties work, why methods receive self automatically, and how you can define your own reusable attribute behaviour, such as validation, that can be shared across many classes, by defining a descriptor class once and using it as an attribute in many classes, each of which will then have that validated attribute, without duplicating validation logic in each class. That reusability, through descriptors, is a powerful form of abstraction, where attribute behaviour itself becomes an object, with its own methods, that can be tested, documented, and reused independently of the classes that use it. A simple descriptor example is a validated attribute, such as a positive number, where the descriptor's __set__ method checks that the assigned value is positive, raising an exception otherwise, and stores it in the instance's dictionary under a private name, while its __get__ method retrieves it, returning a default if it has not yet been set. That descriptor, once defined, can be used in any class that needs a positive number attribute, by assigning an instance of the descriptor to a class attribute with the desired name, and from then on, attribute access on instances of that class will go through the descriptor's methods, rather than through direct dictionary access, which means validation happens automatically, on every assignment, without the class needing to define a property for that attribute, or to repeat validation logic in its __init__ or in its methods. That automatic validation, through a reusable descriptor, is a clean way to enforce invariants for attributes that appear in many classes, and it illustrates how descriptors turn attribute behaviour into a reusable component, rather than a per-class implementation detail. That precedence rule, while advanced, is worth knowing, because it explains why properties cannot be shadowed by instance assignment, while methods can be, and why a descriptor's kind, data or non-data, matters for its behaviour in a class, beyond the methods it defines, which helps you predict how a descriptor will behave in a real class. For beginners, descriptors are a concept to recognise, rather than a tool to use frequently, because most attribute control needs can be met with properties, which are simpler and more familiar, and because defining custom descriptors requires understanding lookup precedence, instance dictionaries, and the descriptor protocol in detail, which is a significant investment for a benefit that only becomes clear when the same attribute behaviour is needed in many classes.
class Positive:
def __set_name__(self, owner, name):
self.name = "_" + name
def __get__(self, obj, objtype=None):
return getattr(obj, self.name, 0)
def __set__(self, obj, value):
if value <= 0:
raise ValueError("must be positive")
setattr(obj, self.name, value)
class Item:
price = Positive()
item = Item()
item.price = 10
print(item.price)Check: How does a descriptor turn attribute behaviour, such as validation, into a reusable component shared by many classes?
Section 13
Hashing is the protocol that lets an object be used in a set or as a dictionary key, and it works through the __hash__ method, which returns an integer for the object. Sets and dictionaries do not search every stored object when they look something up. Instead they use the hash to choose a bucket where the object should live, and they then search only inside that bucket, which is why membership tests and lookups stay fast even for large collections. For that shortcut to work, equality and hashing must agree with each other. Objects that compare equal must have the same hash, because otherwise a set could store two equal objects in different buckets and fail to recognise them as duplicates.
The standard way to keep the two operations in agreement is to derive both of them from the same attributes. If equality compares the x and y coordinates of a point, then hashing should hash a tuple of those same coordinates, using the built-in hash function on that tuple. Equal points will then have equal coordinates and therefore equal hashes, while points with different coordinates will usually have different hashes. A hash need not be unique, since collisions are handled by checking equality after hashing.
The second rule in the Check question is just as important as the first. A hash must not change while an object is stored in a set or used as a dictionary key. The reason is the bucket shortcut described above, because if an object moves to a different bucket after its hash changes, then the set will keep looking in the old bucket and will fail to find it. The object may then appear to vanish, even though it is still stored, and attempts to remove it or to test membership may fail for no obvious reason. This problem arises when a hash depends on a mutable attribute that is changed while the object is already in a collection.
The safest design is to make hashable objects effectively immutable, or at least to base their hash only on attributes that will not change. A point with fixed coordinates is a good example, because its coordinates define both its equality and its hash and they are not expected to change. If a class must be mutable and must also define equality, then it is often better to leave it unhashable, so that it cannot be placed in a set by mistake, rather than to allow a hash that may change.
The example below shows a correct pairing of equality and hashing for a point class. Both methods use the same pair of coordinates, so equal points hash equally and a set correctly treats two equal points as one item.
class Point:
def __init__(self, x, y):
self.x, self.y = x, y
def __eq__(self, other):
return isinstance(other, Point) and (self.x, self.y) == (other.x, other.y)
def __hash__(self):
return hash((self.x, self.y))
print(len({Point(1, 2), Point(1, 2)}))
print(hash(Point(1, 2)) == hash(Point(1, 2)))Testing a hashable class should check the pairing in context rather than checking hash values alone. A good test confirms that equal objects have equal hashes, that a set removes duplicates as expected, and that a dictionary retrieves a value stored under an equal key.
Check: Why must objects that compare equal have the same hash, and why should a hash not change while an object is in a set?
Section 14
Boolean conversion in Python is governed by two dunder methods: __bool__ and, as a fallback, __len__. When an object is used in a boolean context, such as an if statement, a while condition, or a boolean operation, Python first looks for __bool__. If it is defined, its return value decides truthiness. If it is not defined but __len__ is, an object is falsy when its length is zero and truthy otherwise. If neither is defined, every object is truthy. That fallback chain is simple, but its consequences matter, because a class that defines neither method will always test as true, even when a reader might expect an "empty" object to test as false.
The most natural use of truthiness is for collections: an empty shelf is falsy, a shelf with books is truthy. That matches built-in behaviour, where an empty list, dictionary, or string is falsy. A class that models a collection should therefore usually get its truthiness from __len__ alone, without defining __bool__ at all. The fallback does the right thing automatically, and readers will understand it immediately because it matches what they already know from built-ins. Defining __bool__ for a collection, to mean something else, would break that expectation and should be avoided.
For non-collection classes, truthiness needs more thought. A task object, for example, might be truthy when it exists, regardless of its state, which is the default behaviour and probably correct. Defining __bool__ to mean "is pending" would be surprising: generic code that tests if task: would then be testing pendingness, not existence, and a filter or comprehension using truthiness would include or exclude tasks for a reason its author did not intend. When a domain condition like pendingness matters, it deserves an explicit method, such as is_pending(), that callers invoke deliberately. Explicit methods are clearer than overloading an implicit conversion with domain meaning.
There are cases where __bool__ is genuinely appropriate. A result object might be truthy when it holds a value and falsy when it holds an error, or a configuration object might be falsy when it is disabled. In such cases, the meaning should be obvious from the class's purpose, documented in the class docstring, and consistent with how the object is described. Even then, ask whether an explicit method would be clearer. If the answer is yes, prefer the explicit method and leave truthiness at its default.
Testing truthiness is straightforward: check that an empty collection is falsy, a non-empty one is truthy, and that a non-collection object behaves as documented. Also test truthiness in the contexts where it is actually used, such as if, while, and, or, and not, and in filters, where a surprising definition can silently drop items. A truthiness definition that is tested only in isolation, with bool(obj), may hide surprises that appear when generic code relies on it in a comprehension or a filter condition.
class Shelf:
def __init__(self, books):
self.books = books
def __len__(self):
return len(self.books)
print(bool(Shelf([])), bool(Shelf(["A"])))Check: When should you define __bool__ explicitly, and when should you let truthiness follow from __len__ instead?
Section 15
The __dir__ method controls what dir() returns for an object. By default, dir() lists attributes from the instance dictionary, the class, and its parents, which is usually exactly what interactive users and tools need. Customising __dir__ is rarely necessary, and it carries a consistency obligation: if a class generates attributes dynamically through __getattr__, then dir() should list those attributes, and every attribute it lists should actually be accessible. A mismatch in either direction confuses users and tools.
Consider a lazy object that computes attributes on first access. If __getattr__ can supply an attribute called summary, but dir() does not list it, an interactive user exploring the object will never discover it. Conversely, if __dir__ lists an attribute that __getattr__ cannot supply, the user will try it, get an AttributeError, and distrust the listing. Both mismatches are bugs in the class's public interface, even though neither affects code that already knows the attribute names. Discovery tools, documentation generators, and IDE autocompletion all rely on dir(), so an incorrect listing misleads more than just interactive users.
The safest approach for most classes is to leave __dir__ alone. The default is correct for classes whose attributes are stored normally, and dynamic attributes are uncommon enough that most classes never need a custom listing. If your class does have dynamic attributes, and discovery through dir() matters for usability, then implement __dir__ to return the default list plus the dynamic names, and derive both __dir__ and __getattr__ from the same source of truth about which dynamic attributes exist. A single source prevents drift, where one method is updated and the other is forgotten as the class evolves.
That single-source principle applies to protocols generally: related methods should agree, and where possible should derive from a common definition rather than duplicating logic independently. For __dir__ and __getattr__, the common definition might be a list or a function that names the dynamic attributes. Changing that list then updates both methods at once, which is far safer than editing two separate implementations and hoping they stay in sync. If you cannot arrange a single source, document the pairing clearly so a future maintainer knows that changing one method requires changing the other.
In short, treat __dir__ as a promise about what an object offers. Do not make that promise unless you need to, and when you do, keep it accurate by tying it to the same logic that actually supplies the attributes.
class Book:
def __init__(self, title):
self.title = title
print([name for name in dir(Book("Dune")) if not name.startswith("_")])A practical test for __dir__ correctness is to compare its output with the attributes that actually work: for each name it lists, check that accessing it succeeds, and for each dynamic attribute the class supplies, check that it appears in the listing. That two-way test catches both kinds of mismatch, and it is cheap to write for a class with only a handful of dynamic attributes.
Check: Why should a custom __dir__, if defined, agree with __getattr__ about which dynamic attributes exist?
Section 16
The sequence protocol describes ordered collections: items have positions, can be accessed by index, can be sliced, and can be iterated in order. Its core methods are __getitem__, __len__, and __iter__, with __setitem__, __delitem__, and __contains__ added for mutable sequences. Together, they let a custom class be used like a list or tuple in sequence contexts: indexing, slicing, unpacking, iteration, and membership testing. That familiarity is the protocol's benefit, and also its risk.
The risk is partial implementation. A class that supports indexing but not slicing, or iteration but not length, will surprise callers who expect the full protocol. Generic sequence code reasonably assumes that if one sequence operation works, the others work too, because that is true for built-ins. A custom sequence that breaks that assumption will fail in code that looks correct, with an error that points at the calling code rather than at the missing method. The failure may not appear until a different part of a program, written by a different person, uses an operation the original author never tested.
There are two acceptable responses to this risk. The first is to implement the full protocol, including slicing in __getitem__ by handling slice objects as well as integers. Slicing support is not difficult: check whether the index is a slice, and if so, delegate to the underlying sequence's slicing. The second acceptable response is to implement only part of the protocol, but to document clearly which parts are supported and which are not, and why. A one-way collection that cannot be randomly accessed, for example, should not pretend to be a sequence at all; it should present itself honestly as an iterable, which is a weaker, clearly different promise.
What is not acceptable is silent partial implementation: appearing to be a sequence, without saying that some sequence operations are missing. Documentation is the remedy, and it should be in the class docstring, where users will see it before they write code that relies on unsupported operations. Testing should cover every operation the class claims to support, including edge cases such as negative indices, out-of-range indices, and empty slices, because those are where sequence implementations most often diverge from built-in behaviour.
The broader lesson, repeated across this chapter, is that a protocol is a promise. Make the promise completely, or make its limits explicit. Never let users discover the limits by accident, in code that looked reasonable.
class Shelf:
def __init__(self, books):
self.books = books
def __getitem__(self, index):
return self.books[index]
def __len__(self):
return len(self.books)
print(Shelf(["A", "B"])[1])Slicing also interacts with assignment and deletion for mutable sequences: a class that supports __getitem__ with slices but not __setitem__ with slices will again surprise callers who expect slice assignment to work once slice access does. Test slice operations as a group, not one at a time.
Check: Why is partial sequence support, such as indexing without slicing, potentially misleading for users familiar with built-in sequences?
Section 17
The mapping protocol describes dictionary-like objects: access by key rather than by position. Its core methods are __getitem__ with a key, __setitem__ with a key and a value, __delitem__ with a key, __contains__ for key membership, __len__ for the number of entries, and __iter__ for iterating over keys, usually accompanied by keys(), values(), and items() methods. Together, they let a custom class be used with familiar syntax such as obj[key], obj[key] = value, key in obj, and for key in obj, in generic code that expects a mapping.
A common design question is whether to inherit from dict or to implement the mapping protocol through composition, holding an internal dictionary and forwarding operations to it. Inheritance looks simpler at first, because dict's methods come for free, but it has a well-known weakness: some built-in dictionary operations are implemented in C and may bypass overridden Python methods, so validation or custom behaviour added in a subclass may not apply to every operation. A subclass that validates values in __setitem__, for example, may find that other ways of populating the dictionary skip that validation entirely. That bypass is subtle, hard to test comprehensively, and can break invariants that the class was designed to maintain.
Composition avoids the problem. When a class holds an internal dictionary and exposes only the mapping methods it chooses, every access goes through its own methods, so validation applies uniformly and invariants hold without exception. Composition also lets the class hide dictionary methods it does not want to expose, rather than inheriting a large interface that may not fit its domain. The cost is writing a few forwarding methods, which is small compared with the reliability gained.
That said, inheriting from dict is not always wrong. For a simple dictionary subclass that adds a convenience method, without changing storage semantics or adding validation, inheritance is fine and idiomatic. The distinction is whether the subclass changes how the mapping behaves. If it only adds behaviour, inheritance is acceptable. If it restricts or validates behaviour, composition is safer, because it guarantees that the restriction applies to every operation, not just the ones that happen to call the overridden method.
Whichever route you choose, test the mapping through all the operations it claims to support, including methods added by generic code, such as get and update if you provide them, and iteration over keys, values, and items. A mapping that behaves correctly for [] access but incorrectly for update will surprise callers who reasonably expect the two to agree.
class Config:
def __init__(self):
self.data = {}
def __getitem__(self, key):
return self.data[key]
def __setitem__(self, key, value):
self.data[key] = value
def __len__(self):
return len(self.data)
c = Config()
c["theme"] = "dark"
print(c["theme"], len(c))Check: Why is implementing the mapping protocol through composition usually safer than inheriting from dict for a custom mapping with validation?
Section 18
A class rarely implements just one protocol. A realistic collection class might implement representation, comparison, container, sequence, and iteration protocols at once, so that it can be printed, compared, measured, indexed, and looped over, all with familiar syntax. Designing such a class well means thinking about how its protocols interact, not just about each one in isolation, because an inconsistency between protocols is a bug even when each protocol looks correct on its own.
Consider a shelf of books. Its __repr__ should show its contents clearly. Its __eq__ should compare shelves by their books, and its __hash__ decision should follow from that: if shelves are mutable, they should probably be unhashable, since a hash based on mutable contents could change. Its __len__ gives its size, its __getitem__ supports indexing and slicing, its __iter__ yields books in order, and its __contains__ tests membership. Each method is simple on its own. The design work is in making them agree: equality by contents implies that two shelves with the same books compare equal, that iteration order matches the order used by indexing, and that membership testing agrees with what iteration yields. If __contains__ uses a different definition of membership from __iter__, a caller who finds a book by iterating but not by testing membership will be justifiably confused.
Another interaction is between __iter__ and __getitem__. Python can fall back to __getitem__ with increasing indices for iteration when __iter__ is missing, but defining __iter__ explicitly is clearer and avoids surprises with classes whose indexing does not start at zero or is not contiguous. Similarly, defining __len__ gives truthiness for free through the fallback chain, so a collection class usually does not need __bool__ at all. Knowing these fallbacks lets you implement fewer methods while getting more behaviour, but only when the fallback behaviour is exactly what you want, which is worth verifying with tests rather than assuming.
The practical approach is to design the class's protocols together, on paper or in a docstring, before implementing them: list the protocols the class supports, the methods each requires, and the invariants that connect them, such as iteration order matching indexing order. Then implement method by method, testing each protocol and each interaction as you go. That upfront list also becomes documentation, telling users exactly which protocols the class supports, which is information they need when using it in generic code that assumes a protocol based on one successful operation.
class Shelf:
def __init__(self, books):
self.books = list(books)
def __repr__(self):
return f"Shelf({self.books!r})"
def __len__(self):
return len(self.books)
def __iter__(self):
return iter(self.books)
def __contains__(self, book):
return book in self.books
shelf = Shelf(["A", "B"])
print(shelf, len(shelf), "A" in shelf, list(shelf))That upfront list is also useful during code review, because a reviewer can check each protocol against it without guessing which protocols the class was meant to support.
Check: Why should the protocols of a single class be designed together, rather than each in isolation?
Section 19
Copying is governed by two protocols: __copy__ for shallow copies and __deepcopy__ for deep copies, used by the copy module's copy() and deepcopy() functions. A shallow copy creates a new object with the same attribute values as the original, but mutable attributes are shared between the original and the copy, not copied themselves. A deep copy creates a new object and recursively copies all of its mutable attributes as well, so the copy is fully independent, with no shared mutable state. Choosing between them is a decision about independence: should changes through the copy affect the original?
A copy made for editing, where changes should not affect the original until explicitly saved, needs deep copying, or at least copying of the mutable parts that will be edited. A copy made for temporary use, where sharing is acceptable, can use shallow copying, which is faster and uses less memory, especially for large or deeply nested objects where recursive copying would be expensive. The trade-off is between independence and efficiency, and it should be made deliberately, based on how the copy will actually be used, not by habit or by whichever function happens to be at hand.
For most ordinary classes, the defaults work well: shallow copying creates a new object with a shallow copy of its attribute dictionary, and deep copying recursively copies that dictionary. Custom __copy__ or __deepcopy__ methods become necessary when a class holds resources that should not be copied by duplicating internal state, such as a file handle or a network connection, where a copy should either raise an error, because copying makes no sense, or create a new resource, such as opening a new connection, rather than sharing the old one. Those are domain decisions that only the class author can make, and they should be documented, so callers know whether a copy they obtain is shallow or deep, and whether its mutable attributes are shared or independent.
Testing copying should check independence, not just that a copy was created. For a shallow copy, verify that the copy is a distinct object but that its mutable attributes are shared, if that is the intended semantics. For a deep copy, verify that changing a nested mutable attribute through the copy does not affect the original, including for deeply nested structures, where a shallow copy inside a deep copy would be a subtle bug. Also test copying of objects with shared references inside them, where deep copying should preserve the sharing structure within the copy, rather than duplicating shared objects separately, which is a detail the copy module handles through a memo dictionary, and which custom __deepcopy__ implementations must respect by passing that memo along.
In short, copying is not just duplication: it is a statement about independence, and it should be designed and tested as such.
import copy
class Box:
def __init__(self, items):
self.items = items
original = Box([1, 2])
shallow = copy.copy(original)
print(shallow.items is original.items)Check: What is the difference between a shallow copy and a deep copy, in terms of shared mutable attributes?
Section 20
This chapter has covered many protocols, from creation and representation through comparison, containers, iteration, numerics, context managers, callables, attribute access, descriptors, hashing, boolean conversion, sequences, mappings, combined protocols, and copying. Each protocol has its own dunder methods, but they share common design principles, and reviewing those principles together is more useful than memorising any single protocol's method list, because the principles transfer to protocols you have not yet met, and to new protocols you may design yourself.
The first principle is completeness or explicit limits. Implement a protocol fully, or document clearly which parts are supported and which are not. Silent partial implementation misleads users who reasonably assume, from one successful operation, that the rest of the protocol works too. The second principle is consistency: related methods must agree. Equal objects must have equal hashes. Iteration order must match indexing order. Membership must agree with iteration. Comparison methods must agree with each other and with equality. Inconsistency between related methods is a bug, even when each method looks correct in isolation, and it is often discovered late, in code far from the class definition, where it is hard to trace.
The third principle is familiarity. Protocols exist so that custom classes can be used like built-ins, with familiar syntax and semantics. Deviating from built-in semantics, such as giving truthiness a domain-specific meaning, or making a container's membership test mean something other than presence, should be a deliberate, documented decision, not a casual choice, because generic code written for built-ins will misinterpret the deviation. When a domain meaning is needed, an explicit method is usually clearer than overloading an implicit protocol method.
The fourth principle is testing in context. Protocol methods should be tested not only in isolation, but in the generic contexts where they will actually be used: in sets and dictionaries for hashing, in sorting and deduplication for comparison, in with statements with exceptions for context managers, and in filters and comprehensions for truthiness. A method that passes an isolated test but fails in its real context has not really been tested.
Finally, remember that protocols are about duck typing: an object is usable in a context because it implements the right methods, not because it inherits from a particular class. That is what makes Python flexible, and it is also what makes protocol design a responsibility: your class's methods are its promise to every piece of generic code that will use it. Design that promise carefully, document it honestly, and test it thoroughly, and your classes will feel like natural parts of Python rather than foreign objects requiring special handling.
class Book:
def __init__(self, title):
self.title = title
def __repr__(self):
return f"Book({self.title!r})"
print(Book("Dune"))Keep that checklist beside you when you design your next class, and return to it whenever a class starts to feel awkward to use.
Check: How does the protocol view help you design a class that works coherently across many different Python contexts?
Section 21
The numeric tower in Python is built on protocols, not on a single numeric base class. Integers, floats, fractions, and decimals all support arithmetic through dunder methods such as __add__, __sub__, __mul__, and __truediv__, together with their reflected counterparts __radd__, __rsub__, and so on, which Python calls when the left operand does not know how to handle the operation. That reflected mechanism is what lets a custom numeric type work with built-in numbers: if you write 2 + my_number, Python first tries int.__add__, which does not know your type, and then tries my_number.__radd__, which does. Without the reflected method, the operation would fail, even though your type knows perfectly well how to add an integer. Designing a numeric class therefore means implementing operations in pairs: each forward method and its reflected partner, with consistent semantics. A forward method should handle operands it understands and return NotImplemented for operands it does not, rather than raising an exception, so that Python gets the chance to try the reflected method on the other operand. Returning NotImplemented is not a failure; it is a polite refusal that keeps the dispatch chain alive. Raising TypeError immediately, by contrast, ends the chain and may prevent a valid operation that the other operand could have handled. In-place operations add another layer. Methods such as __iadd__ let x += y modify x in place when that makes sense, as it does for mutable numeric-like objects, while immutable numeric types simply return a new object from __iadd__, which Python then rebinds to the name. For immutable types, defining __iadd__ to return a new object is correct, and omitting it is also acceptable, since Python falls back to __add__ followed by rebinding. Conversion protocols also matter for numeric types. __int__, __float__, and __index__ let a custom number be used where an integer or float is expected, such as in int(obj), float(obj), or as a list index. __index__ deserves special attention: it is used for indices and slice bounds, and it should return a plain integer, not just something that can be converted to one. Defining it for a numeric type that represents a count or a position makes that type usable as an index, which is a genuine convenience for callers, while defining it for a type where indexing makes no sense would be misleading. Testing a numeric class means testing operations in both orders, with both custom and built-in operands, and checking that unsupported combinations raise TypeError cleanly after both dispatch attempts have failed. Also test that in-place operations behave correctly for references: after x += y, check whether other names bound to the old object still see the old value, as they should for an immutable type. Those tests catch the most common numeric protocol mistakes, which are missing reflected methods and incorrect in-place semantics.
class Count:
def __init__(self, value):
self.value = value
def __add__(self, other):
if isinstance(other, Count):
return Count(self.value + other.value)
return NotImplemented
def __radd__(self, other):
if isinstance(other, int):
return Count(self.value + other)
return NotImplemented
print((Count(3) + 2).value, (2 + Count(3)).value)Check: Why must a numeric class implement reflected methods such as __radd__ as well as forward methods such as __add__?
Section 22
Comparison protocols become more interesting when a class defines a total ordering: less than, less than or equal, greater than, and greater than or equal, in addition to equality. Python does not require you to define all of them by hand. The functools.total_ordering decorator can fill in the missing comparison methods from __eq__ and one ordering method, such as __lt__, which reduces duplication and, more importantly, reduces the chance of inconsistency between methods written separately. Using it is a good example of letting a standard tool enforce consistency that hand-written methods might break. A total ordering must satisfy certain properties to be useful: it should be transitive, meaning that if a < b and b < c, then a < c, and it should be consistent with equality, meaning that if a == b, then neither a < b nor b < a. Violating transitivity breaks sorting, which assumes that comparing pairs is enough to order a whole collection, and violating consistency with equality breaks deduplication, which assumes that equal objects are interchangeable for ordering purposes. Choosing the ordering key is a design decision that should be documented. A book might be ordered by title, by author, or by publication year, and each choice gives a different sort order, with different consequences for users. The ordering should also be stable in the sense that it does not depend on mutable attributes that may change after objects have been sorted or stored in an ordered structure, for the same reason that a hash should not depend on mutable attributes. If the ordering key can change, sorting results can become stale, and structures that assume a stable order can become inconsistent, which is a bug that is hard to trace because the objects themselves look correct, only their order is wrong. It is also worth deciding whether a class should be orderable at all. Not every class has a natural order, and defining one arbitrarily, such as ordering tasks by their internal identifier, may mislead users into thinking the order is meaningful when it is not. For such classes, it is better to leave ordering undefined, so that attempting to sort raises a clear error, and to provide an explicit sorting key, such as a function that extracts a title, for callers who need a particular order. Sorting with an explicit key, using the key argument of sorted(), is often clearer than relying on a class's default ordering, because the key makes the ordering criterion visible at the call site. When you do define an ordering, test it thoroughly: test all six comparison operators, test transitivity with at least three objects, test consistency with equality, and test sorting a shuffled list and checking that the result is ordered and contains the same objects. Those tests are cheap and catch the most common ordering mistakes before they reach users.
from functools import total_ordering
@total_ordering
class Book:
def __init__(self, title):
self.title = title
def __eq__(self, other):
return isinstance(other, Book) and self.title == other.title
def __lt__(self, other):
return isinstance(other, Book) and self.title < other.title
print(sorted([Book("Z"), Book("A")])[0].title)Check: How does functools.total_ordering help keep a class's comparison methods consistent, and what ordering properties must still hold?
Section 23
Iteration protocols deserve a closer look, because they are among the most widely used protocols in Python, and because the distinction between iterables and iterators, introduced earlier, has practical consequences for how objects behave in loops, comprehensions, and unpacking. An iterable is an object that can produce an iterator, through __iter__. An iterator is an object that produces items one at a time, through __next__, and that raises StopIteration when it is exhausted. Every iterator is also an iterable, because its __iter__ returns itself, but not every iterable is an iterator, and confusing the two is a common source of bugs.
The most important consequence is reusability. A list can be iterated many times, because each iteration creates a fresh iterator over its contents. An iterator, by contrast, can typically be iterated only once, because it is consumed as it yields items, and once exhausted, it stays exhausted. Code that iterates over an iterator twice, expecting the second pass to repeat the first, will silently get no items on the second pass, which is a bug that produces empty results rather than an error. That silent failure makes it especially dangerous, because it may not be noticed until much later, when a result that should contain items is found to be empty.
Designing an iterable class therefore means deciding whether it should be reusable, which most collection-like classes should be, by returning a fresh iterator from __iter__ each time, or whether it is inherently single-use, such as a stream of data that can only be read once, in which case it should be documented as an iterator, not as an iterable, so that callers know not to iterate it twice. That documentation matters, because the distinction is not visible from the class's method list alone: both kinds of class define __iter__, and only the behaviour of repeated iteration reveals which kind they are.
Another consequence is that __iter__ should be cheap and side-effect free, because it may be called many times, implicitly, by loops, comprehensions, unpacking, and built-in functions such as list() and sum(). An __iter__ that performs expensive work, or that changes the object's state, will cause that work or that change to happen repeatedly, in contexts where the caller did not expect it, which is surprising and potentially expensive. If iteration genuinely requires expensive setup, that setup should be documented, and callers should be warned not to iterate casually, or the class should be redesigned so that iteration is cheap, with expensive work done explicitly, through a separate method, rather than implicitly, through __iter__.
Testing iteration should include repeated iteration, to verify reusability or single-use behaviour as documented, exhaustion behaviour, to verify that StopIteration is raised correctly and that a loop ends cleanly, and interaction with built-ins such as list(), sum(), and unpacking, to verify that the iterator works in generic contexts, not just in a hand-written for loop.
class Countdown:
def __init__(self, start):
self.start = start
def __iter__(self):
value = self.start
while value > 0:
yield value
value -= 1
print(list(Countdown(3)), list(Countdown(3)))Check: Why is the distinction between a reusable iterable and a single-use iterator important for code that iterates twice?
Section 24
Context manager protocols were introduced earlier, but they deserve a deeper treatment here, because they are a powerful way to manage resources, and because their error-handling behaviour is subtle. A context manager defines __enter__, which is called when execution enters a with block, and __exit__, which is called when execution leaves the block, whether normally or because of an exception. __exit__ receives information about any exception that occurred, or None values if the block completed normally, and it can suppress the exception by returning a true value, or let it propagate by returning a false value. That suppression power must be used with great care, because silently swallowing exceptions hides errors and makes debugging far harder.
The most common use of context managers is resource management: opening a file, acquiring a lock, or starting a transaction, and guaranteeing that the resource is released, the lock is freed, or the transaction is rolled back, even if an exception occurs inside the block. That guarantee is the protocol's core value, and it is why context managers are preferred over manual setup and cleanup, where an exception between setup and cleanup can easily leave a resource unreleased. A well-designed context manager makes the guarantee explicit: its documentation should state what resource it manages, what happens on normal exit, and what happens on exceptional exit, including whether it suppresses any exceptions, and if so, which ones and why.
Suppression should be rare and specific. A context manager that suppresses all exceptions, for any reason, is almost always a mistake, because it hides programming errors, such as bugs in the block's code, alongside the expected exceptions it was designed to handle. If suppression is needed, it should be limited to a specific exception type, documented clearly, and tested with both the suppressed exception and other exceptions, to verify that only the intended ones are suppressed and that all others propagate normally. A context manager that suppresses exceptions for convenience, without a clear justification, will cause more debugging pain than the convenience is worth.
Context managers can also be used for purposes beyond resource management, such as timing a block of code, redirecting output, or temporarily changing a setting and restoring it afterwards. Those uses follow the same pattern: setup in __enter__, cleanup in __exit__, with cleanup guaranteed even on exception. The pattern is so useful that Python provides the contextlib module, covered in a later chapter, for building context managers from functions, but understanding the underlying __enter__ and __exit__ protocol remains essential, because it explains how with statements work, why cleanup is guaranteed, and when a class-based context manager is clearer than a function-based one, such as when the manager needs to hold complex state across the block.
Testing a context manager must include the exceptional path, not just the normal path, because its guarantee only matters when an exception occurs. Test that cleanup happens on exception, that the exception propagates unless suppression is intended, and that the manager can be reused or not, as documented, since some managers are single-use and should not be entered twice.
class Timer:
def __enter__(self):
print("start")
return self
def __exit__(self, exc_type, exc, tb):
print("stop")
return False
with Timer():
print("work")Check: When should a context manager's __exit__ suppress an exception, and when should it let the exception propagate?
Section 25
Callable objects, introduced earlier, become more powerful when combined with state. A class that defines __call__ can hold configuration in its instance attributes and use that configuration each time it is called, which makes it a natural way to create customised functions, such as a validator with a specific threshold, a formatter with a specific style, or a counter with a specific starting value. Unlike a plain function, a callable object can be inspected, reset, and reconfigured between calls, because its state lives in its attributes, which are accessible to callers and to tests. That inspectability is a real advantage for debugging and testing, where knowing how many times a callable was invoked, or what its current configuration is, can help diagnose problems.
Designing a callable class means deciding what its call signature should be, what state it holds, and how that state changes across calls, if at all. A callable that changes state on each call, such as a counter, should document that behaviour clearly, because callers who treat it like a pure function, expecting the same result for the same arguments every time, will be surprised when results change across calls. A callable that does not change state, such as a validator, is safer in that respect, because it behaves like a function, with the added benefit of configuration through its constructor. The choice between stateful and stateless callables should be deliberate, based on whether state is genuinely needed, or whether a plain function with parameters would be simpler and clearer.
Callables also interact with higher-order functions, which accept functions as arguments, such as map, filter, and sorted with a key function. A callable object can be passed anywhere a function can, which makes it a flexible tool for customising generic code, without needing to define a new function for each configuration. For example, a callable that extracts a specific attribute from an object can be passed as a sorting key, and different instances with different attributes can sort by different criteria, all using the same class. That flexibility, through configuration rather than code duplication, is the callable pattern's main strength, and it is worth recognising when a problem calls for it, rather than for a collection of similar functions.
There is also a readability consideration. A callable object looks like a function when it is called, but it is defined as a class, which can confuse readers who expect a class to be instantiated and then used through named methods, not called directly. To avoid confusion, a callable class should have a clear name, a docstring explaining that instances are callable and what calling them does, and a __call__ method whose signature and behaviour are obvious from that documentation. If calling an instance is not the class's main purpose, but only an incidental convenience, it may be better to provide a named method instead, and to reserve __call__ for classes whose whole purpose is to be called.
class Greeter:
def __init__(self, greeting):
self.greeting = greeting
def __call__(self, name):
return f"{self.greeting}, {name}!"
hello = Greeter("Hello")
print(hello("Ada"))Check: What advantage does a callable object have over a plain function when configuration or inspectable state is needed?
Section 26
The __get__ and __set__ descriptor methods were introduced earlier, but their interaction with class-level and instance-level access deserves a closer look, because it explains several behaviours that otherwise seem inconsistent. When a descriptor is accessed through an instance, Python calls its __get__ with the instance and the class. When it is accessed through the class itself, __get__ is called with None as the instance, which is why properties return the property object itself when accessed through the class: the property's __get__ checks for a None instance and returns itself, rather than trying to compute a value for a non-existent instance. That behaviour is deliberate, and it lets tools inspect a property's definition through the class, without needing an instance.
That distinction also matters for custom descriptors. A descriptor whose __get__ does not handle a None instance may fail, or may return a misleading value, when accessed through the class, which is a bug that only appears in contexts where the class, rather than an instance, is used, such as in documentation tools, in class-level inspection, or in code that accesses attributes through the class for metaprogramming purposes. Handling the None case explicitly, by returning the descriptor itself, is a simple habit that prevents that class of bug, and it matches the convention established by properties and methods, which both behave sensibly when accessed through the class.
Instance-level shadowing, discussed earlier for data versus non-data descriptors, also deserves emphasis here, because it is a frequent source of confusion. A data descriptor, which defines __set__, takes precedence over an instance dictionary entry, so assigning to an attribute managed by a data descriptor always goes through the descriptor, and cannot create a shadowing instance entry. A non-data descriptor, which defines only __get__, does not take precedence, so assigning to its attribute on an instance creates an instance entry that shadows the descriptor, and subsequent reads return the instance entry, not the descriptor's value. That difference is not a bug; it is a defined part of the descriptor protocol, but it must be understood when choosing between data and non-data descriptors, because it determines whether assignment is controlled by the descriptor or can bypass it entirely.
For validated attributes, a data descriptor is almost always the right choice, because validation should apply to every assignment, and shadowing would defeat it. For computed attributes that are meant to be overridable per instance, a non-data descriptor may be appropriate, because it provides a default that an instance can override by assignment. The choice should be made deliberately, based on whether assignment should be controlled or overridable, and documented, so that users of the class know which behaviour to expect when they assign to the attribute.
class Loud:
def __get__(self, obj, objtype=None):
if obj is None:
return self
return "hello"
class Page:
title = Loud()
print(Page.title, Page().title)Check: Why should a descriptor's __get__ handle the case where it is accessed through the class, with no instance?
Section 27
Protocols can also be defined explicitly, using the abc module's abstract base classes, or the typing module's Protocol class, both of which let you state, in code, which methods a class must implement to be considered a member of a protocol, rather than leaving that requirement implicit in documentation or in generic code that assumes certain methods exist. Explicit protocols are useful for large programs, where many classes are expected to implement the same interface, and where a missing method should be caught early, at class definition or at type-checking time, rather than late, at runtime, when generic code first tries to call the missing method and fails with a confusing error far from the class definition. Abstract base classes work through inheritance: a class inherits from an abstract base class and must implement its abstract methods, or it cannot be instantiated, which fails fast and clearly at instantiation time. That approach is familiar and works well when the classes in question genuinely share a family relationship, and when inheritance is appropriate for them. Its limitation is that it requires inheritance, which, as discussed in earlier chapters, is not always the right relationship, and which forces a class to declare its protocol membership explicitly, through its base classes, even when the class's behaviour already satisfies the protocol without needing to inherit from anything. Protocol classes, by contrast, work through structural typing: a class is considered to satisfy a protocol if it defines the required methods, regardless of whether it inherits from the protocol class, which matches Python's duck typing philosophy more closely, and which lets existing classes satisfy a protocol without modification, simply because they already have the right methods. For beginners, the key point is not which explicit protocol mechanism to use, but that protocols can be made explicit when a program grows large enough to need them, and that making them explicit is a trade-off: it catches errors earlier and documents interfaces more clearly, but it adds complexity and ceremony that small programs do not need. A small program with two classes that implement the same methods does not need an explicit protocol; documentation and tests are sufficient. A large program with many classes, maintained by many people, may benefit greatly from explicit protocols, because they make interface expectations visible to everyone, and because they let type checkers verify those expectations automatically, across the whole program, without needing to run it. When you do use explicit protocols, keep them small and focused, with only the methods that generic code actually needs, rather than large interfaces that force classes to implement methods they do not need, which is a form of interface bloat that makes classes harder to write and harder to understand, for no benefit to callers who only use a subset of the interface.
from typing import Protocol
class Drawable(Protocol):
def draw(self) -> None:
...
class Circle:
def draw(self) -> None:
print("circle")
def show(item: Drawable) -> None:
item.draw()
show(Circle())Check: What is the difference between an abstract base class protocol, based on inheritance, and a Protocol, based on structure?
Section 28
Dunder methods for arithmetic also include methods for unary operations, such as __neg__ for negation, __pos__ for positive, __abs__ for absolute value, and __invert__ for bitwise inversion, which let a custom numeric or value type support expressions such as -obj, +obj, abs(obj), and ~obj, with familiar syntax and semantics. Those methods are simpler than binary arithmetic methods, because they take only one operand, and therefore do not need reflected counterparts, but they still require care about semantics: negation should return a new object for immutable types, and should not modify the original, while for mutable types, the choice between modifying in place and returning a new object should match the class's overall mutability design, and should be documented, so that callers know whether -obj changes obj or leaves it unchanged, which is a question that matters when other references to the same object exist.
Conversion to a number, through __int__ and __float__, and conversion to a string, through __str__ and __repr__, are also part of a numeric type's protocol surface, and they should be consistent with the type's arithmetic behaviour. A custom number that supports arithmetic but cannot be converted to a built-in number, when conversion makes sense, will frustrate callers who need to pass it to functions expecting a built-in number, while a custom number that converts lossily, without warning, such as a fraction that truncates to an integer on conversion, may silently lose information in a way that callers do not expect. Conversion methods should therefore be defined when conversion is meaningful, should preserve information where possible, and should document any lossiness, so that callers can decide whether conversion is appropriate for their use, or whether they should keep the value in its custom form, with full precision, rather than converting it to a built-in type that cannot represent it exactly.
Rounding, through __round__, and formatting, through __format__, are additional numeric protocols that let a custom number participate in round() and in formatted string literals, which are common operations for numbers in real programs, and which a numeric class that omits them will handle poorly, by falling back to default behaviour that may not match its domain, such as rounding to an integer when the class represents a decimal value with fractional parts that should be preserved, or formatting with a default representation that is less readable than a custom one would be. Implementing those methods is not difficult, once the class's value representation is clear, and it makes the class feel complete in numeric contexts, rather than partially supported, which is the same completeness principle applied to numeric protocols as to other protocols in this chapter.
Testing unary and conversion methods should check that they do not modify the original for immutable types, that they return the correct type, not just the correct value, and that conversion round-trips where possible, such as converting to a float and back, within the precision limits of the target type, which should be documented if the round-trip is not exact, so that callers do not assume exactness where it does not hold.
class Score:
def __init__(self, value):
self.value = value
def __neg__(self):
return Score(-self.value)
def __abs__(self):
return Score(abs(self.value))
print((-Score(3)).value, abs(Score(-3)).value)Check: Why do unary dunder methods such as __neg__ not need reflected counterparts, unlike binary arithmetic methods?
Section 29
The __del__ method, called when an object is about to be destroyed, is a protocol that should be used with extreme caution, and in most cases, not at all. Unlike __enter__ and __exit__, which guarantee cleanup in a with block, __del__ is called at an unpredictable time, when Python's garbage collector decides to reclaim an object, which may be long after the object is no longer needed, or may never happen at all for objects involved in reference cycles, depending on the garbage collector's behaviour. That unpredictability makes __del__ unsuitable for releasing resources that must be freed promptly, such as file handles, locks, or network connections, where a delay in release can cause resource exhaustion, deadlocks, or other problems that a timely, explicit cleanup, through a context manager, would prevent.
Another problem with __del__ is that it runs during garbage collection, when the program's state may be partially torn down, such as during interpreter shutdown, where modules and globals the method relies on may already have been cleared, causing the method itself to fail with confusing errors, or to behave incorrectly, in a context where debugging is difficult, because the failure occurs during cleanup, not during normal execution, and because it may be reported as a warning or an exception during garbage collection, rather than as a clear error at a specific line of user code. Those shutdown-time failures are a well-known hazard of __del__, and they are a strong reason to avoid it for any cleanup that matters, in favour of explicit cleanup through context managers, which run at a predictable time, in a well-defined program state, where errors can be handled normally.
There are rare cases where __del__ is acceptable, such as for logging that an object was collected, or for releasing a resource that is a last resort, in addition to explicit cleanup, where __del__ acts as a safety net, not as the primary cleanup mechanism, and where its failure would be harmless, because the resource would already have been released explicitly in normal use, and only leaked in abnormal use, where __del__ might catch it. Even in those cases, __del__ should be simple, should not rely on globals or on other objects that may already have been collected, and should handle its own exceptions internally, rather than letting them propagate during garbage collection, where they cause warnings that are hard to attribute to a specific cause.
For beginners, the guideline is simple: do not define __del__ unless you have a specific, well-understood reason, and you understand its timing unpredictability and its shutdown hazards. Use context managers for resource cleanup, use explicit close methods for resources that need closing, and let Python's garbage collector handle memory reclamation, which it does well without custom __del__ methods, for ordinary objects that do not hold external resources, which are the vast majority of objects in most programs, and which therefore do not need any destruction protocol at all, beyond what Python already provides automatically, through reference counting and garbage collection, which reclaim memory safely and predictably for such objects, without needing custom code that could introduce the hazards described above.
class Note:
def __init__(self, text):
self.text = text
note = Note("hello")
print(note.text)
del noteCheck: Why is __del__ unsuitable for releasing resources that must be freed promptly, such as file handles?
Section 30
Pickling protocols, through __getstate__, __setstate__, and __reduce__, control how objects are serialised and deserialised by the pickle module, which is used for saving objects to files, sending them between processes, and caching them. By default, pickle saves an object's attribute dictionary and restores it on unpickling, which works for many ordinary classes. It fails, however, for classes that hold unpicklable resources, such as open files, locks, or connections, or for classes whose attributes include data that should not be saved, such as large caches that can be recomputed, or sensitive data that should not be stored in a file at all.
Customising the pickling protocol lets a class control exactly what is saved and what is restored. __getstate__ can exclude unpicklable or sensitive attributes from the saved state, returning only the attributes that are safe and meaningful to store. __setstate__ can then reconstruct the excluded attributes on unpickling, by reopening a file, recreating a connection, or recomputing a cache, as appropriate for the class's domain. That pair of methods turns pickling from an automatic process that may fail or leak data, into a deliberate design decision about what an object's saved form contains, and how a restored object regains a working state.
Security is the most critical consideration for pickling. Unpickling data from an untrusted source can execute arbitrary code, because maliciously crafted pickle data can invoke constructors and methods during unpickling. That is not a theoretical risk; it is a well-known vulnerability inherent in pickle's design, which reconstructs arbitrary Python objects rather than merely parsing data. The guideline is therefore absolute: never unpickle data from an untrusted source, such as data received over a network from an unknown sender, or data read from a file that an attacker could have modified, unless its authenticity has been verified through a trusted mechanism, such as a digital signature from a trusted key.
Even with verification, prefer safer formats where possible. For data that does not need to preserve Python-specific object types, JSON is a safer choice, because parsing JSON cannot execute code, unlike unpickling, which can by design. JSON cannot represent arbitrary objects, but that limitation is also its safety: it can only represent plain data, which is exactly what most data exchange needs. Choosing pickle should therefore be a deliberate decision, justified by a genuine need to preserve object types, and made with full awareness of the trust assumption it carries. That assumption should be documented where pickle is used, so that future maintainers do not inadvertently break it by unpickling data from a new, untrusted source.
Pickle an object, unpickle it, and verify that the restored object equals the original in all relevant respects, that excluded attributes are reconstructed correctly, and that resources are handled as documented. Round-trip testing catches that failure during development, rather than when a user first tries to save their work and discovers that their object cannot be pickled, which is a costly and embarrassing failure that is easily prevented by testing early, with realistic data.
import pickle
class Point:
def __init__(self, x, y):
self.x, self.y = x, y
restored = pickle.loads(pickle.dumps(Point(1, 2)))
print(restored.x, restored.y)Check: Why must data unpickled from an untrusted source be treated as dangerous, and what safer alternative exists for plain data?
Section 31
The __format__ method controls how an object is formatted in formatted string literals and in calls to format(), through format specifications such as width, alignment, and precision. By default, formatting an object uses its __str__ representation, ignoring most of the specification, which is adequate for simple objects but wasteful for value types that have a natural formatted form, such as a money amount that should show two decimal places, or a measurement that should show a chosen unit. Defining __format__ lets such a class honour format specifications meaningfully, so that f"{price:.2f}" produces a properly rounded amount, rather than a default string that ignores the precision request.
Implementing __format__ requires parsing the format specification, or delegating to a built-in type's formatting for the parts you support. The simplest robust approach is to support a small, documented set of specifications, such as an empty specification and a precision specification, and to raise a clear error for specifications you do not support, rather than silently ignoring them. Silently ignoring a specification is misleading, because a caller who requests a particular precision reasonably expects it to be honoured, and a result that ignores it may be wrong in a way that is not obvious, such as a price displayed with too many decimal places in a financial report, where precision matters for readability and for correctness of presentation.
Consistency with __str__ also matters. Formatting with an empty specification should normally produce the same result as str(), so that f"{obj}" and str(obj) agree, which is what users expect from built-in types, where the two are consistent. If your __format__ produces something different for an empty specification, document why, and consider whether the difference is genuinely needed, or whether it will simply confuse callers who use the two forms interchangeably, as they commonly do, without thinking about which one they are using, because for built-ins the two forms are equivalent, and that equivalence is a reasonable assumption for custom classes as well.
Testing __format__ should cover each supported specification, including edge cases such as zero, negative values, and large values, and should verify that unsupported specifications raise a clear error rather than producing a misleading result. Also test __format__ in formatted string literals, not just through direct calls to format(), because the two are equivalent for built-ins but may diverge for a custom implementation that handles one path correctly and the other incorrectly, which is a divergence that testing only one path would miss, and that users would discover in real code, where formatted literals are far more common than explicit format() calls, and where a formatting bug therefore has a wider impact than its test coverage might suggest.
class Price:
def __init__(self, amount):
self.amount = amount
def __format__(self, spec):
return format(self.amount, spec)
print(f"{Price(3.5):.2f}")Check: Why should __format__ with an empty specification normally agree with __str__, and what should it do with specifications it does not support?
Section 32
Computed attributes can be provided through __getattr__, for example a rectangle's area computed from its width and height, or a person's full name computed from first and last names. That approach avoids storing redundant data, which could become stale if the underlying attributes change, and it keeps the derived value always up to date, by computing it on demand from current inputs. It has a significant drawback, however: computed attributes through __getattr__ are invisible in the class definition, so readers cannot tell from reading the class which attributes it provides, beyond those stored in __init__ or defined as properties.
Properties are usually a better tool for computed attributes, because they are declared explicitly in the class body, with a name that appears in the class definition, in documentation tools, and in dir() output, which makes them discoverable. They can also carry docstrings, like methods, which __getattr__-based attributes cannot easily do, since they have no definition site to attach documentation to. A property for an area is visible, documented, and testable as a named attribute, while a __getattr__ fallback that returns an area is invisible until it is used. That invisibility grows into a real problem as a class gains more computed attributes, each of which must be discovered by reading __getattr__'s implementation rather than by reading the class's attribute list.
Performance is similar for both approaches: both compute on every access unless the result is cached. Caching can help for expensive computations, but it introduces staleness, because if the inputs change, a cached result becomes wrong unless the cache is invalidated, which requires additional logic in either approach. For cheap computations, recomputing on every access is usually the right trade-off, accepting a small cost in exchange for always-correct results. For expensive computations, caching with careful invalidation may be justified, and should be documented, so callers know whether a computed attribute reflects current inputs or a snapshot from an earlier access.
The guideline is therefore to prefer properties for computed attributes, and to reserve __getattr__ for genuinely dynamic attributes, such as delegation to a contained object, or lazy loading of attributes whose names cannot be known in advance, where properties cannot be used because the attribute names are not known when the class is defined. That distinction, between names known at definition time and names known only at runtime, should guide the choice in any given case. When names are known, explicit properties are clearer. When they are not, __getattr__ is the right tool, and its invisibility is an acceptable cost, because explicit declaration is impossible rather than merely inconvenient.
class Rectangle:
def __init__(self, width, height):
self.width, self.height = width, height
@property
def area(self):
return self.width * self.height
print(Rectangle(3, 4).area)Check: Why are properties usually a better tool than __getattr__ for computed attributes whose names are known in advance?
Section 33
The __set_name__ method is called when a descriptor is assigned to a class attribute, and it lets the descriptor learn the name it was assigned under. That matters for descriptors that store values in instance dictionaries under a private name derived from the public name, such as storing a validated price under _price when the descriptor is assigned as price. Without __set_name__, a descriptor would need to be told its name explicitly through its constructor, which duplicates the name in two places: in the descriptor's construction and in its assignment. Refactoring that renames an attribute in one place but not the other would then break the descriptor silently, which is a maintenance hazard that __set_name__ eliminates.
With __set_name__, renaming an attribute requires changing only the assignment, and the descriptor adapts automatically, without needing a separate update to its constructor argument. That is a single-source-of-truth benefit: information is derived from context, rather than stated twice, with the risk of inconsistency that duplication always carries. Duplicated information can diverge over time, as maintainers change one copy and forget the other, which is a common source of bugs. Automatic derivation prevents divergence by construction, rather than by discipline alone, which depends on human memory and attention, while construction depends on the language itself, which does not forget to update a derived value when its source changes.
__set_name__ complements the older descriptor methods, __get__, __set__, and __delete__, by handling assignment-time information that those methods need but cannot obtain on their own. At access time, a descriptor knows which instance it is accessed through, but not necessarily which name it was accessed under, unless it learned that name at assignment time through __set_name__, which stores it for later use during access. That two-phase design, where assignment provides context and access uses that context, is a pattern that appears in many parts of Python beyond descriptors, wherever a mechanism learns from context at definition time and uses that learning at runtime.
For users of libraries and frameworks, __set_name__ is also a usability benefit. Libraries that provide validated or computed attributes declaratively, where users define attributes simply by assigning descriptor instances in a class body, would otherwise require users to repeat each attribute's name in the descriptor's constructor, which is tedious and error-prone for every attribute in every class. With __set_name__, users do nothing extra beyond the assignment they were already going to make, and yet gain correctness and maintainability for free, through the language's support for assignment-time hooks. That is the ideal form of automation, and it explains how many declarative libraries manage to provide rich attribute behaviour without burdening their users with repeated names or manual private-storage management.
class Positive:
def __set_name__(self, owner, name):
self.name = "_" + name
def __get__(self, obj, objtype=None):
return getattr(obj, self.name, 0)
def __set__(self, obj, value):
if value <= 0:
raise ValueError("must be positive")
setattr(obj, self.name, value)
class Item:
price = Positive()
item = Item()
item.price = 5
print(item.price)Check: What problem does __set_name__ solve for descriptors that store values under a private name?
Section 34
The __contains__ method defines what in means for a class's instances, which is more than a convenience: it determines how membership testing works in generic code, such as if item in collection:, and in comprehensions and filters that test membership, where the semantics of in must match the semantics of iteration. An item that appears when iterating should also test as present through in, and vice versa. A collection that violates that agreement will confuse users who test membership one way and iterate another way, and get different answers to what should be the same question: whether a given item is in the collection or not.
Another design decision for __contains__ is whether membership means identity or equality. For collections of objects, two distinct objects may compare equal, and membership testing by equality would report both as present when either is present, while membership testing by identity would report only the exact stored object as present. The choice should be deliberate and documented, because it affects how callers use membership testing. A caller checking whether a book with a given title is on a shelf, by testing a newly created book object with that title, would succeed under equality semantics, if books compare equal by title, but would fail under identity semantics, where only the exact stored object counts. Built-in collections normally use equality for in, which is a familiar semantics that custom collections should follow unless they have a clear reason to deviate, with that deviation documented clearly.
Deviating from familiar semantics without documentation is risky, because generic code written for built-ins will assume built-in semantics and behave incorrectly with a custom class that deviates silently. That risk applies to containment as to other protocols: familiarity is a virtue, and deviation should be justified and documented, rather than casual or unnoticed by the class author, who may not realise that a deviation introduced for convenience in one context will affect generic code in other contexts, such as a filter where membership determines whether an item is included or excluded. A surprising membership definition can cause items to be included or excluded incorrectly, based on a domain-specific condition that the generic code did not intend to test for when it used a membership test.
When a domain-specific membership condition is genuinely needed, consider whether an explicit method, such as has_title(), would be clearer than overloading in with that meaning. Explicit methods let callers state exactly what they are testing, while in should be reserved for generic presence, matching built-in semantics closely, so that generic code can rely on it without surprise. Testing __contains__ should verify agreement with iteration, verify the identity-versus-equality choice with both equal-but-distinct objects and identical objects, and verify behaviour for empty collections and for items that are definitely absent, so that both positive and negative membership results are correct.
class Shelf:
def __init__(self, books):
self.books = books
def __contains__(self, book):
return book in self.books
print("A" in Shelf(["A", "B"]))Check: Why should __contains__ agree with iteration about what a collection holds, and should membership normally use equality or identity?
Section 35
The __missing__ method, defined on dictionary subclasses, is called when a key lookup fails, and it provides a hook for customising what happens on a missing key, beyond raising KeyError. A custom __missing__ might return a default value, create a value on demand, or raise a more informative error explaining why the key is missing and what the caller should do about it. That customisation is valuable when missing keys are common and their handling should be centralised in the dictionary class, rather than repeated at every lookup site with get() and a default argument, which duplicates default logic across many places that must be kept in sync manually.
__missing__ is more flexible than defaultdict, whose default logic is a factory function applied uniformly to all missing keys. __missing__, by contrast, can implement key-specific logic that depends on the missing key itself, such as returning different defaults for different keys, or raising different errors for different keys, based on whether a key is expected to be present or not. That flexibility makes it a powerful tool for dictionary subclasses with domain-specific default logic, such as a configuration dictionary that returns a default configuration value for a missing key, or a grouping dictionary that creates a new empty collection for a missing grouping key, with the created value stored in the dictionary so that subsequent lookups find it.
Centralising missing-key handling in the dictionary class, rather than scattering it across lookup sites, is a separation of concerns that makes programs clearer and more maintainable. When handling lives in one method, called automatically by Python's dictionary machinery for every missing key, it cannot be forgotten at a lookup site, because handling missing keys is the dictionary's responsibility, not the lookup site's. That guarantee is especially valuable as a program grows and the number of lookup sites increases, because each new lookup site automatically gets correct missing-key handling, without its author needing to remember to add it, and without the risk that different sites handle the same missing key differently, which would be inconsistent and confusing.
There are limits to be aware of. __missing__ is only called for __getitem__ lookups, not for get(), in tests, or other dictionary methods, which have their own behaviour for missing keys. A class that relies on __missing__ for a default must therefore document that obj[key] returns a default while obj.get(key) returns None unless given a default, which is a distinction that can surprise callers who expect the two forms to agree. Testing should cover both forms, and should verify that values created by __missing__ are stored correctly, so that a second lookup for the same key finds the created value rather than creating a new one, which is the usual and expected behaviour for on-demand creation.
class DefaultDict(dict):
def __missing__(self, key):
value = []
self[key] = value
return value
d = DefaultDict()
d["a"].append(1)
print(d["a"])Check: How does __missing__ centralise missing-key handling for a dictionary subclass, compared with handling it at every lookup site?
Section 36
The __reversed__ method lets a class define what reversed() returns for its instances, which is useful for ordered collections where reverse order is a natural operation, such as a shelf of books listed from last to first, or a history traversed backwards, from the most recent entry to the oldest. Reverse order is a common presentation pattern for histories, logs, and timelines, where users care most about recent items and expect them to appear first. Supporting reversed(obj) directly lets callers write that intent clearly, rather than materialising a list and reversing it, as in reversed(list(obj)), which is less clear and may be less efficient.
Efficiency is a real consideration for large or lazy collections. Materialising a list of all items first, in order to reverse it, requires storing every item, which may be expensive for a large collection, or impossible for a lazily generated sequence that cannot be fully materialised without significant cost. A class that provides __reversed__ directly can return a reverse iterator without materialising a list, by iterating from last to first through indices in reverse order, or through a reverse traversal of an internal structure, such as a linked list traversed backwards from its tail. For collections with only forward traversal support, reverse iteration may still require materialisation internally, in which case the benefit of __reversed__ is clarity rather than performance, because callers can still write reversed(obj) directly, and the class handles materialisation internally, rather than requiring callers to do it explicitly at every reverse iteration site.
That centralisation benefit, placing reversal logic in the class rather than duplicating it at call sites, applies here as it does to other protocols. When reversal logic lives in one method, called automatically for every reverse iteration, it cannot be forgotten or implemented inconsistently at different sites, and a change to the reversal logic needs to be made in only one place. That consistency matters as a program grows and the number of reverse iteration sites increases, because each new site automatically gets correct reverse behaviour, without its author needing to reimplement it.
Designing __reversed__ should start with a deliberate decision: is reverse iteration a natural operation for this class's domain, and can it be implemented efficiently, or at least clearly, without materialising all items? If reverse order is not meaningful for the class, do not define __reversed__, and let reversed() fall back to its default behaviour, which uses the sequence protocol in reverse, or raises an error for non-sequences, which is an honest signal that reverse iteration is not supported. Testing should verify that reversed iteration yields items in exactly the reverse order of forward iteration, including for empty collections and single-item collections, where off-by-one errors in reverse index handling are most likely to appear.
class Shelf:
def __init__(self, books):
self.books = books
def __reversed__(self):
return reversed(self.books)
print(list(reversed(Shelf(["A", "B", "C"]))))Check: When is defining __reversed__ more valuable than letting callers write reversed(list(obj)) themselves?
Section 37
The __index__ method lets an object be used as a list index, a slice bound, or an argument to functions that require an integer index. Its semantics are stricter than those of __int__, and that distinction matters for correctness, because an index must be an exact integer, not merely something that can be converted to an integer with possible loss of information. A float that truncates to an integer on conversion, for example, would be a misleading index if it were accepted silently, because a caller who passes a float as an index has probably made a mistake, and should be told about it through a clear error, rather than having their float truncated to an integer that may point to the wrong item.
That strictness protects callers from accidental truncation and from other silent conversions that could cause indexing errors that are hard to diagnose, because the index actually used is not the index the caller intended. An index selects a specific item from a collection, and selecting the wrong item due to truncation can propagate incorrect results through a program, causing failures far from where the indexing error occurred, where they are difficult to trace back to their source. Failing fast at the point where a non-integer index is used, rather than later when its incorrect result is discovered, is a significant benefit of strictness for indexing protocols in particular, where exactness is essential for correctness.
__index__ achieves that strictness by requiring an exact integer return value from any object used as an index, and by rejecting objects that do not define __index__, even if they define __int__. A custom type that represents a count or a position should therefore define __index__ returning a plain integer, exactly representing the position intended, while a type where indexing makes no sense should not define it at all, so that attempts to use it as an index fail clearly, rather than succeeding misleadingly through conversion. That distinction, between types that are genuinely index-like and types that are merely convertible to integers, is one that __index__ makes explicit, where __int__ alone would blur it.
Testing __index__ should verify that the method returns a plain integer, that objects using it select the correct items in lists and slices, and that objects without it are rejected as indices, even when they define __int__. Also test negative indices and slice bounds, where an off-by-one error in a custom index type would select the wrong item, and where the error would be especially confusing, because the code looks correct and the index type looks reasonable, while the result is nevertheless wrong, which is exactly the kind of silent error that __index__ strictness is designed to prevent, by ensuring that index values are exact, and that index types are deliberate, rather than accidental conversions that happen to succeed.
class Position:
def __init__(self, value):
self.value = value
def __index__(self):
return self.value
items = ["a", "b", "c"]
print(items[Position(1)])Check: Why is __index__ stricter than __int__, and what silent error does that strictness prevent?
Section 38
Deletion protocols, through __delattr__ for attributes and __delitem__ for items, control what happens when an attribute or an item is deleted, through del obj.attr or del obj[key]. They should be used with the same care as other mutation protocols, because deletion is destructive and cannot easily be undone, and because a deletion method that behaves unexpectedly, such as deleting more than requested, or failing to delete at all, can cause confusion and data loss in programs that rely on deletion to clean up state, or to remove items from a collection, in a predictable way.
Built-in deletion semantics are familiar and should be preserved unless there is a clear reason to deviate, with that deviation documented. Deleting an attribute removes it from the instance dictionary, so subsequent access fails with AttributeError, and deleting an item from a mapping removes its key, so subsequent lookup fails with KeyError. A custom deletion method that deviates from those semantics, for example by deleting a related attribute as a side effect, or by marking an item as deleted while leaving its key present, will surprise callers who expect the familiar behaviour, and will break generic code written for built-ins, which assumes that after deletion, the attribute or key is genuinely gone, and that testing for its presence will return false.
That surprise risk is especially serious for deletion, because deletion is often used in cleanup routines, where its effect is not checked immediately, but is assumed by later code, which may then behave incorrectly if deletion did not have its expected effect. A cleanup routine that deletes attributes to free memory, or to reset state for reuse, will leave stale state behind if deletion fails silently, and that stale state may cause later operations to use outdated values, with consequences that are hard to diagnose, because the deletion itself appeared to succeed, while its effect was different from what the caller intended. Testing deletion should therefore verify both the positive effect, that the attribute or item is gone, and the negative effect, that subsequent access fails with the expected error, and that other attributes and items are unaffected, so that deletion is precise, not over-broad.
There are legitimate reasons to customise deletion, such as releasing a resource associated with an attribute, or updating an index when an item is removed from a collection, but those side effects should be documented, and should not change the basic fact of deletion: after del obj.attr, the attribute should be gone, and after del obj[key], the key should be gone. Side effects that maintain consistency, such as updating a related index, are appropriate, because they preserve the class's invariants while performing the requested deletion, while side effects that change what deletion means, such as refusing to delete under some conditions without raising an error, are not, because they break the caller's reasonable expectation that a successful del statement actually deleted something.
class Config:
def __init__(self):
self.data = {"theme": "dark"}
def __delitem__(self, key):
del self.data[key]
c = Config()
del c["theme"]
print("theme" in c.data)Check: Why should deletion methods such as __delitem__ preserve built-in deletion semantics unless a deviation is documented?
Section 39
Equality across a class hierarchy can violate symmetry, and symmetry is the simple rule that if a equals b then b must equal a. That rule is easy to break when a parent and a subclass define equality differently. The breakage usually starts when the parent compares only parent attributes, while the subclass also compares its own extra attributes. A parent instance and a subclass instance may then agree in one direction and disagree in the other, which is exactly the asymmetry that sets and dictionaries cannot tolerate.
Consider a parent class for animals that compares only a name, and a subclass for dogs that also compares a breed. A plain animal called Rex and a dog called Rex with a breed of collie might compare equal when the parent method runs, because both have the name Rex and the parent ignores breed. The same pair might compare unequal when the subclass method runs, because the subclass checks breed and the plain animal has no breed to match. The result depends on which operand method Python calls first, so equality appears to work in simple tests that check only one direction and fails in real code that checks both directions.
The consequences are worse than a confusing test result. Sets and dictionaries assume symmetry when they store and look up objects, and they may call equality on either operand depending on hash values and internal ordering. An asymmetric pair can therefore be found in a set when searched from one direction and missed when searched from another, and that behaviour can vary between runs, which makes the bug look intermittent and hard to diagnose.
Checking the exact type is a straightforward way to prevent this problem. Instead of accepting any instance of the parent class, the equality method requires that both objects have exactly the same type, using a test such as type(self) is type(other). Under that rule a parent instance and a subclass instance are never equal, because their types differ, while two instances of the same class compare by their attributes as normal. Symmetry is restored because the answer no longer depends on which method runs first, since both directions now agree that cross-class pairs are unequal.
The example below shows this strict approach in a compact form. The parent method checks the exact type before it compares names, so an animal and a dog with the same name are not equal, while two dogs with the same name are equal.
class Animal:
def __init__(self, name):
self.name = name
def __eq__(self, other):
return type(self) is type(other) and self.name == other.name
class Dog(Animal):
pass
print(Animal("Rex") == Dog("Rex"))
print(Dog("Rex") == Dog("Rex"))This strictness has a trade-off that should be understood before it is adopted. A subclass instance will never equal a parent instance, even when every shared attribute matches, which may surprise readers who expect a dog to equal an animal with the same name. That trade-off is usually acceptable, because a predictable rule that is slightly strict is safer than a flexible rule that breaks symmetry. For most hierarchies the exact type check is the clearer choice, and it should be paired with tests that check both directions for every equality case involving objects of different classes in the hierarchy.
Check: How can equality across a class hierarchy violate symmetry, and how does checking the exact type help prevent it?
Section 40
The second half of this chapter has covered numeric protocols, total ordering, iteration reusability, context manager error handling, callable state, descriptor access details, explicit protocols, unary operations, destruction, pickling, formatting, computed attributes, __set_name__, containment, missing keys, reversed iteration, indexing strictness, deletion, and equality in hierarchies. Each topic built on the first half's foundations, and together they form a comprehensive toolkit for protocol design in Python, going beyond the basic dunder methods introduced in earlier chapters, and preparing you to design classes that integrate deeply with Python's built-in operations in many different contexts, from printing and comparison, to iteration and resource management, to serialisation and formatting.
The principles that recur across all these protocols are worth naming explicitly, because they transfer to protocols you have not yet met. The first is completeness or explicit limits: implement a protocol fully, or document clearly which parts are supported and which are not, because silent partial implementation misleads users who reasonably assume, from one successful operation, that the rest of the protocol works too. The second is consistency: related methods must agree, so that equal objects have equal hashes, iteration order matches indexing order, membership agrees with iteration, and comparison methods agree with each other and with equality. Inconsistency between related methods is a bug, even when each method looks correct in isolation, and it is often discovered late, in code far from the class definition.
The third principle is familiarity: protocols exist so custom classes can be used like built-ins, with familiar syntax and semantics, and deviating from built-in semantics should be a deliberate, documented decision, not a casual choice, because generic code written for built-ins will misinterpret the deviation. When a domain meaning is needed, an explicit method is usually clearer than overloading an implicit protocol method. The fourth principle is testing in context: protocol methods should be tested not only in isolation, but in the generic contexts where they will actually be used, such as sets and dictionaries for hashing, sorting and deduplication for comparison, with statements with exceptions for context managers, and filters for truthiness. A method that passes an isolated test but fails in its real context has not really been tested.
Finally, remember that protocols are about duck typing: an object is usable in a context because it implements the right methods, not because it inherits from a particular class. That flexibility is also a responsibility, because your class's methods are its promise to every piece of generic code that will use it. Design that promise carefully, document it honestly, and test it thoroughly, and your classes will feel like natural parts of Python rather than foreign objects requiring special handling. Keep that checklist beside you when you design your next class, and return to it whenever a class starts to feel awkward to use, because awkwardness is often a sign that a protocol has been implemented partially, inconsistently, or in a way that deviates from familiar semantics without a clear reason.
class Book:
def __init__(self, title):
self.title = title
def __repr__(self):
return f"Book({self.title!r})"
print(Book("Dune"))Check: How does reviewing the second half of this chapter's protocols together help you design a class that works coherently across many contexts?
Section 41
The __slots__ mechanism is not a dunder protocol in the same sense as the methods covered so far, but it interacts with attribute protocols in ways that matter for protocol design, because it changes how instance attributes are stored. Normally, each instance has a dictionary for its attributes, which allows arbitrary attributes to be added at any time, at the cost of memory for that dictionary. Defining __slots__ as a tuple of attribute names tells Python to store those attributes in fixed slots, without a per-instance dictionary, which saves memory for classes with many instances, but also prevents adding attributes that are not listed in __slots__.
That restriction interacts with __getattr__ and with descriptors in specific ways. A class with __slots__ and no __dict__ slot cannot have arbitrary instance attributes, so a descriptor that stores values in the instance dictionary under a private name will fail, unless the class also includes that private name in its slots, or includes __dict__ itself, which defeats much of the memory saving. That interaction must be planned when combining slots with descriptors: the private storage names used by descriptors must be listed in __slots__, or the descriptor design must be changed to store values elsewhere, such as in a separate mapping held by the descriptor, which is a different design with its own trade-offs, and which should be chosen deliberately, rather than discovered by accident when a slotted class with descriptors fails at runtime.
Slots also interact with inheritance, because a subclass does not automatically gain the memory benefit of its parent's slots in the way methods are inherited. Each class in a hierarchy defines its own slots, and a subclass without its own __slots__ will gain a dictionary, negating the parent's memory saving for subclass instances, which is a subtlety that can surprise developers who expect slots to be inherited like methods. That subtlety, combined with the restriction on arbitrary attributes, means that slots should be used only when memory saving is genuinely needed, such as for classes with very many instances, and when the class's attribute set is fixed and known in advance.
For most classes, the memory saving from slots is not worth the restriction and the interaction complexity, and default dictionary storage is preferable, because it is flexible, familiar, and compatible with all attribute protocols, without special planning. Slots are an optimisation, not a default, and like all optimisations, they should be applied only when measurement shows they are needed, and when their costs, in flexibility and in protocol interaction complexity, have been weighed against their benefits, for the specific class in question, in the specific program where it will be used, with many instances, and with a fixed attribute set that justifies the restriction, and that will not change frequently during the program's evolution, as new requirements emerge and new attributes are needed.
class Point:
__slots__ = ("x", "y")
def __init__(self, x, y):
self.x, self.y = x, y
print(Point(1, 2).x)Check: How does __slots__ interact with descriptors that store values in an instance dictionary, and when is it worth using?
Section 42
The __ne__ method, for not-equal, is one dunder method you should normally not define. Python derives __ne__ from __eq__ by negation, unless a class defines it explicitly, which is rarely necessary, and which should be avoided unless the class has a specific reason to define not-equal differently from the negation of equal. For a well-designed equality, not-equal is by definition the negation of equal, and defining it separately creates a risk of inconsistency between the two methods, introduced by accident when one method is updated and the other is forgotten, which is a maintenance hazard that deriving __ne__ from __eq__ avoids by construction, rather than by discipline alone.
Discipline depends on human memory and attention, while construction depends on the language itself, which does not forget to update a derived value when its source changes. That is why Python's default derivation of __ne__ from __eq__ is a valuable feature that should be relied on, rather than overridden without a compelling reason, documented clearly, so callers are not surprised by a not-equal definition that differs from the negation of equal. Such a difference would violate a fundamental property of equality that users expect to hold for all objects, regardless of their classes, and breaking it would cause confusing behaviour in sets, dictionaries, and comparisons, where equality and not-equality are assumed to be complementary.
Consider the logical consequence of a violation: an object could be both equal and not-equal to another object at the same time, which is a contradiction that should not be possible for any well-designed class. That contradiction could cause an object to be found in a set when searched one way but not another, or to be treated as both present and absent, depending on which method a particular piece of code happens to call. Those failures would be intermittent and confusing, because they depend on code paths that may use == in one place and != in another, with different results for the same pair of objects, which is exactly the kind of inconsistency that is hardest to debug, because each individual test may pass, while the combination fails.
The broader principle applies to many derived methods in Python, beyond __ne__ alone: where a method is logically determined by another method, deriving it automatically is preferable to defining it independently, when the derivation is logically necessary, and when independent definition could introduce inconsistency between related methods that should agree by definition, rather than by coincidence or by careful maintenance alone. That principle is also why tools such as functools.total_ordering exist, to derive missing comparison methods from a minimal set, rather than requiring each one to be written by hand, with the risk of divergence that hand-writing entails, for any pair of methods whose relationship is logically fixed, and therefore should be enforced by the language or by a decorator, rather than by hand-written code that could drift apart over time.
class Point:
def __init__(self, x, y):
self.x, self.y = x, y
def __eq__(self, other):
return isinstance(other, Point) and (self.x, self.y) == (other.x, other.y)
print(Point(1, 2) != Point(1, 3))Check: Why should __ne__ normally be derived from __eq__ by Python, rather than defined explicitly?
Section 43
The __call__ protocol can also create function-like objects that maintain state across calls, such as a memoising function that caches results, or a rate limiter that tracks how often it has been called. Those are use cases where a plain function with a global cache or counter would work, but where a callable object provides better encapsulation, by holding the cache or counter in its instance attributes, rather than in a global variable that could be modified by other code, or that would be shared across all uses of the function, even when different uses should have separate caches or counters.
That distinction matters for programs where the same function logic is used in different contexts with different state requirements, such as a memoising function used for two different computations that should have separate caches, to avoid cache pollution between them, or a rate limiter used for two different resources that should have separate limits and separate counters for their respective usage patterns. A plain function with global state cannot distinguish between different uses of the same logic in different contexts, which is a limitation that callable objects overcome by providing per-instance state, isolated between instances, and configurable independently for each instance through its constructor arguments, which is a flexibility benefit worth recognising when choosing between a plain function and a callable object for a given task.
State isolation is also a testing benefit. Per-instance state allows each test to create a fresh instance with fresh state for its own use, without affecting other tests or being affected by them. Global state, by contrast, requires special handling to reset between tests, which is easy to forget, and which can cause test interference, where one test's state affects another test's results, producing failures that depend on test order, which are notoriously difficult to diagnose. That testing advantage alone is often sufficient reason to prefer a callable object for stateful logic, even when global state would work in production code, because test reliability matters as much as production correctness, and because test interference wastes time that could be spent fixing real problems.
As with other callable designs, a stateful callable should document its state clearly: what state it holds, how that state changes across calls, and whether calling it is safe to repeat with the same arguments, or whether results change across calls due to caching or counting. A memoising callable, for example, should document that its first call for given arguments computes a result, while later calls return the cached result, which is faster but means that changes in underlying data, after the first call, will not be reflected in later calls, unless the cache is cleared, which is a behaviour callers need to know about, in order to use the callable correctly, and in order to decide when to clear its cache, or when to create a fresh instance, for a new computation, with fresh state, rather than reusing a cached instance, whose state reflects an earlier computation, that may no longer be relevant, to the current task, at hand.
class Counter:
def __init__(self):
self.count = 0
def __call__(self):
self.count += 1
return self.count
counter = Counter()
print(counter(), counter())Check: How does a callable object's per-instance state improve on a plain function with global state for stateful logic?
Section 44
Context managers can be nested, and they can be combined in a single with statement, with multiple managers separated by commas, which is a convenient way to manage several resources at once, such as opening two files simultaneously for copying data from one to the other, or acquiring two locks simultaneously for a critical section that needs both locks held throughout its execution. Those are use cases where nested or combined context managers provide a clean way to ensure that all resources are released correctly, even if an exception occurs during the block's execution, or during the setup of a later manager, after an earlier manager has already been entered.
That partial-setup case is where manual setup and cleanup would be most error-prone, because an exception during setup of a later resource could leave an earlier resource unreleased, if cleanup is not handled carefully for partially completed setup. The with statement handles that complexity automatically, by exiting managers that have already been entered, in reverse order, when an exception occurs during entry of a later manager, or during execution of the block itself. That guarantee makes nested and combined context managers reliable for managing multiple resources simultaneously, in a way that manual code would struggle to replicate correctly in all cases, including partial setup failures and exceptions during execution, which are precisely the cases where resource management is most error-prone, and where automatic handling through with provides the greatest benefit.
Centralising cleanup logic in the managers themselves, rather than duplicating it at every use site with try and finally blocks, is another benefit that grows with the number of resources and combinations in a program. Each try/finally block must be written correctly for each resource, and for each combination of resources used simultaneously, which is a burden that grows quickly, and that context managers reduce dramatically, by providing a uniform way to manage any resource through the same with syntax, regardless of how many resources are involved, or how they are combined in a given block. That uniformity should not be underestimated for programs that manage many resources of many kinds simultaneously, in complex combinations, that would be difficult to manage manually without errors or omissions in cleanup logic.
Those omissions can lead to resource leaks, deadlocks, or other problems that manifest as resource exhaustion or contention, far from where the cleanup omission occurred, which makes them hard to diagnose, because the symptom appears in a different part of the program from its cause. Context managers prevent that diagnostic difficulty by guaranteeing cleanup automatically, for all resources managed through them, regardless of how they are combined or nested, which is a reliability benefit that applies to every program that manages resources, and that makes with statements, with multiple managers, a pattern worth using routinely, whenever more than one resource must be held simultaneously, for any block of code, in any program, whatsoever.
class Resource:
def __init__(self, name):
self.name = name
def __enter__(self):
print(f"open {self.name}")
return self
def __exit__(self, exc_type, exc, tb):
print(f"close {self.name}")
return False
with Resource("a"), Resource("b"):
print("work")Check: How does the with statement handle cleanup when an exception occurs during entry of a later manager, after earlier managers have already been entered?
Section 45
Python defines fallback orders for several protocols, beyond the boolean fallback from __bool__ to __len__ discussed earlier. Iteration falls back to __getitem__ with increasing indices when __iter__ is missing, containment falls back to iteration when __contains__ is missing, and string conversion falls back to __repr__ when __str__ is missing. Those fallbacks make Python flexible, by allowing classes to implement a minimal set of methods and still work in many contexts, through automatic derivation of other operations from those methods, which is a convenience that should be understood and used deliberately, rather than relied on accidentally, because fallback behaviour may not be exactly what a class intends for a given operation.
Iteration through __getitem__ is a good example of a fallback that can mislead. It requires indices starting at zero and increasing contiguously, and it stops at the first IndexError, which it interprets as exhaustion. For a class whose indexing does not meet those requirements, such as a class with non-zero starting indices, or with sparse indices that skip values, fallback iteration would produce incorrect results, by stopping early at the first gap, rather than continuing to later indices that do contain items. That is a bug that explicit __iter__ prevents, by defining iteration directly, rather than through fallback to indexing, which is a reason to define __iter__ explicitly for any class that is meant to be iterable, rather than relying on fallback behaviour that may not match the class's indexing scheme or its intended iteration order.
Explicit definitions also reduce testing burden. A class that relies on fallback iteration must have its fallback behaviour tested explicitly, for each edge case, including empty collections, single-item collections, and collections with non-standard indexing, that would confuse fallback iteration in ways that are hard to predict without testing. A class with explicit __iter__ makes iteration behaviour direct and testable, without needing to test fallback interactions between indexing and iteration, which are implicit and therefore easy to overlook during development and testing. That directness is a clarity benefit as well as a correctness benefit, because readers can see exactly how iteration works, by reading __iter__, rather than inferring it from the combination of __getitem__ and Python's fallback rules.
The general guideline is to treat fallbacks as conveniences for simple cases, and to define methods explicitly whenever a class's behaviour for an operation is important, non-trivial, or different from what the fallback would provide. Fallbacks are a safety net, not a design strategy, and relying on them for core behaviour, such as iteration for a collection class, is a form of implicit design that should be avoided, in favour of explicit methods that state the class's intentions clearly, for readers, for tests, and for generic code that will use the class in contexts its author may not have anticipated, where explicit behaviour is more reliable than fallback behaviour, in every case, without exception.
class Shelf:
def __init__(self, books):
self.books = books
def __iter__(self):
return iter(self.books)
print(list(Shelf(["A", "B"])))Check: Why should a class meant to be iterable define __iter__ explicitly, rather than relying on fallback iteration through __getitem__?
Section 46
A class that defines __eq__ without __hash__ becomes unhashable by default in Python 3, which is a deliberate design decision to prevent inconsistency between equality and hashing. Without that rule, a class with custom equality would retain the default hash, based on identity, which would cause equal objects to have different hashes, violating the fundamental requirement that equal objects must have equal hashes, for sets and dictionaries to work correctly. That requirement is essential, because sets and dictionaries use hashes to locate objects, and an object stored under one hash cannot be found by searching under a different hash, even if the two objects compare equal, which would make set membership and dictionary lookup unreliable for that class. Unhashability as a default is a safe failure mode, because it fails loudly at insertion time, with a clear error stating that the object is unhashable, rather than failing silently at lookup time, with a missing object that should be present but is not found, due to inconsistent hashing. A loud failure at insertion tells the caller immediately that they cannot use the object in a set, and why, which is information they can act on, by either making the object immutable, or by defining a consistent __hash__ based on the same attributes as equality, or by using a different structure, such as a list, that does not require hashing. A silent lookup failure, by contrast, appears later in a program, far from where the object was inserted, and is far harder to diagnose, because the object itself looks correct, only its lookup fails, for a reason that is not visible at the lookup site. That safe failure also protects against a class-level bug that call sites cannot work around. Hashing inconsistency is a property of the class, not of any particular use of it, and no amount of careful coding at call sites can compensate for it, short of avoiding sets and dictionaries entirely for that class, which is a severe limitation for a class meant to be used in generic code, that assumes set and dictionary support for all objects that define equality consistently with their hashing. By making such a class unhashable by default, Python forces its author to confront the hashing question explicitly, either by defining a consistent hash, or by accepting unhashability as a deliberate design decision, documented for users, rather than by silently retaining an inconsistent default hash, that would cause confusing failures in generic code, written by others, who reasonably assume that equality and hashing agree, for any class that defines equality at all.
class Point:
def __init__(self, x, y):
self.x, self.y = x, y
def __eq__(self, other):
return isinstance(other, Point) and (self.x, self.y) == (other.x, other.y)
__hash__ = None
print(Point(1, 2) == Point(1, 2))Check: Why does defining __eq__ without __hash__ make a class unhashable by default in Python 3, and why is that safe?
Section 47
The __repr__ method deserves a final emphasis, because it is the most widely used dunder method in practice, through debugging, logging, and interactive exploration, where objects are frequently displayed, and where a good representation can save significant debugging time, by showing an object's state clearly, at a glance, without needing to inspect its attributes manually, through separate attribute access operations, that are tedious and error-prone when performed repeatedly, during debugging of a complex program, with many objects of many classes, whose states must be understood quickly, to diagnose a problem in a specific part of that program.
A good __repr__ shows the object's class name and its key attributes, in a compact, readable form, that can be understood without needing to know the class's internal implementation details in advance, which is a usability benefit for debugging that should not be underestimated, for any class that will be used in real programs, where debugging is a frequent activity, and where time spent understanding object states is time that could be spent fixing problems instead, if representations were clearer and more informative from the start. That is a reason to define __repr__ for every class you write, regardless of its size or its domain, because the cost of defining it is small, a few lines of code that format the object's key attributes into a readable string, while the benefit is large, in debugging time saved over the life of a program, where that class is used frequently, and where its instances appear in logs, tracebacks, and interactive sessions, that all rely on __repr__ to display objects clearly, to developers who need to understand them quickly, under time pressure, during debugging of a failing program, where clarity matters most.
A poor representation, such as the default memory-address form, provides no useful information about an object's state at all, and therefore wastes debugging time that a good representation would have saved immediately, by showing the state that matters for diagnosing the problem at hand. That practical argument applies to every class without exception, regardless of whether it implements other protocols or not, because representation is a universal need for all objects, in all programs, that are debugged, logged, or explored interactively, in any context whatsoever, and because the default representation is almost never sufficient for understanding an object's state in a meaningful way, for debugging purposes, as a whole, which is why defining __repr__ should be a habit, applied routinely, to every class, from the moment it is first written, rather than added later, when debugging reveals that its absence is costing time, that could have been saved, by a few lines of code, written at the start, when the class's key attributes were already clear, to its author, who knew exactly which attributes mattered, for understanding its instances, at a glance, in any debugging context, whatsoever.
class Book:
def __init__(self, title, author):
self.title, self.author = title, author
def __repr__(self):
return f"Book({self.title!r}, {self.author!r})"
print(Book("Dune", "Frank Herbert"))Check: Why is defining a clear __repr__ worthwhile for every class, even one that implements no other protocol?
Section 48
Protocol design benefits from a checklist that you can apply to any class before considering it complete. That checklist should include questions such as: which protocols does this class support, and which methods does each protocol require? Are all required methods implemented? Do related methods agree with each other, such as equality with hashing, and iteration with membership? Are deviations from built-in semantics documented? Are protocol methods tested in the generic contexts where they will actually be used, rather than only in isolation? And are protocol limits, such as single-use iteration or unhashability, documented clearly for users, in the class docstring and in the method docstrings for each protocol method itself, so that the limits are visible to anyone who reads the class's documentation before using it in a protocol context? That documentation responsibility applies to all protocol classes without exception, regardless of their size or their domain, because protocol support is a public interface promise, that users rely on when writing generic code, that assumes a protocol based on one successful operation, and then uses other operations from the same protocol, without checking whether they are implemented. That assumption is reasonable for built-ins, where protocols are complete, but it can fail for custom classes with partial protocol support, which is why complete, or explicitly partial, protocol implementation is important, for custom classes, and for protocol classes in general, as discussed throughout this chapter, for each protocol in turn, where completeness and explicitness are the two acceptable options, and where implicit partial implementation, without documentation, is the unacceptable option, that should be avoided in all cases, regardless of the protocol involved. Implicit partial implementation misleads users about what a class can do in a given context, and causes bugs that are hard to trace to their source in partial protocol support, rather than in user code, which appears correct, given the assumption that the protocol is complete. That assumption should not be violated silently by a custom class that appears to support a protocol, but that in fact supports only part of it, without saying so clearly, in its documentation, or in its method definitions, where missing methods are visible as absences, but where their absence may not be noticed by a user who only tests the operations they happen to use in their own code, and who therefore does not discover the missing operations until another user, or another part of a program, tries to use them, and fails with a confusing error, that does not clearly indicate that the failure is due to partial protocol support, rather than to a bug in the calling code itself.
class Book:
def __init__(self, title):
self.title = title
def __repr__(self):
return f"Book({self.title!r})"
print(Book("Dune"))Check: What questions should a protocol design checklist ask about a class before considering it complete?
Section 49
Reviewing protocols as a whole also reveals how they relate to other chapters in this book, such as classes and objects, inheritance and composition, iterators and generators, decorators and context managers, and the standard library, where protocols appear repeatedly, as the mechanism by which built-in operations work generically, with many kinds of objects, through a common interface, defined by methods, not by inheritance. That is a unifying principle that explains why Python feels consistent across its built-in types and custom types, and why learning protocols, rather than just learning individual dunder methods, gives you a deeper understanding of the language as a whole, rather than a list of special cases, each to be memorised separately, without seeing how they relate to each other, and to the language's design as a whole. That perspective will be developed further in later chapters, where protocols work together in a single class, to provide a coherent, complete behaviour for that class, in many different contexts, such as being printed, compared, iterated, and used as a context manager, all through different protocols, each implemented by a few dunder methods, that together make the class feel like a natural part of Python, rather than a foreign object that requires special handling in each context. That is the ultimate goal of protocol implementation: to make custom classes indistinguishable, in their usability, from built-in types, in the contexts where they are meant to be used, which is a high standard, but one that is achievable for well-designed classes, that implement protocols completely, consistently, and with clear documentation of which protocols they support, and how they support them. That documentation is information that callers need, to use a class effectively in generic contexts, where its protocol support is assumed, rather than checked explicitly for each operation, which is a common pattern in Python, where generic code assumes a protocol based on one operation succeeding, and then uses other operations from the same protocol, without checking whether they are implemented. That assumption is reasonable for built-ins, where protocols are complete, but it can fail for custom classes with partial protocol support, which is why complete, or explicitly partial, protocol implementation is important, for custom classes, and for protocol classes in general, as discussed throughout this chapter, for each protocol in turn, where completeness and explicitness are the two acceptable options, and where implicit partial implementation, without documentation, is the unacceptable option, that should be avoided in all cases, regardless of the protocol involved.
class Book:
def __init__(self, title):
self.title = title
def __repr__(self):
return f"Book({self.title!r})"
print(Book("Dune"))That cross-chapter perspective is the point of learning protocols as a system, rather than as isolated tricks: once you see that iteration, context management, comparison, and representation all work through the same mechanism, you can predict how a new built-in operation will interact with your classes, even before you have met that operation in detail.
Check: How do protocols connect this chapter to classes, inheritance, iterators, and context managers covered elsewhere in the book?
Section 50
This final section brings the chapter together, by returning to its central idea: that dunder methods are not isolated tricks, but protocols, that let custom classes participate in Python's built-in operations, as naturally as built-in types do, in the contexts where they are meant to be used. That perspective transforms how you design classes, from thinking about individual methods in isolation, to thinking about the promises a class makes to generic code, that will use it in many contexts, without knowing its specific class, or its internal implementation details, in advance, which is the essence of polymorphism through protocols, rather than through inheritance alone. That perspective should be carried forward into every class you design from now on, by asking four questions. First, which protocols should this class support? Second, how should those protocols interact with each other consistently? Third, how should deviations from built-in semantics be documented? And fourth, how should protocol methods be tested in their real generic contexts, rather than only in isolation? Those four questions together form a practical design process for protocol-based classes, that is simple enough to apply routinely, and thorough enough to catch most protocol design mistakes before they reach users, who depend on your classes for their own programs to work correctly, in generic contexts that you may not have anticipated when you designed them in the first place. That last point is a reality of class design that makes protocol thinking essential, rather than optional, for any class that will be used beyond its author's own code, in programs written by others, who will assume protocol completeness and consistency, based on familiar built-in behaviour, and who will be surprised and frustrated when a custom class violates those assumptions silently, without documentation or warning of any kind. That user experience failure is one that protocol design, done well, prevents, by making promises explicit, complete, and consistent from the start, so that users can rely on them confidently, when writing code that uses your classes in ways you may not have foreseen, but that your protocol design nevertheless supports correctly, because it was based on principles rather than on ad hoc decisions for individual operations, in isolation from each other, and from the class as a whole.
class Book:
def __init__(self, title):
self.title = title
def __repr__(self):
return f"Book({self.title!r})"
print(Book("Dune"))Start with the smallest protocol set that makes the class genuinely useful, implement it completely, and add further protocols only when a real use case demands them, documenting each addition as you go, so that the class grows by deliberate design rather than by accidental accumulation of half-implemented methods. Keep that four-question checklist beside you whenever you design a new class, and revisit it during review.
Check: What four questions should you ask when designing a class's protocol support, and why do principles scale better than ad hoc decisions?
Practical Task
Build a Shelf class for a small personal library that implements several protocols coherently. It should store books as strings, support len(), indexing and slicing, iteration, membership testing with in, a clear __repr__, and equality by contents, and it should be unhashable because it is mutable (define __eq__ without __hash__, or set __hash__ to None explicitly). Add a __contains__ that agrees with iteration, and a __reversed__ that yields books in reverse order. Write tests, using plain assert statements, that check: a new shelf's length and representation; indexing, slicing, and iteration agree on order; membership agrees with iteration for a present and an absent book; reversed iteration is exactly the reverse of forward iteration; two shelves with the same books compare equal in both directions, and shelves with different books do not; and attempting to put a shelf in a set raises a TypeError because it is unhashable. Also test that iterating twice gives the same result both times, proving the shelf is a reusable iterable rather than a single-use iterator.
Chapter Questions
- Why must a class that defines
__eq__also consider__hash__, and what goes wrong if equal objects have different hashes? - Explain the difference between an iterable and an iterator, and why iterating an iterator twice can silently give an empty result the second time.
- When should a context manager's
__exit__suppress an exception, and why is suppressing all exceptions almost always a mistake? - Why is implementing a mapping protocol through composition usually safer than inheriting from
dictwhen the mapping must validate its values? - A class implements
__getitem__for indexing but not for slices. Why is that partial protocol support misleading, and what are the two acceptable ways to handle it?