Chapter 16 — Iterators, Generators and Comprehensions
Section 1
Iteration is one of the most common operations in Python, and it is worth understanding precisely what happens when you write a for loop. When Python executes for item in collection:, it does not simply walk through the collection by index. Instead, it asks the collection for an iterator, by calling iter() on it, and then repeatedly calls next() on that iterator, collecting each item it returns, until the iterator signals that it is exhausted, by raising StopIteration. The loop catches that exception internally, ends cleanly, and moves on to the code after the loop, without any error reaching your program. That mechanism, hidden behind simple syntax, is what makes for loops work uniformly over lists, strings, dictionaries, files, and custom objects, regardless of how each one stores its items internally. That uniformity is powerful, because it means that any object that can supply an iterator can be used in a for loop, in a comprehension, in unpacking, and in built-in functions such as list(), sum(), and sorted(), all of which rely on the same iterator mechanism, rather than on each type's internal storage details. A list stores items in an array, a dictionary stores key-value pairs in a hash table, a file yields lines from disk, and a generator computes items on demand, yet all of them can be looped over with identical syntax, because all of them implement the same protocol: __iter__ to produce an iterator, and __next__ on the iterator to produce items, one at a time, until exhaustion. That protocol is the bridge between diverse storage and computation strategies, on one side, and uniform looping syntax, on the other. Understanding the mechanism also helps you debug loops that behave unexpectedly. If a loop runs zero times, the iterator was exhausted immediately, or the collection was empty, which are different causes with different fixes, and distinguishing them requires checking the collection's contents, rather than assuming the loop itself is broken. Another consequence of the mechanism is that iteration is lazy in a specific sense: items are produced one at a time, on demand, rather than all at once, up front. For a list, that laziness is invisible, because all items already exist in memory, and producing them one at a time is cheap. For a file, or a generator, laziness matters greatly, because it means that a loop over a huge file does not need to load the whole file into memory, and a loop over a generator does not need to compute all its items in advance, which would be wasteful, or impossible, for an infinite sequence, where there is no complete set of items to compute in advance, at all. That on-demand production is what makes iteration scale to large and infinite data, without requiring unbounded memory, or unbounded computation, before the first item can be used. Finally, iteration's uniformity means that learning to write a custom iterable, or a generator, immediately gives you a tool that works with the whole ecosystem of Python's looping constructs, without needing to learn a separate interface for each one.
for letter in "abc":
print(letter)Check: What two steps does a for loop perform, using iter() and next(), and how does it know when to stop?
Section 2
An iterable is an object that can produce an iterator, through its __iter__ method, which should return a fresh iterator each time it is called. That freshness is what makes an iterable reusable: each iteration starts from the beginning and yields the same items, in the same order, assuming the iterable's contents have not changed between iterations. Lists, tuples, strings, and dictionaries are all reusable iterables, because their iterators are independent objects that track their own position, separately from the collection itself. Multiple iterators over the same collection can therefore coexist, each at its own position, without interfering with each other, which matters for nested loops, where an outer loop and an inner loop both iterate over the same collection simultaneously, each needing its own iterator. An iterator, by contrast, is an object that produces items itself, through its __next__ method, and that raises StopIteration when it is exhausted. Its __iter__ method returns itself, rather than a fresh iterator, which means that iterating an iterator a second time continues from where the first iteration left off, or yields nothing, if the iterator is already exhausted. That is the crucial difference: a reusable iterable restarts on each iteration, while a single-use iterator does not restart at all. Code that iterates an iterator twice, expecting the second pass to repeat the first, will silently get no items on the second pass, which produces empty results rather than an error, and is therefore especially dangerous, because it may not be noticed until much later, when a result that should contain items is found to be empty. That silent failure appears in common patterns, such as a program that first counts or validates a collection, and then processes it in a second pass, expecting the same items to be available again. For a reusable iterable, that pattern works correctly, because the second pass creates a fresh iterator. For a single-use iterator, the second pass yields nothing, because the first pass already consumed it. The fix is to recognise which kind of object you hold, and to materialise a single-use iterator into a list, if you need to iterate it more than once, or to create a fresh iterator for each pass, by calling the generator function again, rather than reusing an exhausted generator object. That decision, between materialising and recreating, should be made deliberately, based on whether the data fits comfortably in memory, and whether recreating it is cheap, or expensive, for the specific case at hand. Documentation also matters here. A function that consumes an iterator argument should say so, because callers who do not know that their iterator will be consumed may try to reuse it afterwards, and fail silently. Similarly, a function that returns an iterator, rather than a list, should say so, because callers who expect a reusable result may iterate it twice, and be surprised by the second, empty result.
numbers = [1, 2, 3]
it = iter(numbers)
print(next(it), next(it))
print(list(numbers))Check: What is the key behavioural difference between a reusable iterable and a single-use iterator when iterated twice?
Section 3
StopIteration is the signal that an iterator uses to announce that it has no more items, and it behaves differently from most exceptions, in that it is caught and handled internally, by looping constructs, rather than propagating to your code as an error, under normal circumstances. You will rarely see it in a program that uses for loops, comprehensions, or built-in functions to consume iterators, because those constructs handle it silently and correctly, by ending the loop, or the comprehension, or the conversion, cleanly, at the point where the iterator is exhausted, without any error or warning reaching your program. That is the intended behaviour for a signal that means, not an error, but simply the end of a sequence, that has been fully consumed, through iteration, to its natural conclusion, in a normal, expected way, for any finite iterator, that will eventually be exhausted, after yielding all of its items, in order, to its consumer. When you call next() manually, however, StopIteration does propagate, and you must handle it, either by catching it in a try block, or by supplying a default value, as the second argument to next(), such as next(it, None), which returns the default, instead of raising, when the iterator is exhausted. That default pattern is convenient for code that needs to peek at the next item, or to consume an iterator, one item at a time, with explicit control over what happens at exhaustion, rather than relying on a loop to handle it implicitly. It appears in programs that process streams, or that merge multiple iterators, where manual next() calls, with defaults, provide a clean way to handle the end of one stream, while other streams continue to yield items, for further processing, in a merged output, that combines items from multiple sources, in a specific order, determined by the merging logic itself. A related rule is that StopIteration raised inside a generator function is converted into a RuntimeError, rather than propagating as StopIteration, to the generator's consumer. That rule prevents a subtle bug, where a generator that calls next() on another iterator, without handling its exhaustion, would terminate silently and prematurely, when the inner iterator is exhausted, because Python would interpret the escaping StopIteration as the generator's own exhaustion, ending it without any error, even though the generator had more work to do, after the inner iterator ended. Converting that escaping exception into a RuntimeError makes the bug visible, by failing loudly at the point where the inner iterator's exhaustion was not handled, rather than silently ending the generator early, with incomplete results, that appear correct, but that are, in fact, truncated, at the point where the inner iterator ended, which is a failure mode far more confusing than a clear error, at the point of the unhandled exhaustion itself. Testing exhaustion behaviour should therefore include both paths: loops that end cleanly, without any exception reaching your code, and manual next() calls, with and without defaults, that behave as documented, at exhaustion.
it = iter([1])
print(next(it))
print(next(it, "done"))Check: How does a for loop handle StopIteration differently from a manual next() call, and what does next(it, default) do at exhaustion?
Section 4
A simple example is a countdown iterator, that yields numbers, from a starting value, down to one, by holding the current value, as its state, decrementing it, on each __next__ call, and raising StopIteration, when the current value, reaches zero. That minimal implementation shows the protocol in its simplest form, without additional complexity, such as handling changes to an underlying collection, or supporting reverse iteration, or other advanced features, that a real collection's iterator might need, to support, for its users, in real programs, where iteration is used extensively, through loops, comprehensions, and built-in functions, that rely on iteration, to work correctly, for all iterable objects, regardless of how they implement iteration internally, whether through explicit iterator classes, or through generator functions, that provide iteration, with less code, and with automatic state management, through suspension, and resumption, of the generator function's execution, at each yield. The trade-off between iterator classes and generator functions is between explicitness and conciseness. An iterator class holds its state in instance attributes, which are accessible to tests and to debugging tools, that can inspect an iterator object's attributes directly, to understand its current position, and its remaining items, in a way that a generator's suspended state cannot easily be inspected, from outside the generator function itself. A generator function, by contrast, holds its state implicitly, in its suspended execution, which is less code, and easier to write correctly, for complex iteration logic, with nested loops and conditional branches, that would be tedious to translate into explicit state variables, and __next__ logic, that advances that state, on each call, in a way that is correct, for all cases, including edge cases, such as empty sequences, single-item sequences, and sequences that terminate early, due to a condition encountered, during generation itself. Choosing between the two should therefore be based on the iteration logic's complexity, and on whether explicit state inspection matters, for debugging, or for testing, in a given program. Simple, regular iteration, such as counting down, or stepping through a collection, with a clear state variable, is well suited to an iterator class, and is a good teaching example, for understanding the protocol, in detail. Complex iteration, with nested loops, or with conditions that determine which items to yield, and in what order, is usually better suited to a generator function, whose sequential code reads naturally, and whose state management is automatic, freeing the author, to focus on the generation logic itself, rather than on the mechanics, of tracking state, across __next__ calls, by hand, in every case, whatsoever.
class Countdown:
def __init__(self, start):
self.current = start
def __iter__(self):
return self
def __next__(self):
if self.current <= 0:
raise StopIteration
value = self.current
self.current -= 1
return value
print(list(Countdown(3)))Check: What state must a custom iterator class track, and when should its __next__ raise StopIteration?
Section 5
Generator functions are the most convenient way to create iterators in Python, because they let you write iteration logic, as ordinary sequential code, with yield statements, marking the points, where an item, is produced, and where, execution, suspends, until, the, consumer, asks, for, the, next, item, by, calling, next(), on, the, generator, object, returned, by, calling, the, generator, function, itself. Calling a generator function does not execute any of its body immediately; instead, it returns a generator object immediately, without executing any code inside the function at all, until the first next() call, which runs the function body until the first yield, returns the yielded value to the caller, and suspends the function, preserving all of its local variables, and its execution position, exactly as they were, at the yield, so that the next next() call resumes execution, immediately after the yield, with all local state intact, as if the function had never paused at all. In a generator function, that logic reads like ordinary sequential code, with loops and conditions, that are easy to understand, and to modify, without needing to translate them into explicit state variables, and __next__ logic, that advances that state, on each call, in a way that is correct, for all cases, including edge cases, such as empty sequences, single-item sequences, and sequences that terminate early, due to a condition encountered, during generation itself, which are cases that generator functions handle naturally, through ordinary control flow, such as return, to end generation early, or break, to exit a loop, that generates items, before its natural termination condition is reached, in the first place. Another benefit of generators is laziness: items are computed only when requested, rather than all at once, up front, which means that a generator over a huge, or infinite, sequence, does not need to compute, or store, all of its items, before the first one can be used, which would be wasteful, or impossible, for an infinite sequence, where there is no complete set of items, to compute in advance, at all. That laziness also means that a consumer can stop early, with a break, or by taking only the first few items, without paying the cost, of generating the rest, which is a significant efficiency benefit, for programs that search for a first matching item, in a large sequence, or that sample the first few items, from a stream, for inspection, before deciding, whether to process the whole stream, or not, at all, in a given program, as a whole, and where, generating items, eagerly, would waste time, and memory, on items, that are never used, by the consumer, in any case, whatsoever.
def countdown(start):
value = start
while value > 0:
yield value
value -= 1
print(list(countdown(3)))Check: How does calling a generator function differ from calling an ordinary function, in terms of when its body first executes?
Section 6
It helps to keep two ideas apart: the generator function and the generator object. A generator function is simply a function definition that contains at least one yield statement. Calling it looks like an ordinary function call, but it behaves very differently. No line of the function body runs when you call it. Instead, the call returns a generator object straight away, and that object holds the suspended state of that particular call: its local variables, its position in the body, and the point at which it last yielded a value.
Each call to the generator function creates a brand new generator object with its own independent state. Two generators made from the same function do not share progress. You can advance one halfway through its sequence while the other has not started, and neither one affects the other. This independence is useful in practice. A program can create several generators from one function for different purposes, passing different arguments to each call, and each generator will follow its own path through the same code, with its own values for the function's parameters and locals.
A generator object is also an iterator, and that fact explains its most important limitation. Its __iter__ method returns itself rather than a fresh iterator, and its __next__ method resumes the suspended function until the next yield or until the function returns. When the function returns, or falls off the end of its body, the generator raises StopIteration and is exhausted. There is no way to rewind it. Its state has reached the end of the function, and no method exists to reset that state back to the beginning.
That is why iterating a generator a second time yields nothing. The second loop does not fail loudly; it simply finds an exhausted iterator and produces no items at all. This silent emptiness is dangerous when a generator is passed to a function that consumes it, and the caller then tries to use it again for another purpose. The remedy is straightforward once you understand the model. If you need the sequence again, call the generator function again to create a fresh generator object. If recreating the sequence is expensive, or its source is no longer available, materialise it into a list on the first pass and reuse the list instead.
def numbers():
yield 1
yield 2
gen = numbers()
print(next(gen))
print(list(gen))
print(list(gen))The first list call drains the remaining item, and the second finds nothing left, which is exactly why a fresh call, not a reused object, is the way to repeat a sequence.
A practical consequence is that functions receiving a generator should treat it as a borrowed, consumable resource. If a function needs to iterate its argument twice, it should either accept an iterable and call iter() on it afresh for each pass, or materialise the argument into a list at the start. Documenting which choice a function makes protects callers from the silent-empty-result trap.
Check: Why is each call to a generator function independent, and why can a generator object itself not be restarted after exhaustion?
Section 7
Beyond simple iteration, generators support a small set of methods that let a consumer interact with a suspended generator. The most interesting of these is send(), along with the related methods throw() and close(). You may not need them often, because a for loop or a comprehension covers most generation tasks. But they are worth understanding at a basic level, since they appear in coroutines and in data processing pipelines where a consumer needs to influence a generator while it is running, rather than only receiving items from it.
The next() function resumes a generator and lets it run until its next yield, returning the yielded value. The send() method does the same thing, but it also delivers a value into the generator. That value becomes the result of the yield expression at which the generator was suspended. In other words, the generator can write value = yield "ready", and when the consumer calls gen.send("hello"), the name value is bound to "hello" and execution continues from just after the yield. The generator can then use the sent value to decide what to yield next, or to adjust its internal state in response to its consumer.
This two-way channel turns a generator from a fixed sequence into an interactive one. A plain generator produces whatever its arguments determined in advance. A generator that receives values through send() can adapt as it runs: it can change direction, accept new parameters, or stop early in response to a signal from its consumer. That is cleaner than sharing state through global variables or mutable objects, because the communication happens at explicit, well-defined suspension points, and each value sent is visible at the exact place where it is received.
There is one detail to remember when starting such a generator. A freshly created generator has not yet reached its first yield, so there is no suspended expression waiting for a value. You must prime it with next() first, or equivalently with send(None), before sending a real value. Sending a non-None value to a just-created generator raises an error, because there is nowhere for that value to go yet.
The close() method provides the other half of this interaction. It raises GeneratorExit inside the generator at its suspension point, giving it a chance to run cleanup code, such as releasing a resource, before it ends. A generator that does not handle the exception simply terminates, which is the normal outcome when a consumer no longer needs further items. The throw() method works similarly, but it raises a chosen exception inside the generator instead, letting the generator handle an error condition from its consumer.
def echo():
value = yield "ready"
yield f"got {value}"
gen = echo()
print(next(gen))
print(gen.send("hello"))Used sparingly, these methods give you a controlled conversation between producer and consumer, without giving up the simplicity of writing generation logic as ordinary sequential code.
Check: What does send() do differently from next() when resuming a suspended generator?
Section 8
A generator expression is the concise way to create a generator without defining a separate function. Its syntax mirrors a list comprehension, but with parentheses instead of square brackets. Writing (x * x for x in range(5)) creates a generator object that yields squares one at a time, on demand. Nothing is computed when the expression is evaluated. The first square is produced only when something asks the generator for an item, and each later square is computed only when it is requested in turn.
That laziness is the whole point, and it is also the difference from a list comprehension. A list comprehension such as [x * x for x in range(5)] computes every item immediately and stores all of them in a new list in memory. A generator expression stores no items at all. It remembers the expression, the source iterable, and its position, and it produces each item just in time. For a small sequence the difference hardly matters. For a large one it can be the difference between a program that runs comfortably and one that exhausts memory by building a list it never needed all at once.
Generator expressions shine as arguments to functions that consume iterables. Functions such as sum(), max(), min(), any(), and all() take any iterable, so you can pass a generator expression directly and let the function pull items through it. When the generator expression is the only argument, its parentheses may be dropped, giving the familiar idiom sum(x * x for x in range(5)). The total is computed without ever building a list of squares, which is both shorter to write and lighter on memory than the list version.
Like generator functions, generator expressions are single-use. Once consumed, they are exhausted, and iterating them again yields nothing. They also support filtering with an if clause, just as comprehensions do, so (x * x for x in range(10) if x % 2 == 0) yields squares of even numbers only. And they chain naturally: the output of one generator expression can feed another, forming a lazy pipeline in which each item passes through every stage before the next item is generated.
squares = (x * x for x in range(5))
print(list(squares))
print(sum(x * x for x in range(5)))Reach for a generator expression when a simple transformation or filter is all you need, and keep the full generator function for logic that needs statements, loops, or cleanup.
One more distinction is worth noting. A generator expression captures its variables lazily in some cases, and its loop variable does not leak into the surrounding scope, just as with comprehensions. If the source iterable is itself a generator, the expression will pull from it as items are requested, so chaining a generator expression onto another lazy source keeps the whole chain lazy. That composability is why generator expressions appear so often as the first stage of a pipeline rather than as a standalone result.
Check: How does a generator expression differ from a list comprehension in terms of when items are computed and stored?
Section 9
A list comprehension is a compact way to build a list by transforming and filtering items from an iterable in a single expression. The expression [x * x for x in range(5)] iterates over range(5), squares each item, and collects the results into a new list. The equivalent loop needs a list created before the loop, an append call inside the loop, and the loop itself. The comprehension states the same operation more directly: its expression part says what each result looks like, and its for part says where the items come from. A reader can see the whole operation at a glance, without tracing control flow through a loop body.
Comprehensions can also filter. Adding an if clause, as in [x * x for x in range(10) if x % 2 == 0], keeps only the items that satisfy the condition, so the result contains squares of even numbers and nothing else. Transformation and filtering combine in one expression, with no need for an if statement and an indented append inside a loop. For simple cases this declarative style is easier to read than the loop, because the intent is visible in the structure of the expression rather than spread across several statements.
But clarity has a limit, and a comprehension can cross it. When the expression grows complex, when several conditions are combined, or when the transformation itself needs intermediate variables or branching, the single-line form becomes dense and hard to parse. A reader must then untangle the expression mentally before understanding what it produces, which defeats the purpose of using a comprehension in the first place. In such cases an explicit loop is clearer, even though it is longer. Named variables, separate statements, and ordinary control flow let a reader follow the logic step by step.
A useful rule of thumb is to keep a comprehension to one transformation and one simple condition. If you find yourself nesting comprehensions, chaining several conditions, or writing an expression that needs a comment to explain it, rewrite it as a loop. The loop will be longer, but length is cheap and confusion is expensive. Programs are read far more often than they are written, and the form that a future reader understands fastest is the right one, even when a shorter form exists.
squares = [x * x for x in range(5)]
evens = [x for x in range(10) if x % 2 == 0]
print(squares, evens)Choose the comprehension when the operation is simple enough to grasp at a glance, and choose the loop whenever the comprehension would need untangling.
There is also a middle ground worth knowing. A comprehension can be split across several lines, with each clause on its own line, which makes longer comprehensions considerably easier to scan. Line breaks do not change the semantics, and a multi-line layout can rescue a comprehension that would be unreadable on one line. Even so, layout alone cannot save a comprehension whose logic is genuinely tangled; at that point the loop remains the honest choice.
Check: When is a list comprehension clearer than an explicit loop with append, and when does a loop become clearer instead?
Section 10
The comprehension idea extends beyond lists to dictionaries and sets, with syntax that mirrors the list form but different brackets and different result types. A dictionary comprehension such as {x: x * x for x in range(5)} builds a mapping from each number to its square. Its expression part is a key-value pair, separated by a colon, and its for part supplies the items. A set comprehension such as {x * x for x in range(5)} uses curly braces without a colon and builds a set of squares instead. The two forms look similar, so the colon is the detail that tells them apart.
Each form inherits the strengths of list comprehensions: a single expression states the whole construction, with no need for an empty collection created beforehand and no explicit assignment or add call inside a loop. A dictionary comprehension is the natural way to build a lookup table from a sequence, for example mapping names to scores or words to counts. A set comprehension is the natural way to collect unique values, because a set removes duplicates automatically. If a transformation produces repeated values and only distinct values matter, a set comprehension deduplicates by construction, with no extra code.
Choosing between the three forms is a decision about the result you need, and it should be made deliberately. Ask first what the result is for. If order matters and duplicates must be preserved, use a list comprehension, since lists keep both. If uniqueness matters and order does not, use a set comprehension, since sets guarantee distinct elements. If each item must be reachable by a key, use a dictionary comprehension, since only a mapping supports lookup by key. Using the wrong form causes subtle bugs: duplicates surviving where they should have been removed, or removed where they should have been preserved, or a lookup that cannot be done efficiently.
All three forms share the same cautions as list comprehensions. Keep the expression and any condition simple, so the result is clear at a glance. A dictionary comprehension whose key expression is complicated, or a set comprehension with a tangled condition, is just as hard to read as a complicated list comprehension, and the same remedy applies: rewrite it as an explicit loop when clarity demands it. The bracket style may change, but the readability rule does not.
squares = {x: x * x for x in range(5)}
unique = {x % 3 for x in range(10)}
print(squares, unique)Let the shape of the result you need decide the form, and let readability decide whether a comprehension is the right tool at all.
A common beginner mistake is confusing the set and dictionary forms, since both use curly braces. The colon is the only visible difference, and forgetting it turns an intended dictionary into a set of keys, or adding one accidentally turns a set into a dictionary. Reading the braces carefully, and testing the result's type on small inputs, catches this mistake immediately, before it propagates into code that expects the wrong collection.
Check: How do you choose between a list, set, and dictionary comprehension for a given transformation result?
Section 11
The yield from statement lets one generator delegate part of its work to another iterable, by yielding all of that iterable's items, in order, as if they were the delegating generator's own items. Writing yield from other is roughly equivalent to a loop that yields each item from other, one at a time, but it is more concise, and it also handles two-way communication correctly. Values sent into the delegating generator, with send(), are passed through to the sub-generator, and values returned by the sub-generator, through its return statement, become the value of the yield from expression, in the delegating generator, which can then use that return value in its subsequent computation, or yield it as its own result, as appropriate for its domain.
That delegation is useful for building generators from smaller pieces, such as a generator that yields items from several sources, in sequence, by delegating to each source, in turn, with yield from, for each one, rather than writing a nested loop, for each source, that yields its items individually. The nested-loop version is more code, and less clear about the structure of the overall generation, as a composition of smaller generators, each responsible for one source, or one stage, of the overall sequence, that the delegating generator produces, for its consumer, whatever that consumer may be, whether a loop, a comprehension, or a built-in function, that relies on iteration, to process items, one at a time, until none remain.
Delegation also clarifies responsibility. Each sub-generator is responsible for its own sequence, and the delegating generator is responsible only for ordering, and combining, those sequences, into a single, coherent, output sequence, that its consumer can iterate, without needing to know, that it is composed, of multiple sources, or multiple stages, of generation, in the first place. That separation of concerns makes each generator easier to understand, easier to test in isolation, and easier to reuse, in other delegating generators, that need the same sub-sequence, for a different overall purpose, or in a different combination, with other sub-generators, that produce different sequences, for different overall outputs, in different programs.
Testing delegation should verify that items appear, in the correct order, across sub-generators, including at the boundaries, between one sub-generator, and the next, where off-by-one errors, or missing items, are most likely, to occur, if delegation is implemented, incorrectly, through manual loops, that mishandle, the transition, between sources, in a way, that yield from, handles, automatically, and correctly, by delegating, the entire sub-iteration, to the sub-generator, itself, without any manual, boundary handling, in the delegating generator's, code, at all. Testing should also verify return values, from sub-generators, when they are used, by the delegating generator, to ensure, that they are, captured, correctly, and, used, appropriately, in subsequent computation, rather than being lost, or misinterpreted, due to a misunderstanding, of how yield from, returns, a sub-generator's, return value, to its delegating generator, as the value, of the yield from, expression, itself.
def combined():
yield from [1, 2]
yield from [3, 4]
print(list(combined()))Check: What does yield from do beyond yielding each item from a sub-iterable, in terms of send() and return values?
Section 12
Infinite generators are generators that never terminate, on their own, because they yield items, indefinitely, in a loop, that has, no, termination, condition, such, as, a, generator, that, yields, counting, numbers, starting, from, zero, and, increasing, by, one, on, each, iteration, without, any, upper, bound, on, the, numbers, it, will, yield, over, time. Such a sequence cannot be materialised as a list, because it has no end, and because attempting to convert it to a list would run forever, consuming memory indefinitely, as it generated more and more items, without ever reaching a termination point, at which list conversion could complete, and return a finished list, to its caller. That limitation of eager materialisation, for infinite sequences, is one that lazy generators overcome, by producing items, on demand, one at a time, only when requested, by a consumer, that controls, how many items, it takes, from the infinite generator, before stopping, deliberately, through a break, a slice, or another stopping condition, that the consumer itself determines, based on its own needs, rather than on the generator's termination, which does not exist, for an infinite generator, by definition.
Consuming an infinite generator safely therefore requires a stopping condition in the consuming code, such as taking only the first ten items, with itertools.islice(), which is a standard tool, for slicing iterators, including infinite ones, without materialising them, by yielding only the requested number of items, from the iterator, and then stopping cleanly, without consuming any further items, from it. That is a safe and efficient way, to sample an infinite sequence, for inspection, or for processing, of its first few items, in a program, that needs only those items, for its immediate purpose, and that does not need the entire infinite sequence, which cannot be processed in full, in any case, whatsoever, regardless of the program's resources, or its available time, for processing that sequence, as a whole.
Infinite generators are useful for modelling sequences that are conceptually unbounded, such as counting numbers, timestamps at regular intervals, or a stream of events arriving over time, from a source, that does not have a predetermined end point, at which it will stop producing events, for its consumers, to process, in real time, as they arrive, rather than in a batch, after the source has finished producing them, which may never happen, for a continuous stream, that is intended to run indefinitely, for the life of a program, that monitors it continuously, for new events, as they occur, in the world outside the program itself. In those cases, infinite generators provide a natural model, for such streams, by yielding each event, as it arrives, without needing to know, in advance, when, or whether, the stream will ever end, at all, in the first place.
import itertools
def count_up():
n = 0
while True:
yield n
n += 1
print(list(itertools.islice(count_up(), 5)))Check: Why must consuming code, not the generator itself, supply the stopping condition for an infinite generator?
Section 13
Generator pipelines chain multiple generators together, so that the output of one stage becomes the input of the next stage, with each stage transforming, or filtering, items, as they pass through it, one at a time, without materialising intermediate collections, between stages. That pattern is powerful for processing large data sets, in a memory-efficient way, because each item is processed through the entire pipeline, before the next item is generated, by the first stage, which means that memory usage stays roughly constant, regardless of how many items the pipeline processes, in total, over its lifetime. That benefit matters for programs that process files with millions of lines, or streams with unbounded numbers of events, where materialising intermediate collections would be wasteful, or impossible, due to memory constraints, in a given program, or machine, where memory is limited, and where processing items one at a time, through a pipeline, is far more efficient, than materialising a full list, at each stage, and then processing it, item by item, in a second pass, that could have been combined, with generation itself, into a single lazy pipeline.
A simple pipeline might read lines from a file, filter out blank lines, strip whitespace from the remaining lines, and transform them in some way, such as converting them to uppercase, or parsing them into structured records, for further processing downstream, in the pipeline itself, with each stage implemented as a generator function, or a generator expression, that takes an iterable as input, and yields transformed items as output, for the next stage to consume, in turn, without knowing how the previous stage produced its items, or how the next stage will consume them. That separation of concerns makes each stage easier to understand, easier to test in isolation, and easier to reuse, in other pipelines, that need the same transformation, or filter, for a different overall purpose, or in a different combination, with other stages, that produce different overall outputs, in different programs.
Testing pipelines should include testing each stage in isolation, with small known inputs, and expected outputs, to verify that each stage transforms, and filters, correctly, on its own, without interference, from other stages, in the pipeline itself, as well as testing the pipeline as a whole, with realistic inputs, to verify that stages compose correctly, in sequence, without losing items, duplicating items, or reordering items, in ways that the pipeline's specification does not intend, for its output, as a whole. Testing should also verify laziness, by checking that a pipeline does not consume its entire input, when only its first few outputs are requested, such as by taking only the first item, from a pipeline over an infinite input, and verifying that it returns quickly, without attempting to process the entire infinite input first, which would never terminate, and which would indicate that a stage in the pipeline is materialising its input eagerly, rather than processing it lazily, one item at a time, as a pipeline stage should do, by design, in the first place.
def lines():
yield " hello "
yield ""
yield " world "
clean = (line.strip() for line in lines() if line.strip())
print(list(clean))Check: Why does a generator pipeline use roughly constant memory regardless of how many items it processes in total?
Section 14
Nested comprehensions, and comprehensions with multiple for clauses, extend the comprehension idea, to more complex transformations, by allowing a comprehension to iterate over multiple iterables, in sequence, or to nest one comprehension inside another, for creating nested structures, such as a list of lists, representing a grid, or a matrix, of values, derived from rows and columns, of that grid itself, through a comprehension that iterates over rows, in its outer for clause, and over columns, in its inner for clause, producing a value, for each combination, of row and column, in order, from first to last, for each dimension, in turn. That is a concise way, to create grids and matrices, without needing nested loops, with explicit append calls, for each row, and each cell, within that row, itself, which is more code, for the same result, and which is less clear, about its intent, than a comprehension, that states its grid structure, directly, in its nested for clauses, and its value expression, in a single readable expression, that can be understood, at a glance, by a reader familiar, with comprehension syntax, in general, and with nested comprehensions, in particular, as a common idiom, in Python code, for creating nested structures, from flat, or nested, source data, in a concise and expressive way. Multiple for clauses, in a single comprehension, such as [x + y for x in [1, 2] for y in [10, 20]], create a flat list, of combinations, by iterating over the first iterable, in its outer loop, and over the second iterable, in its inner loop, for each item, from the first iterable, in turn, which is equivalent to nested loops, in the same order, with the leftmost for clause, corresponding to the outermost loop, and the rightmost for clause, corresponding to the innermost loop, in the equivalent nested loop structure, itself. That correspondence should be understood, when reading, or writing, comprehensions, with multiple for clauses, because reversing the order, of the clauses, changes the order, of combinations, in the result, in a way, that may not be obvious, from reading the comprehension quickly, without thinking carefully, about which clause is outer, and which is inner, in the equivalent nested loop structure, as a whole, and where the order of clauses matters, for correctness, of programs, that depend on the order, of combinations, in their results, such as programs, that generate test cases, or that process combinations, in a specific order, determined by their specification, itself.
grid = [[r * 10 + c for c in range(3)] for r in range(3)]
pairs = [(x, y) for x in [1, 2] for y in [10, 20]]
print(grid, pairs)Test nested comprehensions with small grids first, where the expected output is easy to check by hand, before using them for larger data.
Check: In a comprehension with multiple for clauses, which clause corresponds to the outermost loop, and why does clause order matter?
Section 15
Comprehension scope, in Python 3, is a detail, that matters, for correctness, when comprehensions use variables, that also appear, outside the comprehension, itself, because, in Python 3, a comprehension has its own scope, for its loop variables, which means that a loop variable, inside a comprehension, does not leak out, into the surrounding scope, after the comprehension completes, unlike in Python 2, where loop variables, from comprehensions, did leak, into the surrounding scope, and could overwrite variables, with the same name, outside the comprehension, itself, causing confusing bugs, where a variable's value, changed unexpectedly, after a comprehension executed, in the same scope, as that variable, which is a behaviour, that Python 3 changed, deliberately, to prevent such leakage, and to make comprehensions, safer to use, in programs, where variable names, may be reused, in different contexts, without intending to interfere, with each other, across those contexts, as a whole. That new scope rule means, that you can use a loop variable name, inside a comprehension, without worrying, that it will overwrite, a variable, with the same name, outside the comprehension, itself, which is a safety benefit, that should be relied on, when writing comprehensions, in Python 3 code, as a whole.
That scope rule also means, that a comprehension cannot assign, to variables, outside itself, through its loop variables, alone, and that any variable, you need, after a comprehension completes, must be assigned, explicitly, outside the comprehension, itself, rather than relying, on a loop variable, from inside the comprehension, to persist, after it completes, which is a distinction, that matters, for programs, that need a value, derived from a comprehension's iteration, after the comprehension itself, has finished executing, in the first place, as a whole. The correct approach, is to assign the comprehension's result, to a variable, outside the comprehension, and then use that result variable, after the comprehension completes, rather than trying to use a loop variable, from inside the comprehension, which will not be available, in the surrounding scope, after the comprehension completes, in Python 3, at all, in any case, whatsoever, as a whole.
Testing comprehension scope, is straightforward, by checking that a variable, with the same name, as a comprehension's loop variable, retains its original value, after the comprehension executes, and that attempting to use the loop variable itself, after the comprehension, raises a NameError, if it was not defined, before the comprehension, itself, which are tests, that verify the scope rule, directly, and that prevent confusion, about whether a loop variable, is available, after a comprehension, in a given program, where that availability, matters, for correctness, of subsequent code, that may, or may not, rely on the loop variable's value, after the comprehension completes, in the first place, as a whole. Understanding the scope rule, helps you to write comprehensions, safely, without worrying about variable leakage, or about accidental overwriting, of variables, outside the comprehension, itself, in any program, whatsoever, as a whole.
x = 99
squares = [x * x for x in range(3)]
print(x, squares)Check: Why does a comprehension's loop variable not leak into the surrounding scope in Python 3, and what does that mean for using it afterwards?
Section 16
Conditional expressions inside comprehensions allow a comprehension to transform items differently, depending on a condition, for each item, such as [x if x >= 0 else 0 for x in [-1, 2, -3, 4]], which creates a list, where negative numbers, are replaced, by zero, and non-negative numbers, are kept, unchanged, by using a conditional expression, in the comprehension's value part, before its for clause, itself. That is different, from a filtering condition, after the for clause, that excludes items, entirely, from the result, rather than transforming them, into a different value, for inclusion, in the result, itself, which is a distinction, that matters, for correctness, when choosing, between a conditional expression, and a filter condition, in a comprehension, based on whether an item, should be transformed, or excluded, for a given condition, in a given program, where that distinction, determines, whether the result, contains the same number, of items, as the source, with some transformed, or fewer items, than the source, with some excluded, entirely, from the result, itself, as a whole.
That distinction should be made deliberately, based on the intended result, for each item, that fails the condition, in question, whether it should appear, in the result, in a transformed form, or not appear, in the result, at all, which is a decision, that affects the length, of the result, and its correspondence, to the source items, in order, and in position, where a conditional expression, preserves length, and positional correspondence, between source and result, while a filter condition, breaks that correspondence, by removing items, from the result, entirely. That difference matters, for programs, that rely on positional correspondence, between source and result, such as programs, that process parallel lists, by index, or that align results, with source items, for display, or for further processing, that assumes the same length, and order, for both source and result, as a whole, and where using the wrong form, for a given purpose, can cause subtle bugs, due to length mismatches, or positional misalignment, between source and result, that are hard to diagnose, because the comprehension itself, looks correct, at a glance, while its result, has the wrong length, or alignment, for its intended use, in a given program, as a whole.
Keep conditional expressions simple, so a comprehension stays readable at a glance, rather than becoming a dense expression that would be clearer as a loop.
clamped = [x if x >= 0 else 0 for x in [-1, 2, -3, 4]]
positives = [x for x in [-1, 2, -3, 4] if x >= 0]
print(clamped, positives)A comprehension that combines a conditional expression with a filter condition deserves extra care, because a reader must distinguish which condition transforms an item and which one excludes it entirely.
Check: What is the difference between a conditional expression in a comprehension's value part and a filter condition after its for clause?
Section 17
Choosing between lazy and eager evaluation is one of the central decisions in this chapter, because it shapes memory use, performance, and even whether a computation is possible at all. Lazy evaluation, through generators and generator expressions, produces items on demand, one at a time, without storing them all. Eager evaluation, through list, dictionary, and set comprehensions, produces and stores every item up front, in a new collection, before any further processing begins. Neither approach is always better. The right choice depends on how the result will be used, and that question should be asked deliberately for each case.
Lazy evaluation is the right choice when the data is large, unbounded, or only partly needed. If storing every item at once would be wasteful or impossible, a generator avoids the problem entirely by never holding more than one item at a time. If the sequence is infinite, eager materialisation is impossible by definition, since there is no complete set of items to store. And if the consumer may stop early, after finding a desired item or taking a few samples, laziness avoids wasted work on items that are never used. Eager evaluation would compute all of them regardless, paying the full cost for a partial result.
Eager evaluation is the right choice when the result will be reused. A generator is single-use, so iterating it a second time yields nothing. If the data must be scanned several times, indexed by position, searched repeatedly for membership, or measured with len(), a materialised collection supports all of those operations directly, while a generator would have to be recreated on every pass. Regeneration is wasteful when generation is expensive, and impossible when the source is gone, for example a file that has been closed or a stream that has already been consumed. Materialising on the first pass preserves the data for later passes without depending on the original source.
There is also a middle path, and it is often the best one. A pipeline can stay lazy through its early stages, where streaming saves memory, and materialise at the point where reuse or random access becomes necessary. That way each stage uses the tool that fits its own requirements, instead of forcing the whole program into one mode. The decision is therefore not a global preference for lazy or eager code, but a local judgment, stage by stage, about what each result must support.
lazy = (x * x for x in range(5))
eager = [x * x for x in range(5)]
print(list(lazy), eager[2], len(eager))Ask how the result will be consumed, how often, and whether its source will still exist, and the lazy-versus-eager choice will usually answer itself.
A final consideration is error timing. Eager evaluation surfaces errors during construction, since every item is computed up front, while lazy evaluation defers errors until the faulty item is actually requested, which may be much later in the program. When early error detection matters, eager construction can be the safer choice, even for data that would otherwise suit laziness.
Check: When should you prefer a lazy generator over an eager list, and when is materialising into a list the better choice?
Section 18
The itertools module provides a collection of iterator tools, that are useful, for common iteration tasks, such as chaining multiple iterables, into a single sequence, slicing iterators, without materialising them, grouping items, by a key function, and generating combinations, and permutations, of items, from a given iterable, itself. Those are operations, that would otherwise require, custom iterator code, to implement, correctly, and efficiently, for each use case, in a given program, where those operations, are needed, for data processing tasks, that involve combining, slicing, grouping, or generating combinations, of items, from one, or more, source iterables, in a specific order, determined by the operation itself, rather than by ad hoc code, written for each use case, separately, which would be more code, and more error-prone, than using a standard tool, from itertools, that has been tested, and optimised, for its specific operation, in the first place, as a whole. The itertools tools are designed, to work lazily, with iterators, producing items, on demand, one at a time, without materialising intermediate collections, between operations, which is a memory benefit, for large data sets, where materialising intermediate collections, would be wasteful, or impossible, due to memory constraints, in a given program, or machine, where memory is limited.
Common itertools tools include chain(), which chains multiple iterables, into a single sequence, by yielding items, from each iterable, in turn, until all iterables, are exhausted, in sequence, itself; islice(), which slices an iterator, by yielding only a specified range, of its items, without materialising the entire iterator, first, itself; zip_longest(), which zips multiple iterables, together, padding shorter ones, with a fill value, when they are exhausted, before longer ones, in the zip, itself; and groupby(), which groups consecutive items, with the same key function result, into groups, that can be iterated, as separate iterables, for further processing, downstream, in a pipeline, itself. Those tools cover many common iteration patterns, without needing custom code, for each pattern, in a given program, where those patterns, appear frequently, in data processing tasks, that involve combining, slicing, zipping, or grouping, items, from one, or more, source iterables, in a specific order, determined by the operation itself, rather than by ad hoc code, written for each use case, separately, as a whole.
Using itertools tools correctly, requires understanding, that they return iterators, not lists, and that those iterators, are single-use, and lazy, like other iterators, discussed in this chapter, which means, that they can be iterated, only once, and that their items, are produced, on demand, rather than all at once, up front, which is a benefit, for memory efficiency, and a limitation, for reusability, that must be considered, when choosing, between an itertools iterator, and a materialised list result, for a given use case, in a given program, where reusability, matters, for correctness, of subsequent operations, that need to iterate the result, multiple times, or to index it, by position, which are operations, that lazy iterators, do not support, efficiently, without materialisation, into a list, first, itself, as a whole.
import itertools
print(list(itertools.chain([1, 2], [3, 4])))
print(list(itertools.islice(range(10), 3)))Check: Why do itertools tools return lazy single-use iterators rather than lists, and what does that imply for reuse?
Section 19
Writing a custom iterable class that is reusable comes down to one design rule: its __iter__ method must return a fresh iterator on every call. That is the difference between an iterable and an iterator. A class that implements both __iter__ and __next__ and returns itself from __iter__ is a single-use iterator, not a reusable iterable. Once it is exhausted, it stays exhausted, and a second loop over it yields nothing. A class whose __iter__ returns a new iterator object each time is a reusable iterable. Every iteration starts from the beginning and yields the same items in the same order, provided the contents have not changed between iterations.
Reusability matters most when iterators must coexist. In a nested loop, an outer loop and an inner loop both iterate over the same collection at the same time, and each needs its own position. A reusable iterable gives each loop its own iterator, so the inner loop does not disturb the outer loop's progress. A single-use iterator cannot support that pattern at all, because the inner loop would consume the very iterator the outer loop is using. The same problem appears whenever a collection is passed to two consumers, or iterated once to validate it and again to process it.
The simplest way to build a reusable iterable is to define __iter__ as a generator method. A method that contains yield is a generator function, so each call to it returns a fresh generator object with its own independent state. Writing __iter__ as a method that yields items from an internal collection, one at a time, therefore gives you reusability for free, with no separate iterator class and no manual position tracking. The generator handles suspension and resumption, and each call starts cleanly from the first item.
This approach combines the convenience of generators with the reusability of iterables. The class itself stays simple, because its __iter__ method reads like ordinary sequential code that yields items in order, rather than like a separate iterator class whose state must be understood apart from the collection. Edge cases such as empty collections are handled naturally by the loop inside the method, which simply yields nothing. If you need explicit state inspection for debugging, a separate iterator class may still be justified, but for most collections the generator method is the clearer design.
class Shelf:
def __init__(self, books):
self.books = books
def __iter__(self):
yield from self.books
shelf = Shelf(["A", "B"])
print(list(shelf), list(shelf))Both lists contain the same items, which is the visible proof that the class is reusable rather than single-use.
One further benefit is that a generator-based __iter__ automatically supports multiple simultaneous iterations without any extra bookkeeping. Each for loop over the object calls __iter__ and receives its own generator, so nested loops and parallel consumers never interfere. A hand-written iterator class would need a separate iterator object per call to achieve the same safety, which is exactly what the generator method provides for free.
Check: How does defining __iter__ as a generator method make a class a reusable iterable rather than a single-use iterator?
Section 20
The first twenty sections of this chapter have covered iteration mechanics, the iterable versus iterator distinction, StopIteration, custom iterator classes, generator functions, generator state, advanced generator methods, generator expressions, list, dictionary, and set comprehensions, yield from, infinite generators, generator pipelines, nested comprehensions, comprehension scope, conditional expressions, lazy versus eager evaluation, itertools tools, and custom iterable classes. Together, those topics form a comprehensive toolkit, for iteration in Python, that goes beyond simple for loops, over lists, and that prepares you, for processing data, of many kinds, in many ways, all through the same simple iteration protocol, that underlies all of Python's looping constructs, from for loops, to comprehensions, to built-in functions, that consume iterables, in their implementations, themselves, as a whole. The remaining sections, of this chapter, will develop those concepts, further, with more advanced topics, such as iterator invalidation, during mutation, custom iterator design patterns, generator cleanup, and error handling, and performance considerations, for choosing, between lazy and eager approaches, in real programs, that process large data sets, efficiently, and correctly, in a memory-efficient way, as a whole.
The principles, that recur, across all of these topics, are worth naming, explicitly, because they transfer, to iteration tasks, you have not yet met, and to new iteration patterns, you may design, yourself, in future programs, that process data, in ways, that are not covered, by the examples, in this chapter, alone, but that follow, the same principles, of laziness, reusability, and protocol consistency, that govern, all iteration, in Python, as a whole. Laziness means, producing items, on demand, rather than all at once; reusability means, distinguishing, between iterables, that can be iterated, many times, and iterators, that can be iterated, only once; and protocol consistency, means, ensuring, that related iteration methods, agree, with each other, such as iteration, agreeing, with membership, and with length, for collections, that support those operations, in addition to iteration, itself, as a whole. Together, those principles, form a practical framework, for designing iteration behaviour, in any class, or generator, that you write, in future programs, that process data, through iteration, in any form, whatsoever, as a whole.
class Book:
def __init__(self, title):
self.title = title
print(Book("Dune").title)Keep those three principles in mind as you work through the rest of this chapter, and test each new iteration pattern against them: does it produce items lazily when laziness matters, does it behave reusably or single-use exactly as documented, and do its related methods agree with each other in every context where they will be used? That framework will serve you well beyond this chapter, whenever you design a new iterable, a new generator, or a new data-processing pipeline in a real program. Use the remaining sections to practise those principles on harder cases, where the right choice is less obvious than in the examples so far.
Check: How do laziness, reusability, and protocol consistency together form a framework for designing iteration behaviour?
Section 21
Mutating a collection while iterating over it is one of the most common iteration bugs in Python, and it deserves a clear rule: do not add or remove items from a list, set, or dictionary while a loop is iterating over it, unless you have a specific, well-tested reason, and you understand exactly how that mutation affects the iterator's position. The problem is that an iterator tracks its position in the underlying collection, and changing the collection's size, or its internal layout, can shift items, so that the iterator skips some items, visits others twice, or, in the case of dictionaries and sets, raises a RuntimeError for changing size during iteration. Those outcomes depend on internal details that are not part of the collection's public interface, which makes the behaviour hard to predict, and hard to test, for all cases, without knowing those internals. A classic example is removing items from a list inside a loop over that same list. When an item is removed, the items after it shift down by one position, but the iterator's index advances by one, as usual, so the item that shifted into the removed item's position is skipped, without ever being examined, by the loop's body. That skipping is silent: the loop completes normally, with no error, but with some items never processed, which is a bug that may not be noticed, until a test, or a user, discovers that certain items were missed, in a specific case, where removal happened, at a specific position, in the list, during that particular iteration, as a whole. The fix is simple, and should become a habit: iterate over a copy, such as list(items), while mutating the original, or build a new list, with a comprehension, that contains only the items you want to keep, rather than removing unwanted items, from the original, during iteration, itself. Dictionaries and sets are stricter: Python detects size changes during iteration, and raises a RuntimeError, which is a loud failure, that at least prevents silent skipping, but that still breaks the program, at the point where the mutation occurs, rather than at a point, where it can be handled, cleanly, by the program's logic, itself. The same remedies apply: iterate over a copy of the keys, such as list(d), while mutating the dictionary, or build a new dictionary, with a comprehension, that contains only the entries you want to keep, rather than deleting entries, from the original, during iteration, over it, itself. Those remedies are simple, and they should be used routinely, whenever a loop needs to filter, or update, a collection, based on its own contents, during iteration, over that collection, as a whole. There is one mutation that is safe during iteration: changing the value of an existing item, or an existing dictionary entry, without changing the collection's size, or its keys, does not disturb an iterator's position, because no items shift, and no internal layout changes, in a way, that affects iteration, itself.
items = [1, 2, 3, 4]
kept = [x for x in items if x % 2 == 0]
print(kept)Check: Why does removing items from a list during iteration over that same list cause some items to be skipped silently?
Section 22
Iterator invalidation is the general term for what happens when an iterator's underlying collection changes, in a way, that makes the iterator's position, or its assumptions, about that collection, no longer valid, for continuing iteration, correctly, in the first place, as a whole. For lists, invalidation is silent, as discussed in the previous section, where removals cause skipping, without any error, which is the most dangerous form, of invalidation, because it produces incorrect results, without any warning, that something went wrong, during iteration, itself. For dictionaries and sets, invalidation is loud, because Python tracks a version counter, for those collections, and raises a RuntimeError, when an iterator detects, that the version has changed, during iteration, due to a size-changing mutation, such as adding, or deleting, an entry, during a loop, over that dictionary, or set, itself, which is a failure, that at least prevents, silent incorrect results, but that still breaks the program, at an unexpected point, during its execution, as a whole. Another approach, to avoiding invalidation, is to build a new collection, rather than mutating an old one, during iteration, which is the functional style, favoured by comprehensions, where a new list, dictionary, or set, is created, from an old one, through transformation, and filtering, without modifying the old one, at all, during the comprehension's execution, itself, which is safe, by construction, because the source collection, is never mutated, during iteration, over it, and because the result collection, is a new object, that is not being iterated, while it is being built, in the first place, as a whole. That style, is not always, appropriate, for programs, that need to update, a large collection, in place, for memory, or performance, reasons, where building a new collection, would be wasteful, or impractical, due to the collection's size, or due to references, held elsewhere, to the original collection, object, that must see, the updates, made, during iteration, itself, but, for many, ordinary, programs, building a new collection, is the simplest, and safest, way, to avoid, invalidation, entirely, without needing, to reason, about which mutations, are safe, during iteration, in a given case, whatsoever, as a whole. Testing for invalidation, should include tests, that mutate a collection, during iteration, in the ways, that a program, might do, by accident, or by design, to verify, that those mutations, either, raise a clear error, for dictionaries, and sets, or, are handled, correctly, through copying, or through building, a new collection, for lists, where silent skipping, would otherwise occur, without any warning, that something went wrong, during iteration, itself, which are tests, that catch invalidation bugs, early, during development, rather than late, during user runs, where they appear, as missing, or duplicated, items, in results, that should have been, complete, and correct, for their intended purpose, in a given program, as a whole.
data = {"a": 1, "b": 2}
for key in list(data):
if data[key] == 1:
del data[key]
print(data)Check: Why does Python raise RuntimeError for size changes during dictionary iteration, while list iteration fails silently by skipping items?
Section 23
The built-in functions enumerate() and zip() are iteration helpers, that deserve a focused treatment, because they appear, in almost every Python program, that processes collections, and because using them, correctly, makes iteration code, clearer, and less error-prone, than manual alternatives, such as tracking an index, with a separate counter, variable, that must be initialised, incremented, and used, correctly, on every iteration, of a loop, over a collection, itself, which is a pattern, that is easy, to get wrong, through forgetting, to increment, the counter, or through incrementing, it, in the wrong place, in the loop's body, causing off-by-one errors, that are hard, to spot, by reading the code, quickly, without tracing, its execution, step by step, through each iteration, of the loop, as a whole. enumerate() solves that problem, by yielding pairs, of index and item, for each item, in an iterable, in order, starting from zero, by default, or from a specified start, value, if given, as its second argument, which is a clean way, to obtain both, an item, and its position, in a single loop, without manual counter management, at all, in the first place, as a whole. Both helpers, return iterators, not lists, which means, that they are lazy, and single-use, like other iterators, discussed in this chapter, and that their results, should be materialised, into a list, if they need, to be iterated, multiple times, or indexed, by position, which are operations, that lazy iterators, do not support, efficiently, without materialisation, into a list, first, itself, as a whole. That laziness, is a benefit, for memory efficiency, when processing large, or infinite, iterables, in parallel, or with indices, because it avoids, materialising, all pairs, or tuples, at once, before processing begins, in the first place, as a whole, and where, the helpers, themselves, are implemented, efficiently, in C, for built-in, iterables, making them, faster, than manual, counter, or indexing, code, written, in Python, for the same purpose, in a given program, as a whole. Choosing, between enumerate(), zip(), and manual, indexing, should, therefore, favour, the helpers, in almost, all, cases, because, they, are, clearer, safer, and, faster, than, manual, alternatives, for, the, same, tasks, in, real, programs, where, iteration, with, indices, or, in, parallel, is, a, common, requirement, for, data, processing, tasks, that, involve, multiple, collections, or, positional, information, about, items, within, a, single, collection, itself, as, a, whole, and, where, manual, alternatives, should, be, reserved, for, cases, where, the, helpers, cannot, express, the, required, iteration, pattern, directly, such, as, iterating, with, a, step, size, greater, than, one, or, with, a, non-standard, indexing, scheme, that, enumerate(), or, zip(), do, not, support, directly, in, their, standard, forms, as, a, whole.
names = ["Ada", "Bo"]
for i, name in enumerate(names, start=1):
print(i, name)
for a, b in zip([1, 2], ["x", "y"]):
print(a, b)Check: Why is enumerate() safer than tracking an index with a manual counter variable inside a loop?
Section 24
Sorting with a key function is an iteration-related topic, because sorted() consumes an iterable and produces a new list in sorted order, without modifying the original. The key function determines the ordering criterion by extracting a comparison key from each item. Sorting books by title, for example, needs only a small function that returns each book's title. That is cleaner than defining a total ordering for the class itself, especially when different contexts need different orderings. One program may sort books by title and another by author, and a key function lets each caller choose its criterion without changing the class.
Key functions are also efficient, because Python computes each item's key exactly once, before sorting begins. This is sometimes called the decorate-sort-undecorate pattern, although sorted() handles it internally and you never see the decorated form. Computing each key once matters when key extraction is expensive, for example pulling a field out of a nested structure or deriving a value through a function call. The alternative, a comparison function called repeatedly during sorting, would recompute the same keys many times for the same pairs of items, wasting work on every comparison. With a key function, the expensive part happens once per item, and sorting itself compares only the precomputed keys.
Sorting in Python is stable, which means that items with equal keys retain their original relative order in the result. Stability is more useful than it first sounds. It enables multi-criteria sorting by successive passes: sort first by a secondary criterion, then by the primary criterion, and the second sort preserves the first sort's order among items that tie on the primary key. An alternative is a single key function returning a tuple of criteria in priority order, which states all criteria at once and is often clearer. Both techniques rely on stability, or on tuple comparison, to produce a well-defined order when keys tie.
Testing sorting should cover the cases where it most often goes wrong. Test stability with equal keys, to confirm that tied items keep their input order. Test that the key function extracts the intended criterion from each item, not a neighbouring field. Test empty and single-item inputs, where edge cases in key logic are most likely to appear. And test sorting an iterator as input, to confirm that sorted() consumes it correctly and returns a new list, leaving the original iterator exhausted and unusable afterwards.
books = [{"title": "Zebra"}, {"title": "Apple"}]
print([b["title"] for b in sorted(books, key=lambda b: b["title"])])A good key function states the ordering criterion once, lets Python compute it once per item, and leaves stability to handle the ties.
Another practical point is that the key function may return values of any comparable type, not only numbers or strings. Sorting by a date, by a tuple of several fields, or by a computed score all work the same way, provided the keys are mutually comparable. Mixing incomparable key types, such as numbers and strings, will raise an error during comparison, so key functions should return a consistent type for every item.
Check: Why does sorted() compute each item's key exactly once, and why is sorting stability useful for multi-criteria sorts?
Section 25
The built-in functions map() and filter() offer an older way to transform and filter iterables, predating comprehensions in Python's history. They remain useful in specific cases, even though comprehensions are generally preferred for readability in modern code. The preference for comprehensions is a guideline, not an absolute rule, and understanding where map() and filter() are clearer will help you choose well. The deciding factor is usually whether the function you need already exists with a name.
map() is at its clearest when the transformation function already exists as a named function. Writing map(str.upper, words) states the operation directly: apply an existing function to each item. The equivalent comprehension, [str.upper(word) for word in words], introduces a loop variable that adds nothing to the reader's understanding, since the transformation is already fully described by the function name. In that case the map() call is shorter and arguably clearer, because it contains no moving parts beyond the function and the iterable.
filter() follows the same logic. When a predicate function already exists, filter(is_valid, records) states the filtering directly, without a loop variable or a condition expression that merely calls the predicate. The comprehension form, [x for x in records if is_valid(x)], is not wrong, but it wraps a simple idea in extra syntax. As with map(), the advantage disappears when the predicate must be written as a lambda. A comprehension then states the condition more readably than a lambda passed to filter().
Both functions return lazy iterators, not lists, which means they compose into pipelines like generator pipelines. A map() stage can feed a filter() stage, and each item passes through the whole chain before the next item is produced, with no intermediate collections. That laziness is a genuine benefit for large inputs, and it is one reason these functions have survived alongside comprehensions rather than being replaced by them.
The practical rule is therefore about readability for the specific case. Use a comprehension when the transformation or filter needs a lambda or a complex expression. Use map() or filter() when a named function already says exactly what you mean. And keep the choice consistent within a program, since mixing forms arbitrarily for similar tasks makes code harder to follow than either form used consistently.
print(list(map(str.upper, ["a", "b"])))
print(list(filter(lambda x: x % 2 == 0, [1, 2, 3, 4])))When the function already has a name, let map() or filter() use it directly, and save comprehensions for logic that needs expressing inline.
It is also worth noting that map() with several iterables applies its function to items taken in parallel, stopping when the shortest iterable is exhausted, which mirrors zip() behaviour. That form is rarer in everyday code, but it shows that map() is a general tool for applying functions across iterables, not merely a loop replacement. When the function takes one argument and the iterable is a single source, the single-iterable form is the one you will reach for most often.
Check: When is map() with an existing named function clearer than an equivalent list comprehension?
Section 26
The aggregation functions any(), all(), sum(), min(), and max() consume an iterable and reduce it to a single result. Each embodies a different summary: whether any item satisfies a condition, whether all items do, the total of the items, or the smallest or largest item. These operations appear constantly in programs that process collections, because summarising data into one value is often the step before a decision or a display. All five accept any iterable, including generators, and they consume items one at a time rather than requiring a materialised list first.
Two of these functions can short-circuit, and that property matters a great deal. any() returns True as soon as it finds a truthy item, without examining the rest. all() returns False as soon as it finds a falsy item, for the same reason. Short-circuiting is what makes these functions usable on infinite iterables: any() can answer a question about an endless sequence, provided a matching item appears early enough, without ever consuming the whole sequence. That would be impossible for a function that had to see every item before deciding.
By contrast, sum(), min(), and max() must consume their entire input. A later item could always change the total, or displace the current minimum or maximum, so no early answer is safe. On an infinite iterable these functions never return, and on a very large one they pay the full cost of consuming everything. That distinction, between short-circuiting aggregations and full-consumption aggregations, should guide your choice whenever the input is large or unbounded. If you only need to know whether a condition ever holds, any() gives you that answer as cheaply as possible.
Aggregation functions also pair naturally with generator expressions. Writing sum(x * x for x in range(5)) computes the sum of squares lazily, with each square produced and consumed in turn and none stored. The idiom combines the conciseness of comprehension syntax with the memory efficiency of generators, and it is one of the most common patterns in Python code. The same form works for any() and all(), where a generator expression supplies the condition to test for each item.
print(any(x > 3 for x in [1, 2, 5]))
print(all(x > 0 for x in [1, 2, 3]))
print(sum(x for x in [1, 2, 3]))Before choosing an aggregation, ask whether it can stop early, and whether the input might be too large, or too endless, for full consumption.
A subtle point about min() and max() is that they accept an optional key function, just like sorted(), and an optional default value returned for empty iterables instead of raising an error. These options make the aggregation functions more flexible than they first appear: you can find the longest word, the cheapest item, or the earliest date with a single call, and handle the empty case gracefully without a separate length check beforehand.
Check: Why can any() return early on an infinite iterable, while sum() cannot?
Section 27
Flattening a nested iterable means turning a collection of collections into a single flat sequence of items, in order. A list of lists, for example, becomes one list containing every item from every inner list, first group to last, and first item to last within each group. The task appears wherever data is stored grouped but must be processed together: chapters containing sections, shelves containing books, or files containing lines. Grouping is useful for organisation, but many operations, such as counting, searching, or transforming every item, are simpler once the grouping is removed.
The simplest way to flatten one level of nesting is a comprehension with multiple for clauses, such as [item for group in groups for item in group]. The clauses read like nested loops in the same order, with the leftmost clause as the outer loop and the rightmost as the inner loop. Once that correspondence is understood, the comprehension is easy to read: for each group in turn, for each item in that group, produce the item. An alternative for the same one-level job is itertools.chain.from_iterable(), which yields items from each nested iterable in turn without building the flat list at once. The lazy version is preferable when the nested data is large and only a single pass is needed.
Deeply nested structures, with several levels of grouping, need recursive flattening. A generator function can walk the structure, yielding from each nested iterable recursively until it reaches a leaf item, and the recursion handles arbitrary depth without the caller knowing how deep the nesting goes. But one type needs special handling: strings. A string is iterable, so a naive recursive flattener would split every string into its individual characters, which is almost never what is wanted. Strings in data usually represent single values, such as names or messages, not collections of characters to be flattened.
The remedy is to check for strings before treating an item as an iterable to flatten. A common pattern is to test whether the item is a string, or more generally whether it is an instance of the types that should count as leaves, and yield it directly in that case. Only non-string iterables are recursed into. Forgetting this check is an easy mistake, because a generic flattening function that treats all iterables uniformly looks correct until it meets its first string and shatters it. Testing with string items included is therefore essential, not optional, for any flattening code.
groups = [[1, 2], [3], [4, 5]]
print([item for group in groups for item in group])Flatten one level with a comprehension or chain, flatten many levels with recursion, and always treat strings as leaves rather than containers.
One more technique is worth mentioning for the one-level case. The built-in sum() cannot flatten lists, but itertools.chain() with several iterables as separate arguments joins them into one sequence, which is handy when the groups are known individually rather than collected in a list. Both chain() and chain.from_iterable() return lazy iterators, so either choice keeps the flattening lazy, and the decision between them is simply whether your groups arrive as separate arguments or as one iterable of groups.
Check: Why must a recursive flattening function treat strings specially, rather than flattening all iterables uniformly?
Section 28
Chunking is the opposite of flattening: a flat sequence is divided into groups of a fixed size. The groups might be batches of ten records for efficient processing, or pages of twenty items for display in a user interface that shows one page at a time. The task appears wherever data is handled better in pieces than all at once, whether the reason is efficiency, such as sending records to a service in batches rather than one by one, or usability, such as letting a reader browse through pages instead of facing an overwhelming single list.
Chunking a sequence that supports indexing, such as a list, is straightforward. Slicing it in steps of the chunk size produces a list of chunks, each containing up to the requested number of items, in order from first to last. The slicing approach is concise and easy to read once slicing syntax is familiar, and it works well whenever the whole sequence is already in memory as a list or another indexable type. Its cost is that it assumes the data can be indexed and that its length is known.
Chunking an iterator, which supports neither indexing nor length, needs a different approach. The standard tool is itertools.islice(), called repeatedly to take one chunk at a time until the iterator is exhausted. Each call yields up to the chunk size items, and the loop ends when a call returns fewer items than requested, or none at all. This lazy method never materialises the whole sequence, which makes it essential for large or infinite inputs, where building a complete list first would be wasteful or impossible. It also fits naturally into pipelines, where each chunk can be processed as soon as it is produced.
Testing chunking must cover the edge cases where it most often fails. A sequence whose length is an exact multiple of the chunk size should produce only full chunks, with no extra empty chunk at the end. A sequence whose length is not a multiple should produce a final partial chunk containing the leftover items, neither dropped nor padded with filler values. And an empty sequence should produce no chunks at all, not a single empty chunk that downstream code would process pointlessly. These cases are small, but getting them wrong corrupts results silently.
def chunks(items, size):
for i in range(0, len(items), size):
yield items[i:i + size]
print(list(chunks([1, 2, 3, 4, 5], 2)))Use slicing for indexable data, repeated islice() for iterators, and test the final partial chunk and the empty input every time.
An alternative lazy chunking method uses zip() with repeated references to the same iterator, a classic recipe that groups items by pulling several at a time from one shared iterator. It is clever, but it is also easy to misread, and the islice() loop is usually clearer for teaching and maintenance. Clarity matters here because chunking code tends to live in batch-processing programs where a subtle bug can silently drop records.
Check: Why does chunking an iterator require a different approach from chunking an indexable list, and what edge case affects the final chunk?
Section 29
Generator cleanup matters because generators can hold resources that need releasing when a consumer stops early. Consider a generator that opens a file and yields its lines, or one that acquires a lock before generating items. If the consumer finds what it wants after a few items and stops iterating, the generator is left suspended, still holding its resources. Left alone, those resources would be released only when the generator is garbage collected, at an unpredictable time, if at all during the program's run. For files, that can mean handles held open; for locks, it can mean other code blocked waiting.
The close() method solves this by raising GeneratorExit inside the generator at its suspension point. That exception gives the generator a chance to clean up before it ends. The usual pattern is a try/finally block wrapping the generation logic: the finally block runs whether generation ended naturally through exhaustion, or early through close(), or even through an exception during processing. This guarantee mirrors context manager cleanup, and it is the reason generators that manage resources should always wrap their work in try/finally, with the release code in the finally block rather than after the loop.
There is one strict rule for cleanup code. A generator that is being closed must not yield further items. If code in a finally block, or in a handler for GeneratorExit, attempts to yield after the close has begun, Python raises a RuntimeError, signalling a bug in the cleanup logic. The reasoning is simple: the consumer has explicitly said it wants no more items, so producing more would contradict that decision and confuse any code that assumes a closed generator is finished. Cleanup should release resources and then let the generator end quietly.
Testing cleanup should verify both paths. Check that resources are released when the generator is closed early, not only when it runs to exhaustion, since early stopping is the case where cleanup is most likely to be forgotten. Check that a closed generator raises StopIteration on later next() calls, rather than yielding further items or raising a different exception. And remember that garbage collection will eventually close an abandoned generator, but its timing is not guaranteed, so production code should close generators explicitly instead of relying on collection.
def managed():
try:
yield 1
yield 2
finally:
print("cleaned up")
gen = managed()
print(next(gen))
gen.close()Close generators you abandon, guard their resources with try/finally, and never yield during cleanup.
A related cleanup concern is that a generator holding a file should preferably be written with the file opened inside a with block within the generator itself, so that closing the generator also exits the with block and closes the file. This pattern combines two cleanup mechanisms: the with statement guarantees the file closes when the generator ends, and close() guarantees the generator ends promptly when the consumer stops early. Together they make resource handling robust.
Check: How does close() give a generator a chance to release resources when a consumer stops iterating early?
Section 30
The first thirty sections of this chapter have built a complete toolkit for iteration in Python. They covered the mechanics of the iteration protocol, the distinction between reusable iterables and single-use iterators, the role of StopIteration, custom iterator classes, generator functions and their state, advanced generator methods, generator expressions, list, dictionary, and set comprehensions, yield from, infinite generators, generator pipelines, nested comprehensions, comprehension scope, conditional expressions, lazy versus eager evaluation, the itertools module, custom iterable classes, mutation during iteration, iterator invalidation, enumerate() and zip(), sorting with keys, map() and filter(), aggregation functions, flattening, chunking, and generator cleanup. That is a wide range of tools, but they all rest on the same simple protocol.
Three principles recur across all of those topics, and they are worth naming explicitly, because they transfer to iteration tasks you have not yet met. Laziness means producing items on demand rather than all at once, which keeps memory use low and makes infinite sequences possible. Reusability means knowing whether an object can be iterated many times, like a list, or only once, like a generator, and designing and documenting that behaviour deliberately. Protocol consistency means making sure related operations agree with each other: iteration should agree with membership and with length for collections that support those operations, so that generic code behaves sensibly in every context.
These principles also give you a way to judge new iteration designs. When you meet an unfamiliar iterable, or design one yourself, ask whether it produces items lazily when laziness matters, whether it behaves reusably or single-use exactly as documented, and whether its related methods agree with each other. A design that answers all three questions cleanly will usually behave well in loops, comprehensions, unpacking, and built-in functions alike, because all of those constructs rely on the same protocol. A design that answers them inconsistently will surprise its users in exactly the contexts its author did not anticipate.
The remaining sections of this chapter build on these foundations with harder cases: reversed and sorted iteration, lazy file processing, dictionary views, unpacking, sentinel iteration, iterator patterns for linked and hierarchical structures, performance measurement, debugging, common pitfalls, asynchronous iteration at a high level, and a final review. None of these topics requires revisiting the basic mechanics in detail, because the protocol and the three principles already explain how each one works.
class Book:
def __init__(self, title):
self.title = title
print(Book("Dune").title)Keep laziness, reusability, and protocol consistency in mind as you work through the rest of the chapter, and test each new pattern against them before trusting it in real code.
It is worth pausing on why these principles were stated as a trio rather than as a longer list. Laziness governs when work happens, reusability governs how often a result can be consumed, and protocol consistency governs whether different operations on the same object agree. Between them they cover the questions that actually arise when designing iteration, and adding more principles would mostly restate these three in narrower terms.
Check: How do the principles from the first thirty sections help you design iteration behaviour for new data-processing tasks?
Section 31
Two built-ins change the order in which items are consumed, without changing the underlying collection: reversed() and sorted(). They look similar, since both take an iterable and return items in a different order, but their laziness differs fundamentally, and that difference decides when each one can be used. Understanding it will save you from trying to reverse a generator, or sort an infinite sequence, and wondering why the program fails or never finishes.
reversed() returns an iterator that yields items in reverse order, and for sequences it can do so lazily. A list, tuple, or string supports reverse iteration either through a __reversed__ method or through the sequence protocol of length and indexing, which Python uses as a fallback to walk from the last index down to the first. No copy is made and no items are precomputed; each item is produced when requested, in reverse order. That laziness is possible because a sequence already knows all its items and can address them by position.
Generators and other arbitrary iterables cannot be reversed this way, because their items are produced on demand and cannot be accessed backwards. There is no last item to start from until the whole sequence has been produced. Reversing such an iterable therefore requires materialising it into a list first, and then reversing that list. Attempting reversed() directly on a generator raises an error, which is Python telling you plainly that reverse iteration needs random access to items that do not yet exist.
sorted() is different again, because it must always materialise its input. Sorting requires seeing every item before any of them can be emitted in sorted order, since an item seen late could belong at the very front. No lazy strategy can avoid that: the smallest item cannot be identified until all items have been examined. Consequently sorted() builds a new list, sorts it, and returns the list, and it can never be used lazily over an infinite iterable, because collection would never finish. The two functions can be combined by sorting first into a list and then iterating that list in reverse, which costs no extra materialisation, since the list already exists.
print(list(reversed([1, 2, 3])))
print(sorted([3, 1, 2]))
print(sorted([3, 1, 2], reverse=True))Reverse a sequence lazily with reversed(), sort eagerly with sorted(), and materialise first whenever the input is not a sequence.
A related function worth knowing is sorted() with the reverse=True argument, which sorts in descending order directly, rather than sorting ascending and then reversing. The result is identical to reversing a sorted list, but stating the direction in the sort call is clearer about intent. For descending order, prefer the argument; reserve an explicit reversed() pass for cases where you genuinely need to walk an already-sorted list backwards.
Check: Why can reversed() be lazy for sequences, while sorted() must always materialise its input first?
Section 32
Iterating over a file line by line is one of the most important practical applications of iteration in Python. Files can be very large, with millions of lines, and loading such a file into memory as a list of lines before processing begins would be wasteful at best and impossible at worst. File objects are themselves iterables: iterating over an open file in a for loop yields its lines one at a time, reading from disk only as each line is requested. A program can therefore process a file of any size with roughly constant memory, no matter how many lines the file contains.
That laziness combines naturally with generator pipelines. Each stage of a pipeline can process lines as they are read, filtering out unwanted lines, transforming the rest into structured records, and aggregating those records into summaries or counts, without ever storing intermediate collections of lines or records. The whole program becomes a chain of small stages, each pulling from the previous one, and memory use stays flat because only one line, and one record derived from it, exists at a time. This is the standard way to process logs, data exports, and other large text files in Python, and it scales to files far larger than available memory.
File iteration should always happen inside a with statement. The with block guarantees that the file is closed when the block exits, whether iteration finished normally, stopped early with a break, or was interrupted by an exception during processing of a line. A file iterator abandoned without closing can hold its file handle open until garbage collection, at an unpredictable time, and on systems with limits on open handles per process, leaked handles can eventually cause new files to fail to open. The with statement removes that risk with a single, readable construct, and there is no good reason to iterate a file outside one.
Testing file iteration should use small files with known contents, including the edge cases where line handling most often goes wrong. An empty file should yield no lines at all. A file with a single line should yield exactly that line. A file whose last line lacks a trailing newline must still yield that last line, since a missing newline is not an end-of-file marker. And blank lines interspersed among other lines must be yielded as empty lines, not mistaken for the end of the file. Checking all lines in order, including the final one with or without its newline, catches most file-reading bugs before they reach real data.
with open("notes.txt", "w") as f:
f.write("one\ntwo\n")
with open("notes.txt") as f:
print([line.strip() for line in f])Iterate files lazily, inside a with block, and test the last line and the blank lines, not just the happy path.
Check: Why should file iteration happen inside a with statement, and what edge cases should file-iteration tests cover?
Section 33
The methods keys(), values(), and items() return dictionary views, and a view is not a snapshot. It is a live, iterable window onto the dictionary's current contents. If you create a view and then modify the dictionary before iterating over the view, the view reflects those modifications when it is eventually consumed. Adding a key after creating a keys view means the new key appears in the view; removing one means it disappears. The view holds a reference to the dictionary, not a copy of its contents at creation time.
That dynamic behaviour is useful when a program needs a live picture of a dictionary without copying it. Copying keys into a list creates a snapshot that goes stale the moment the dictionary changes, and keeping such a copy in sync requires extra code. A view stays in sync automatically, at no copying cost, which is the right choice when the consumer should always see the latest contents. The trade-off is that the consumer must accept that the contents may change between creating the view and iterating it, and must be written to tolerate that.
Views are subject to the same invalidation rules as direct dictionary iteration. Changing a dictionary's size during iteration over one of its views raises a RuntimeError, just as changing it during direct iteration does. The failure is loud, which prevents silent incorrect results, but it still breaks the program at an unexpected point. The remedies are the same as for direct iteration: iterate over a copy such as list(d.keys()) while mutating the dictionary, or build a new dictionary with a comprehension containing only the entries you want to keep.
Views also support operations beyond iteration. Membership testing with in on a keys view or an items view uses the dictionary's hash table, making it far faster than converting the view to a list and searching linearly. Keys views and items views additionally support set operations such as intersection and union, treating the dictionary's keys or items as a set for comparison purposes. These operations are another reason to keep a view rather than converting it, when the program's needs fit what views provide.
The choice between a view and a materialised list therefore comes down to whether dynamic reflection or a static snapshot is correct for the use at hand. Choose a view when the consumer should track the dictionary's current contents, and choose a list copy when the consumer needs the contents frozen at a specific moment, immune to later changes. Making that choice deliberately, and documenting it where it matters, prevents a whole class of bugs in which code reads either stale data or data that changed under it.
data = {"a": 1, "b": 2}
keys = data.keys()
data["c"] = 3
print(list(keys))A view follows the dictionary as it changes, while a list copy freezes one moment in time.
Check: How does a dictionary view differ from a list copy of its keys, when the dictionary changes after the view is created?
Section 34
Unpacking iterables through assignment is an iteration operation in disguise. When you write a, b = [1, 2], Python iterates over the right-hand iterable to obtain values for each target on the left, in order. The mechanism works for any iterable on the right, including lists, tuples, strings, generators, and custom iterables, provided the iterable produces exactly as many items as there are targets. If it produces too few or too many, unpacking fails with a ValueError. That loud failure is a feature: it prevents silent mismatches between targets and values in code that relies on unpacking to assign several variables at once from a single source.
Starred unpacking extends the mechanism to iterables of variable length. Writing first, *rest = [1, 2, 3, 4] assigns 1 to first and the list [2, 3, 4] to rest, collecting all remaining items. The starred target may appear in any position: at the beginning, in the middle, or at the end of the target list, and the items are divided accordingly. This is useful for handling the first item of a sequence specially while collecting the rest for further processing, or for splitting off a trailing item while keeping everything before it. It turns a pattern that would need slicing and indexing into a single readable assignment.
Only one starred target is allowed per assignment, and the reason is ambiguity. With two starred targets, Python could not decide how to distribute items between them, since any split of the middle items would satisfy both targets. A single starred target removes the ambiguity: every non-starred target takes exactly one item, and everything left over, however many items that may be, goes to the starred target. If the iterable contains exactly as many items as there are non-starred targets, the starred target simply receives an empty list, which is a valid and often useful result.
Unpacking also appears in function arguments, in both directions. In a definition, *args collects extra positional arguments into a tuple and **kwargs collects extra keyword arguments into a dictionary, letting a function accept a variable number of arguments. At a call site, f(*items) spreads an iterable into positional arguments and f(**mapping) spreads a mapping into keyword arguments. The two directions are inverses: one gathers arguments into a collection inside the function, and the other scatters a collection into arguments at the call.
first, *rest = [1, 2, 3, 4]
print(first, rest)
a, b = "xy"
print(a, b)Test unpacking for exact matches, for mismatches that should raise ValueError, and for starred targets receiving an empty remainder.
Unpacking also works with other assignment targets, including indexed positions and attributes, so a statement like point[0], point[1] = pair assigns into an existing structure rather than creating new names. The same iteration mechanism applies: the right-hand side is consumed in order, and a mismatch in count raises ValueError just as with plain names. This flexibility makes unpacking useful for swapping values and for splitting structured data without temporary variables.
Check: How does starred unpacking handle iterables whose length is not known in advance, and why is only one starred target allowed?
Section 35
Sometimes a for loop over an iterable does not directly express the termination condition for a task. Reading input from a user until they enter a specific value, such as an empty line or the word "quit", or draining a queue until it is empty, are cases where the stopping rule is a value to watch for, not the exhaustion of a collection. A while loop with an explicit condition handles these cases well, because the condition can be checked on each pass and the loop can stop as soon as the sentinel appears, without needing an iterable that represents the whole input in advance.
Python also provides a built-in two-argument form of iter() for exactly this pattern. Calling iter(callable, sentinel) creates an iterator from a callable and a sentinel value. The iterator repeatedly calls the callable with no arguments and yields each return value, until the callable returns the sentinel, at which point iteration stops cleanly, without yielding the sentinel itself. This turns any repeated-call source, such as reading a chunk from a file or receiving a message from a queue, into an iterable that can be used in a for loop or a comprehension, with no explicit while loop and no manual termination checking in the loop body.
The two-argument form is less well known than the one-argument form, but it is equally useful, and it often produces clearer code for sentinel-based patterns. Compare a while loop that reads, checks, and breaks, with a for loop over iter(read_chunk, b""): the second version states the termination rule once, in the iterator's construction, and the loop body contains only the processing logic. That separation makes the code easier to read and harder to get wrong, since the stop condition cannot be accidentally skipped by a misplaced break or a forgotten check.
Choosing between an explicit while loop and a sentinel iterator should be based on readability for the specific case. An explicit loop is clearer when termination involves complex logic beyond a simple sentinel comparison, such as checking several conditions or doing extra work before deciding whether to continue. A sentinel iterator is clearer when the termination rule is exactly a sentinel comparison for a callable that produces values repeatedly. Both forms need testing for the immediate-sentinel case, where the callable returns the sentinel on its very first call: the result should be an empty iteration, with zero items, not an error and not a single sentinel item passed through as data.
values = iter([1, 2, 0, 3])
result = []
for v in values:
if v == 0:
break
result.append(v)
print(result)When the stop rule is a single sentinel value, let iter() with two arguments express it, and keep the while loop for richer conditions.
Check: How does two-argument iter(callable, sentinel) turn repeated calls into an iterable, and when does it stop?
Section 36
Designing iteration for linked lists, trees, and graphs requires deciding what order items should be produced in, because these structures do not store their items in a flat array with simple index-based access. Items are connected through links or references, and the iteration order is a design decision that should be made deliberately, based on how the structure's items will be used, and documented clearly so users know what order to expect. A structure whose iteration order is undocumented will surprise its users, since different orders are all plausible and programs may depend on the order without realising it.
For a linked list, iteration naturally follows links from head to tail, yielding each node's value in turn until a node with no next link is reached. This is a simple pattern, and a generator method expresses it directly: walk a pointer from node to node, yielding each value, until the pointer becomes None. No index tracking or separate iterator class is needed, and the order is unambiguous, since a linked list has exactly one natural traversal. Any program iterating a linked list expects head-to-tail order, and documenting that expectation costs a single sentence.
Trees offer more than one natural order, and the choice matters. Depth-first iteration yields a node's value before its children's values, following each branch down before moving to the next, in a recursive pattern. Breadth-first iteration yields nodes level by level, from the top of the tree downwards, so all nodes at one depth appear before any node at a greater depth. These orders serve different purposes: depth-first may find a deeply nested value sooner, while breadth-first finds shallow values first and is natural for level-by-level processing. A tree class should therefore document which order it uses, or offer both, rather than leaving callers to guess.
Graphs are more complex still, because they may contain cycles. Following links naively around a cycle would revisit the same nodes forever, never reaching a termination point. Graph iteration therefore requires tracking visited nodes, typically in a set, and skipping any node already visited during the current iteration. That tracking belongs inside the iterator's implementation, not in caller code, since every iteration site would otherwise need to repeat it, and any site that forgot would loop forever. Linked-list iteration needs no such tracking, because a list has no branches and no cycles: each node has at most one successor, and following links can only end at the tail.
class Node:
def __init__(self, value, next_node=None):
self.value = value
self.next = next_node
def __iter__(self):
node = self
while node is not None:
yield node.value
node = node.next
print(list(Node(1, Node(2, Node(3)))))Define the order deliberately, document it, and hide cycle protection inside the iterator where cycles are possible.
Check: Why does graph iteration need visited-node tracking, while linked-list iteration does not?
Section 37
Performance choices between lazy and eager approaches involve trade-offs among memory use, computation time, and reusability, and they should be evaluated by measurement rather than by assumption. Generators carry overhead for suspending and resuming on each item produced, which can make them slower than list comprehensions for small collections, where that per-item overhead dominates total processing time. Lists carry memory overhead for storing every item at once, which can make them slower, or infeasible, for large collections, where allocation and cache effects dominate instead. The crossover point between the two depends on collection size, item complexity, and machine characteristics such as memory speed and cache size, so no universal rule can predict it.
The measurement-based approach uses Python's timeit module, or a similar timing tool, to compare approaches on realistic data for the program at hand. Realistic data matters, because synthetic micro-benchmarks may not reflect actual usage patterns: item sizes, access patterns, and data distributions all affect the result. Measurements should cover both time and memory, since an approach that is faster but uses far more memory may be unsuitable on a machine with limited memory available. They should also account for reusability, because an approach that wins on a single pass may lose overall if the data must be processed several times and regenerated on every pass, rather than reused from a list built once.
In practice, for most ordinary programs, the difference between lazy and eager approaches is negligible compared with other performance factors, such as algorithmic complexity or time spent waiting on input and output. A program that reads from disk or a network will spend most of its time waiting, not iterating, and choosing the wrong algorithm will cost far more than choosing the slightly slower iteration style. The choice between lazy and eager should therefore be based primarily on correctness and clarity: does the code need laziness for memory or for infinite sequences, does it need reuse or random access, and which form reads most clearly for this task.
Optimisation should be reserved for cases where measurement shows that iteration is actually a bottleneck on realistic data. Reaching for a generator, or a list, purely for speed, without evidence, is premature optimisation: it can make code harder to read and maintain while providing no measurable benefit to overall performance. When measurement does identify iteration as the bottleneck, the fix should be targeted at the measured problem, whether that is memory pressure calling for laziness, or repeated regeneration calling for materialisation, and verified by measuring again after the change.
import timeit
print(timeit.timeit("[x*x for x in range(100)]", number=1000))
print(timeit.timeit("list(x*x for x in range(100))", number=1000))Measure first on realistic data, choose for correctness and clarity, and optimise only the bottleneck the measurements reveal.
Check: Why should lazy-versus-eager performance choices be based on measurements with realistic data, rather than on assumptions?
Section 38
Debugging iterators is harder than debugging lists, for a reason that follows directly from their nature. Iterators are lazy and single-use, so inspecting an iterator's contents by converting it to a list consumes it, leaving nothing for the code that was supposed to iterate over it next. A program run under debugging therefore behaves differently from the same program run normally, which is a classic observer effect: the act of looking at the data changes the data's availability. In a pipeline of several generator stages, this can make a fault appear to move, or vanish, depending on where the inspection happened.
A practical technique for inspecting iterators without destroying them is sampling with itertools.islice(). Taking only the first few items lets you check that a generator produces the expected items in the expected order, without consuming its entire output. This is useful in interactive debugging, where a quick look at the first items often reveals a wrong transformation or a wrong order, and in tests, where small known inputs should be verified before larger realistic inputs are tried. Sampling does still consume the sampled items, so it is best used on a fresh generator created for inspection, or at the very start of processing, rather than in the middle of a live pipeline.
Another technique is to convert a generator pipeline into a list-based pipeline temporarily during debugging. Materialising the intermediate result at each stage, and inspecting those results directly, turns a pipeline problem into a bisection problem: check each stage's output in order until you find the first stage whose output differs from expectations, then fix that stage in isolation. The technique costs memory and defeats the purpose of laziness, but during debugging correctness matters more than memory efficiency, and test inputs are usually small. The materialisation must be removed before the code returns to production, or the program will carry the debugging cost permanently.
A third technique is temporary logging inside a generator, printing each item as it is yielded, so you can see exactly where a pipeline's output first diverges from expectations. Logging shows the live flow of items through stages without materialising anything, which makes it a good complement to sampling. Like materialisation, it must be removed once the fault is found, since logging left in production code would slow the pipeline and clutter its output for no benefit after debugging is complete.
import itertools
def gen():
yield 1
yield 2
yield 3
print(list(itertools.islice(gen(), 2)))Debug iterators by sampling fresh generators, bisecting pipelines stage by stage, and logging temporarily, never by consuming the live iterator to look at it.
A final debugging aid is worth mentioning: giving a generator a descriptive name and a docstring that states what it yields and in what order. When a pipeline misbehaves, clear stage names make log output and tracebacks far easier to interpret, because the faulty stage can be identified from its name alone. Good naming is not a substitute for sampling or bisection, but it makes both techniques faster by pointing the investigation at the right stage first.
Check: Why does inspecting an iterator by converting it to a list change a program's behaviour, and what sampling technique avoids that?
Section 39
Several pitfalls recur with iterators, generators, and comprehensions, and recognising them in advance will save you debugging time. The first is reusing an exhausted iterator. A second iteration over a consumed iterator yields nothing silently, rather than raising an error, so the mistake produces empty results instead of a visible failure. It is especially dangerous when iterators pass between functions, where one function consumes an iterator and another later expects to consume it again without realising it is already spent. The remedy is to document iterator consumption clearly at function boundaries, and to materialise iterators into lists whenever reuse across functions or multiple passes is needed.
The second pitfall is modifying a collection during iteration over it. List iteration fails silently by skipping items, because removals shift later items into positions the iterator has already passed. Dictionary iteration fails loudly with a RuntimeError for size changes during iteration, which at least prevents silent corruption but still breaks the program. These are different failure modes for the same underlying mistake. The remedy is the same in both cases: iterate over a copy, such as list(items), while mutating the original, or build a new collection with a comprehension containing only the items you want to keep, rather than deleting from the original during iteration.
The third pitfall is letting StopIteration escape from a generator function. Calling next() on an inner iterator without handling its exhaustion, inside a generator, now raises a RuntimeError instead of silently ending the generator early with incomplete results. The RuntimeError is a helpful signal, not a nuisance: it marks the exact place where inner exhaustion was mishandled. The fix is to catch StopIteration explicitly, or better, to iterate the inner iterator with a for loop, which handles exhaustion automatically and removes the need for manual next() calls altogether.
A fourth pitfall, worth naming alongside the three above, is writing a comprehension too complex to read, with multiple clauses and conditions intertwined so that no reader can tell at a glance what it produces. The remedy is to rewrite such comprehensions as explicit loops, with named variables and clear control flow, accepting more lines in exchange for readability. Code is maintained far longer than it takes to write, and a comprehension that cannot be read quickly will cost more in future debugging than the loop would have cost in lines.
it = iter([1, 2])
print(list(it))
print(list(it))The second print shows the first pitfall in miniature: an exhausted iterator silently yields nothing, exactly when a reader might expect it to repeat.
A fifth pitfall, closely related to the fourth, is assuming that a comprehension's loop variable is available afterwards. In Python 3, comprehension variables live in their own scope and do not leak, so code that tries to use the loop variable after the comprehension will fail with a NameError unless the name was already defined outside. Assigning the comprehension's result to a named variable, and using that result afterwards, avoids the problem entirely.
Check: What are three common iterator pitfalls, and what remedy applies to each one?
Section 40
Sections thirty-one through forty extend the basic iteration toolkit from the first thirty sections into territory closer to real programs. They covered reversed and sorted iteration, lazy file iteration, dictionary views, unpacking, sentinel iteration, custom iterator patterns for linked and hierarchical structures, performance considerations, debugging techniques, and common pitfalls. Each topic builds on the same protocol and the same three principles introduced earlier, laziness, reusability, and protocol consistency, but applies them to cases where the right choice is less obvious than in the simple examples of the first half.
The extension is visible in the kinds of problems these sections solve. The first thirty sections taught you how iteration works and how to build basic iterables and generators. Sections thirty-one to forty teach you how iteration behaves in context: how it interacts with files that must be closed, dictionaries that may change, structures whose order must be designed, and programs whose performance must be measured rather than assumed. That shift, from mechanism to judgment, is what turns knowledge of the protocol into skill at using it. Knowing that a generator is single-use is a fact; recognising that a debugging print of a generator destroys the data it was meant to inspect is judgment built on that fact.
The same principles keep recurring, which is why they were named explicitly at the midpoint of the chapter. Laziness explains why file iteration and pipelines stay memory-efficient, and why sorting cannot be lazy. Reusability explains why a generator method makes a class reusable while an iterator class does not, and why tests must create fresh generators. Protocol consistency explains why related operations should agree, so that a class which supports iteration also behaves sensibly under unpacking, membership testing, and conversion to a list.
The remaining sections complete the toolkit. They introduce asynchronous iteration at a high level, so you can recognise it in library code without yet mastering it, and they close with a review of design principles that connect this chapter to classes, protocols, and data-processing pipelines elsewhere in the book. The practical task that follows will ask you to combine these ideas into a working pipeline, which is the real test of whether the principles have become habits.
class Book:
def __init__(self, title):
self.title = title
print(Book("Dune").title)Use the final sections to practise judgment, not just mechanism, and test each new pattern against laziness, reusability, and consistency before trusting it.
One theme running through these ten sections deserves emphasis: iteration behaviour is part of an object's public interface, not an implementation detail. Callers write loops, comprehensions, and unpacking statements against the behaviour they expect, and when that behaviour is inconsistent or undocumented, their code breaks in ways that look like their own fault. Treating iteration as interface, with promises worth stating and testing, is the habit that separates reliable library code from code that merely works for its author.
Check: How do the topics in sections 31–40 extend the basic iteration toolkit from the first thirty sections?
Section 41
Asynchronous iteration extends the iteration protocol to programs that wait for data from external sources, such as network connections or other operations that complete at unpredictable times rather than immediately when requested. Where synchronous iteration defines __iter__ and __next__, asynchronous iteration defines __aiter__ and __anext__, and where synchronous code uses a for loop, asynchronous code uses an async for loop. The concepts carry over directly: items arrive one at a time, the iterator is single-use, and exhaustion is signalled by an exception, StopAsyncIteration, which plays the same role as StopIteration for synchronous iterators.
You will meet asynchronous iteration in libraries that provide data streams, such as network clients or database drivers that yield results as they arrive over time, rather than all at once after a query completes. Recognising the pattern matters even before you can write it yourself, because library documentation and examples use async for freely, and a reader who has never seen it may mistake it for a variant of the ordinary loop rather than a parallel protocol with its own rules. The key recognition is that the shape is familiar, laziness and exhaustion work the same way, but the execution model is different, since each step may involve waiting.
Asynchronous iteration is introduced here only at a high level, deliberately. It is not required knowledge for this chapter's core topics, which focus on synchronous iteration with for loops, comprehensions, generators, and iterator classes. The full treatment belongs to a later chapter devoted to asynchronous programming, where the syntax, semantics, and use cases will be explained with examples showing how asynchronous iterators are defined and consumed in real programs. Trying to master both models at once would blur the distinction between them, and the synchronous model deserves to be solid first, since the asynchronous model is best understood as its counterpart.
For now, the practical guidance is simple. Do not attempt to use asynchronous iteration in ordinary synchronous programs without first understanding asynchronous programming in general, because mixing the two models incorrectly causes errors that are hard to diagnose without understanding both. Master synchronous iteration thoroughly, as covered in this chapter, and then learn asynchronous iteration as an extension of those concepts when you reach the later chapter. The investment will transfer: laziness, single-use semantics, and exhaustion signalling behave the same way in both worlds.
async def agen():
yield 1
yield 2
print("async iteration is covered in detail in a later chapter")When you see async for in library documentation, you will know it is the asynchronous counterpart of the loop you already understand, applied to data that arrives over time.
A concrete example helps make the recognition point stick. A database driver might offer an asynchronous cursor that yields rows as they arrive from the server, consumed with async for row in cursor. The structure mirrors an ordinary loop over rows, but each iteration may wait for network data. Recognising that shape lets you read such code correctly, even before you understand how asynchronous execution schedules the waiting, which is the subject of the later chapter.
Check: How does asynchronous iteration relate to synchronous iteration, and why is it only introduced at a high level here?
Section 42
Iterator algebra is a way of thinking about combining iterator tools into expressions that describe data transformations as compositions of simple operations, rather than as sequences of imperative steps inside a loop body. The tools involved are familiar ones: chain() joins sequences end to end, islice() takes a slice of an iterator, zip() pairs items from several iterables, enumerate() adds indices, filter() keeps items that satisfy a predicate, and map() transforms each item with a function. Each tool does one thing, with well-defined behaviour for its inputs and outputs, and a pipeline is built by feeding the output of one tool into the next.
A simple example makes the idea concrete. Suppose you need to take a sequence, keep only items satisfying a condition, transform the survivors with a function, and take only the first ten results. As a composition, that is islice(map(transform, filter(condition, data)), 10): a filter stage, a map stage, and a slice stage, in that order. No explicit loop is needed, and no manual filtering, transformation, or counting logic is intertwined in a loop body. The expression states the intent directly, once the individual tools are understood, and each stage can be reasoned about in isolation before considering how they compose.
This compositional view also helps with reasoning about laziness and efficiency. Every operation in the composition is lazy, producing items on demand rather than all at once, so the composed pipeline as a whole is lazy as well. Items flow through all stages one at a time, and no intermediate collections are materialised between stages. That property can be established compositionally: knowing that lazy operations composed together remain lazy means you do not need to analyse the entire pipeline at once to determine its memory behaviour. The same reasoning applies to single-use semantics, since a pipeline built from single-use iterators is itself single-use.
Composition is not always the clearest choice, and that judgment matters. A composition of many tools can become dense, with nested calls that must be read inside out, and a reader unfamiliar with the tools may find an explicit loop easier to follow. The composition is better only when a reader can still follow it without effort. A good test is to express a familiar loop both ways and compare them honestly: if the composed version is shorter but harder to parse, the loop wins, and the algebraic form should be reserved for pipelines simple enough to read at a glance.
import itertools
data = range(20)
result = list(itertools.islice(filter(lambda x: x % 2 == 0, map(lambda x: x * x, data)), 3))
print(result)Think of iterator tools as composable operations, and design pipelines by choosing and ordering small stages whose behaviour you already understand.
Check: How does thinking of iterator tools as composable operations help you design and reason about data pipelines?
Section 43
A log file pipeline shows how the concepts of this chapter combine in a real program. Imagine a large log file that must be processed line by line without loading it entirely into memory. The pipeline begins by iterating over the file lazily, line by line, inside a with statement that guarantees the file is closed when iteration completes or when an exception occurs. Later stages filter out lines that do not contain a specific error marker, parse the remaining lines into structured records with timestamps and messages, and aggregate those records into counts by error type for reporting. Each stage is a generator function or generator expression that takes an iterable as input and yields transformed items for the next stage to consume.
That staged structure makes the program easy to understand, because each stage has a single responsibility that can be described in one sentence: filter to error lines only, parse lines into records, count records by error type. A reader can grasp the whole program by reading the stage descriptions in order, without following data through a monolithic loop. The structure also makes each stage independently testable. A filtering stage can be tested with small known inputs and expected outputs, without needing a large log file, and a parsing stage can be tested on its own edge cases, including malformed lines that should be handled gracefully rather than crashing the pipeline.
The pipeline as a whole can be tested with a small test log file containing known lines: error lines, non-error lines, blank lines, and malformed lines. The expected output is known in advance, so any deviation points to a specific stage, and the stage tests narrow it further. That two-level testing, stages in isolation and the pipeline end to end, catches most faults early, including the subtle ones where a stage handles normal lines correctly but mishandles an unusual line that appears rarely in real logs.
Laziness is what makes the pipeline practical for very large files. Lines pass one at a time through all stages before the next line is read, so memory use stays roughly constant regardless of the file's size. The program is therefore feasible for logs of any size on machines with limited memory. Laziness also allows early stopping: if only the first few errors are needed for inspection, the consumer can stop after a few records without processing the entire file, a benefit eager processing could only provide with additional code to limit processing to a prefix of the file.
def error_lines(lines):
for line in lines:
if "ERROR" in line:
yield line.strip()
print(list(error_lines(["ok", "ERROR disk full", "ok"])))Build pipelines from single-responsibility lazy stages, test each stage alone and the whole pipeline together, and let laziness handle scale.
Check: How does a staged generator pipeline make a log-processing program easier to test and to run on very large files?
Section 44
A data cleaning pipeline shows how comprehensions and generators combine in a program that turns raw data into clean records. Raw data might come from a CSV file or a list of dictionaries representing records from an external system. Cleaning involves removing blank records, stripping whitespace from text fields, converting numeric fields from strings to numbers, and validating that required fields are present and have plausible values. Each cleaning step becomes a separate stage in a pipeline that processes records one at a time, without materialising intermediate collections of cleaned records between stages.
Within such a pipeline, comprehensions and generators each have their natural place. Comprehensions suit simple transformations that fit in a single expression, such as stripping whitespace from every string field in a record through a dictionary comprehension mapping each field name to its stripped value. The operation is uniform, needs no branching beyond the expression itself, and the comprehension states it directly. Generators suit stages with more complex logic, such as validation that checks several fields together, or parsing that must handle malformed records gracefully by logging a warning and skipping the bad record rather than failing the entire pipeline for one bad entry in a large data set.
Combining the two forms in one pipeline gives the best of both: conciseness for simple steps and clarity for complex ones. Forcing every stage into a comprehension would make complex stages unreadable, while writing every stage as a generator function would bury simple transformations in boilerplate. The choice per stage should follow the same readability rule used throughout this chapter: use a comprehension when the operation is simple enough to grasp at a glance, and a generator function when it needs statements, branching, or error handling that a single expression cannot express cleanly.
Testing a cleaning pipeline should cover each step in isolation, with records that exercise that step's logic, including blank fields, fields containing only whitespace, numeric fields with invalid values, and records missing required fields. Whole-pipeline tests should use realistic dirty data mixing clean and dirty records, verifying that the output contains only clean records, that dirty records are handled gracefully, and that originally clean records pass through unchanged. Keeping a few awkward dirty records permanently in the test data ensures every future change is tested against the same hard cases, rather than against clean data alone, which would hide regressions in cleaning logic.
def clean(records):
for r in records:
name = r["name"].strip()
if name:
yield {"name": name, "age": int(r["age"])}
print(list(clean([{"name": " Ada ", "age": "30"}, {"name": " ", "age": "5"}])))Use comprehensions for simple cleaning steps, generators for stages that need branching or graceful failure, and keep dirty records in your tests permanently.
Keep this pattern in mind whenever you design an iterator.
Check: When should a cleaning pipeline use a comprehension for a stage, and when should it use a generator function instead?
Section 45
Testing generators needs techniques that account for their laziness and single-use nature. A generator object cannot be inspected by converting it to a list and then reused for further testing in the same test case, because the conversion consumes it. A test that converts a generator to a list to check its contents, and then asserts something else about the same generator, will find the second assertion operating on an exhausted iterator. The failure can be confusing, since the first assertion passed and the generator looks correct, yet the second assertion reports empty results for no obvious reason.
The remedy is to create a fresh generator for each assertion, or each group of assertions, that needs to iterate over the generator's output. Calling the generator function again with the same arguments produces a new generator object with fresh state, and each assertion group then works from the beginning of the sequence. This pattern should become a habit for testing generator functions. Where a test would naturally reuse a list, it must instead recall the generator function, and structuring tests around small helper calls that return fresh generators makes that habit easy to follow.
Generator tests should also verify laziness itself. A generator should not compute items until they are requested, and that property can be tested directly. One method is to create a generator over an infinite input, take only its first few items with itertools.islice(), and verify that it returns quickly, without attempting to process the entire infinite input first. Another method is to use a side effect, such as appending to a list inside the generator function, to track how many items have been computed at each point during the test. Both methods distinguish a truly lazy generator from one that eagerly computes everything before yielding its first item.
Tests should also cover exhaustion behaviour and early termination. A fully consumed generator should raise StopIteration on later next() calls, or yield no items on later iteration attempts, and that behaviour should be asserted rather than assumed. Early termination deserves its own tests: when a consumer stops before exhaustion, the generator's cleanup logic, typically a try/finally block, should run correctly and release any resources the generator holds. Together, these tests cover the generator's whole lifecycle, from creation through iteration to exhaustion or early termination and cleanup.
def gen():
yield 1
yield 2
assert list(gen()) == [1, 2]
assert list(gen()) == [1, 2]
print("fresh generators pass")Fresh generators per assertion group, explicit laziness checks, and exhaustion and cleanup tests together give generator tests their rigour.
Another useful testing habit is to test a generator's behaviour when its input is itself a generator, rather than a list. Pipelines often feed generators into generators, and a stage that accidentally materialises its input, or iterates it twice, will pass list-based tests while failing on genuine generator inputs. Including at least one generator-input test per stage catches that class of bug, which is otherwise easy to miss until the pipeline meets real streaming data.
Check: Why must generator tests create a fresh generator for each assertion group, rather than reusing one generator object?
Section 46
Refactoring loops into comprehensions or generators is a common task when improving existing Python code for readability or efficiency, and it should be done carefully. The first step is to understand exactly what the original loop does, in terms of its transformation, filtering, and aggregation logic, before expressing any of it in a new form. Only then can you decide whether the logic is simple enough to be clear as a single comprehension expression, or whether it involves complexity that belongs in a generator function, where statements and branching can express it without sacrificing readability for compactness.
Refactoring must preserve behaviour exactly, including edge cases. Empty inputs should still produce empty outputs, rather than raising an error or producing a single empty item. Single-item inputs should behave identically, and every branch of the original loop's logic should be exercised by the refactored form. One case deserves special attention: a loop that breaks early upon finding a desired item. A comprehension cannot express early termination directly, because comprehensions always process their entire input. A loop with a break should therefore become a generator function with an explicit return or break for early termination, not a comprehension that would process the whole input unnecessarily, and would never terminate at all on an infinite input.
Verification should come from tests written before refactoring begins. Tests comparing the original loop's output with the refactored code's output, on a variety of inputs including empty inputs, single-item inputs, and inputs exercising every branch, establish a baseline that the refactored code must match. Writing those tests first prevents subtle behaviour changes from slipping through unnoticed until users encounter them in production. The tests also document what the original loop actually did, which is valuable when the original code is unclear and the refactoring is partly an exercise in understanding it.
Refactoring is also a good moment to simplify deliberately. If a loop contains a branch that never executes for realistic inputs, it can be removed, but only with a test proving it was dead, rather than carried into the new form out of habit. Unexamined preservation of dead logic is how refactored code accumulates clutter. A refactoring that both preserves behaviour and removes proven dead weight leaves the codebase genuinely better, not merely differently shaped.
squares = []
for x in range(5):
squares.append(x * x)
refactored = [x * x for x in range(5)]
assert squares == refactored
print("refactor preserves behaviour")Understand the loop first, test before changing it, keep early termination in a generator, and simplify only what tests prove is dead.
One more refactoring caution concerns side effects. A loop that prints, logs, or writes as it goes cannot be refactored into a comprehension without changing when those effects happen, because a comprehension's purpose is to build a collection, not to perform actions. Side-effecting loops should become generator functions whose callers decide when to consume them, preserving the original timing of effects while still gaining the clarity of a named, reusable operation.
Check: Why can a loop with early termination not always be refactored into a comprehension, and what should it become instead?
Section 47
Knowing when not to use generators is as important as knowing when to use them, because generators are not suitable for every iteration task. Their limitations should guide the choice of tool for each task, rather than a blanket preference for generators forcing them into jobs where a list or another collection would serve better. Recognising those limits is a sign of mature judgment about iteration tools, not a failure to appreciate generators.
The first limitation is single-use semantics. Generators cannot be iterated twice, so they are unsuitable for tasks requiring multiple passes over the same data without regenerating it on each pass. Regeneration may be expensive, or impossible if the source is gone, and a materialised list avoids the problem entirely. The second limitation is lack of random access. A generator yields items in order and forgets them, so it cannot support indexing by position or searching backwards through data already consumed. Tasks that need to revisit earlier items, or jump to a specific position, need a collection that retains its items.
The third limitation is unknown length. A generator does not know how many items it will produce until it has produced them all, so it cannot answer len() before exhaustion. Programs that need to display progress as a percentage of total items, or allocate resources based on the total count in advance, need a materialised collection with a known length. The fourth limitation is in-place modification. A generator yields items one at a time without providing a way to write modified values back into the underlying collection, so tasks that must update items during iteration need the collection itself, not a stream of its values.
Choosing a list over a generator is therefore correct whenever a task requires multiple passes, random access, known length, or in-place modification. That choice should be made deliberately from the task's requirements, not from habit or fashion. Often the best design is mixed: use a generator for the early streaming stages of a pipeline, where laziness saves memory, and materialise into a list at the point where random access or repeated passes become necessary. Each stage then uses the tool that fits its own requirements, and the program as a whole gets both efficiency and capability.
data = [3, 1, 2]
print(data[1], len(data))
data[1] = 9
print(data)Indexing, length, and in-place update in three lines show exactly what a generator cannot offer, and why lists remain essential.
A related limitation is that generators cannot be pickled or copied in the usual ways, since their suspended state is not a simple data structure. Programs that need to save progress, send work to another process, or snapshot a computation must use materialised collections instead. This is another reason pipelines often end with a list: the list is a portable, storable result, while the generator is a transient process for producing it.
Check: For which four kinds of task requirements is a list a better choice than a generator?
Section 48
Designing a reusable iterable class for a library, where other programmers will use it in their own programs, demands more attention to documentation, testing, and protocol completeness than a class used only within a single program by its own author. Library users will assume that a class claiming to be iterable supports all the operations built-in iterables support in generic contexts: looping, unpacking, conversion to lists, and use in comprehensions and built-in functions that consume iterables. A class that supports only some of those operations will surprise its users in exactly the contexts its author did not anticipate, and the surprise will surface as bug reports against the library.
Documentation for a library iterable should state four things clearly. First, whether it is reusable or single-use, so callers know whether they may iterate it twice. Second, what order items are yielded in, so callers can rely on the order or know that they cannot. Third, whether iteration reflects the class's current contents dynamically, like a dictionary view, or a snapshot taken when iteration starts, so callers know whether concurrent modification changes what they see. Fourth, whether the class supports additional operations such as length, membership, or indexing, so callers know what else they may assume. That documentation belongs in the class docstring, where users encounter it at the moment they first meet the class, not only in a separate guide they may never read.
Testing for a library iterable should be equally comprehensive. It should cover repeated iteration for reusable iterables, exhaustion behaviour for single-use iterators, interaction with built-in functions such as list(), sum(), and sorted(), and behaviour when the underlying collection changes during iteration. These tests belong in the library's test suite, so that future changes to the class cannot break its iteration behaviour without being caught before release. Library code is used in contexts its author cannot anticipate, and only a thorough suite can guard promises made to unknown callers.
One point deserves explicit documentation even when it feels obvious: what happens when the underlying data changes during iteration. Library users will test that case, whether by accident or by design, and a clear statement prevents both surprise and bug reports. Publishing these promises in the docstring, rather than leaving them implicit, turns iteration behaviour from an implementation detail into a contract that users can rely on and maintainers must preserve.
class Library:
def __init__(self, items):
self.items = list(items)
def __iter__(self):
return iter(self.items)
def __len__(self):
return len(self.items)
lib = Library([1, 2])
print(list(lib), len(lib))A library iterable's docstring is a contract: state reusability, order, snapshot behaviour, and supported operations, and test every promise it makes.
Protocol completeness also means considering the iteration-related dunder methods beyond __iter__ itself. A library class that supports iteration but not __len__ will frustrate callers who reasonably expect to measure it, while a class that defines __len__ inconsistently with iteration, reporting a count its iterator never yields, will break generic code in subtler ways. Either define the supporting methods consistently, or document their absence plainly, so callers can plan around it.
Check: What should library documentation for an iterable class state about reusability, order, and dynamic versus snapshot behaviour?
Section 49
Reviewing iteration design principles as a whole reveals how they connect to other chapters in this book. Iteration appears repeatedly as the mechanism by which built-in operations work generically with many kinds of objects, through a common interface defined by methods rather than by inheritance. Classes and objects supply the dunder methods, protocols describe the agreements those methods form, decorators can wrap iteration behaviour, context managers handle the cleanup that generators also need, and the standard library builds its tools, from itertools to file objects, on the same protocol. Seeing iteration as a system, rather than as isolated tricks, explains why Python feels consistent across built-in and custom types.
That perspective will be developed further in later chapters, where iteration is used to build larger systems whose clarity depends even more on accurate modelling of iteration behaviour between their parts. Data processing pipelines chain generators into lazy stages of transformation and filtering, and their correctness depends on each stage honouring the protocol: producing items on demand, signalling exhaustion cleanly, and cleaning up when stopped early. Custom data structures need iteration orders defined deliberately and documented clearly, as this chapter has shown for linked lists, trees, and graphs. In both cases, the skill of designing iteration thoughtfully transfers directly from the small examples here to systems of real size.
The honesty principle should guide all of these design decisions. It states that a class or generator should either implement iteration completely and consistently with its documented behaviour, or document clearly which parts of iteration it supports, which parts it does not, and why. Implicit partial support, without documentation, is the unacceptable option. A class that iterates but silently misbehaves under unpacking, or a generator that claims laziness while computing eagerly, violates the trust that generic code places in the protocol. Python's built-ins assume that any iterable will behave like the ones they already know, and partial implementations break that assumption in ways that surface far from their cause.
That cross-chapter perspective is the point of learning iteration as a system rather than as a set of tricks. Once you see that iteration, context management, and comparison all work through protocols, you can predict how a new built-in operation will interact with your classes, even before you have met that operation in detail. Protocols compose, and a class that honours one protocol honestly is likely to cooperate with operations built on others.
class Book:
def __init__(self, title):
self.title = title
print(Book("Dune").title)Design iteration honestly and completely, document what you do not support, and the rest of the language will cooperate with your classes.
The connection to decorators is worth a sentence as well. A decorator that wraps a generator function receives a generator object when it calls the wrapped function, and it can transform, filter, or log that generator's items before yielding them onward. This makes decorators a natural way to add behaviour to generators without modifying their code, and it shows how the protocol's simplicity enables composition at every level, from single functions to whole libraries.
Check: How do iteration principles connect this chapter to classes, protocols, and data-processing pipelines elsewhere in the book?
Section 50
This final section returns to the chapter's central idea: iteration is not merely a way to loop over lists, but a protocol that lets custom classes and generators participate in Python's built-in operations as naturally as built-in types do. That perspective transforms how you design data processing code. Instead of thinking about individual loops in isolation, you think about pipelines of lazy stages that transform and filter data as it flows through them, one item at a time, without materialising intermediate collections. Instead of asking only what a loop does, you ask what promises its iteration behaviour makes to every consumer that might use it.
Those promises can be reduced to four questions worth asking for every iteration design. First, does the task require lazy or eager evaluation, and does the design deliver the one it needs? Second, must its results be reusable, or is single-use acceptable, and is that behaviour documented? Third, is its iteration behaviour consistent with related operations, such as membership and length, so that generic code sees a coherent object? Fourth, how will its iteration behaviour be tested in its real, generic contexts, rather than only in isolation, where protocol violations often hide? A design that answers all four questions cleanly will usually behave well wherever it is used.
Principle-based design scales, while ad hoc decisions do not. A principle, once understood, applies to every new context without being revisited: ask the four questions, answer them honestly, and document the answers. An ad hoc decision must be revisited for every new context in which a class or generator is used, with additional cost and additional risk each time a new context reveals an iteration gap or inconsistency that principled design would have caught from the start. As a program evolves and its users request features depending on iteration behaviour that was not originally planned, a principled and documented design can be extended cleanly, while an ad hoc one must be reworked under pressure.
That is the lasting lesson of this chapter. The protocol is small, two methods for synchronous iteration, but its consequences reach every corner of Python programming, from a simple for loop to a library used by strangers. Design for the protocol, document your promises, test them in generic contexts, and your code will cooperate with the language instead of fighting it.
class Book:
def __init__(self, title):
self.title = title
print(Book("Dune").title)Ask the four questions, answer them honestly, document the answers, and let principles, not habit, decide how your code iterates.
A useful way to practise these questions is to apply them to code you have already written. Take a class or generator from an earlier project, ask the four questions about it, and write down the answers. Gaps you find, an undocumented single-use result, a length that disagrees with iteration, or a test that only ever ran in isolation, are exactly the issues this chapter exists to prevent, and fixing them in familiar code makes the principles concrete.
Check: What four questions should you ask when designing iteration behaviour, and why do principles scale better than ad hoc decisions?
Practical Task
Build a lazy log-processing pipeline that reads a text file line by line, without loading it all into memory. Write a generator function that yields stripped, non-blank lines from a file opened inside a with statement, a second stage that keeps only lines containing the word "ERROR", and a third stage that parses each kept line into a dictionary with a message field. Combine the stages so that a test file with blank lines, ordinary lines, and error lines produces only the error records, in order. Write tests, using plain assert statements, that check: the pipeline yields records lazily (taking only the first record from an infinite-style generator input returns quickly); iterating the same generator object twice gives an empty second result, proving it is single-use; a fresh generator gives the same result again; and a file without a trailing newline on its last line still yields that line. Also include a list comprehension that builds a list of messages from the records, and a dictionary comprehension that counts records by message.
Chapter Questions
- Why is a generator object single-use, while a class whose
__iter__is a generator method is reusable, and how does that difference affect testing? - Explain how
StopIterationis handled differently by aforloop, by a manualnext()call, and bynext()with a default value. - When should you choose a lazy generator pipeline over an eager list comprehension, and when is materialising into a list the better choice?
- Why does removing items from a list during iteration over that same list skip items silently, while changing a dictionary's size during iteration raises
RuntimeError? - A generator holds an open file and a consumer stops early. How does
close()and atry/finallyblock ensure the file is released promptly, and what happens if the generator yields afterGeneratorExit?