Chapter 36: Performance and Profiling Basics

Section 1

Performance, for a Python program, means how quickly it completes its work, how much memory it uses, and how responsively it behaves while running. Those three aspects are related but distinct, and a program can be fast in one and poor in another. A program that finishes quickly but uses enormous memory may fail on a machine with limited memory. A program that uses little memory but takes a very long time may frustrate users who wait for it. Understanding performance as a balance, rather than as speed alone, helps you judge whether a program is good enough for its purpose, instead of chasing speed at the expense of everything else.

Performance requirements depend on context. A script run once a day, with a generous time window, can afford to be slower than a web service that must answer thousands of requests promptly. A tool run interactively must feel responsive, even if its total work takes a while, because users notice delays directly. A batch job that processes data overnight cares mainly about total time and about finishing before its deadline. Stating a program's performance requirements, in terms of acceptable time, memory, and responsiveness, before optimising, gives you a target to aim for, and a way to know when to stop, instead of optimising indefinitely without a clear goal.

It is also important to distinguish perceived performance from measured performance. A program that shows progress, or that produces partial results early, may feel faster to a user than one that is silent until it finishes, even if both take the same total time. That perception matters for interactive programs, where user experience is part of performance, but it should not replace measurement, because a program that feels fast while actually being slow may still miss a deadline for a scheduled job, where no user is watching. Both perception and measurement matter, for different kinds of programs, and a good performance approach considers both, while grounding decisions in measurement where deadlines or resource limits are at stake.

Performance work, like other engineering work, involves trade-offs. Making a program faster may make it more complex, harder to read, or more memory-hungry, and those costs must be weighed against the benefit of speed. A program that is already fast enough should not be made more complex for a marginal speedup, because complexity introduces bugs and maintenance burden, as discussed in earlier chapters. A program that is too slow, on the other hand, may need a fundamental change, such as a better algorithm or a different data structure, rather than a series of small tweaks, because small tweaks cannot fix a fundamentally inefficient approach. Recognising which situation you are in, through measurement, is the first step in performance work.

Check: Why should performance be judged as a balance of time, memory, and responsiveness, not speed alone?

Section 2

The most important rule in performance work is to measure before optimising, because guessing where time is spent is often wrong, and optimising the wrong part wastes effort while leaving the real bottleneck untouched. A program may feel slow because of a particular operation that you suspect, but measurement may reveal that most time is actually spent elsewhere, such as in file reading, in a library call, or in waiting for a network, rather than in the code you suspected. That surprise is common, even for experienced developers, because programs interact with data, libraries, and systems in ways that are hard to predict from code alone, and only measurement reveals the true distribution of time.

Measuring performance starts with a baseline: a timed run of the program on realistic data, with a clear record of how long it took, how much memory it used, and what input it processed. That baseline serves as a comparison point for later changes, allowing you to judge whether an optimisation actually helped, by how much, and whether it introduced a regression elsewhere, such as increased memory use. Without a baseline, optimisation becomes guesswork, where a change may feel faster but actually be slower, or may improve one case while worsening another, without anyone noticing, because there is no reference to compare against. Recording baselines, with data and conditions, is therefore a basic performance discipline.

Measurement should be repeated, because a single timing can be affected by system load, by caching, by background processes, and by other factors outside the program's control, leading to misleading results if taken alone. Running a program several times, under similar conditions, and looking at typical times, rather than the best or worst single run, gives a more reliable picture, and it helps you distinguish a real improvement from random variation. That repetition, while it takes a little extra time, prevents false conclusions, such as claiming a speedup that was actually due to a warm cache on the second run, rather than to a code change, which is a common measurement trap for beginners.

Measurement tools range from simple timing, using a clock to record start and end times for a whole program or for a section, to profiling, which records how much time is spent in each function, as discussed later in this chapter. Simple timing is sufficient for many purposes, such as comparing two versions of a program, or checking whether a program meets a deadline, while profiling is needed when you need to know which parts of a program are slow, in order to focus optimisation effort. Choosing the right tool for the question, rather than using the most detailed tool for every question, saves time and keeps performance work proportionate to its purpose.

Check: Why is a single timing measurement unreliable for judging an optimisation?

Section 3

Timing a program can be done at several levels, from whole-program timing to timing of individual sections, and each level answers a different question. Whole-program timing, measured by running the program and recording how long it takes from start to finish, answers the most important question for users: how long must they wait? That timing, performed with realistic data and repeated several times, as discussed earlier, is the baseline against which optimisations are judged, and it should be the first measurement for any performance work, because it grounds all later, more detailed measurements in user-visible reality, rather than in internal details that may not matter to users.

Section timing, where a program records how long each major stage takes, such as reading input, processing data, and writing output, answers the next question: where does time go, at a coarse level? That coarse breakdown often reveals a surprising bottleneck, such as reading taking far longer than processing, which suggests a different optimisation, such as faster file handling or less data reading, than processing optimisation would. Section timing can be implemented simply, by recording clock times at stage boundaries and printing or logging the differences, without needing a specialised profiling tool, and it is a good first step before profiling, because it narrows the search to the slowest stage, saving profiling effort for where it matters most.

Function-level timing, through profiling, answers a finer question: which functions, within a slow stage, consume the most time, and how often are they called? A profiler records, for each function, how many times it was called and how much time was spent in it, including time spent in functions it called, allowing you to identify hot spots, which are functions that dominate total time, either because they are slow themselves, or because they are called very many times, even if each call is fast. That distinction, between slow functions and frequently called functions, matters for optimisation, because a slow function may need a better algorithm, while a frequently called function may need to be called less often, such as by caching its result or by restructuring a loop to avoid repeated calls.

Profiling tools in Python's standard library provide this function-level information without requiring changes to the program's code, by running the program under a profiler that records calls and times, and then reporting a summary sorted by total time or by call count. Using such a tool, for a program that has already been identified as slow through whole-program and section timing, is a efficient way to focus optimisation, because it replaces guessing with evidence about where time goes, at a useful level of detail. That evidence, combined with an understanding of the program's logic, guides optimisation to the functions where it will have the greatest impact, rather than to functions that are easy to optimise but that contribute little to total time.

Check: How does section timing help narrow down where a program spends its time?

Section 4

Benchmarking is the practice of measuring a program's performance under controlled conditions, with a standard workload, so that results can be compared fairly, across versions, across machines, or across implementations. A benchmark should use realistic, representative data, should be run repeatedly, as discussed for measurement generally, and should report not just a single time, but a summary, such as typical time and variation, so that readers can judge whether a difference is real or merely noise. That controlled, repeatable measurement is what distinguishes benchmarking from casual timing, where conditions may vary and results may not be comparable, even if they look precise, because a single number without context can mislead more than it informs.

Profiling and benchmarking serve different purposes, and confusing them can lead to poor performance decisions. Profiling tells you where time goes within a program, by breaking down total time by function or by stage, as discussed earlier, and it is used to find bottlenecks and to guide optimisation. Benchmarking tells you how fast a program is overall, for a given workload, and it is used to compare versions, to verify that a program meets a requirement, or to detect regressions, where a new version is slower than an old one. A program can be profiled without being benchmarked, and benchmarked without being profiled, but the most effective performance work uses both: benchmarking to measure overall speed and to detect changes, and profiling to explain why speed changed and where to focus optimisation next.

Benchmarks should be designed to be stable, meaning that they produce similar results when run under similar conditions, and sensitive, meaning that they detect real performance changes, without being swamped by noise from system load or from unrelated factors. Achieving stability requires controlling conditions, such as running benchmarks on a quiet machine, with no other heavy programs running, and repeating runs, as discussed earlier, to average out variation. Achieving sensitivity requires using a workload that is large enough, and that exercises the parts of a program that matter, so that a real optimisation produces a measurable difference, rather than being hidden by noise from a too-small workload, where timing variation dominates any real change. That design, of stable and sensitive benchmarks, takes effort, but it pays off in reliable performance comparisons.

Microbenchmarks, which measure a tiny operation, such as a single function call or a single loop iteration, are a specific kind of benchmark that should be used with caution, because they can be misleading if extrapolated to whole programs. A microbenchmark may show that one operation is faster than another, in isolation, but that difference may be irrelevant in a real program, where the operation is a tiny fraction of total time, or where its performance depends on context, such as data size or cache state, that a microbenchmark does not capture. Microbenchmarks are useful for comparing alternatives for a hot spot identified by profiling, where the operation's performance genuinely matters, but they should not be used to justify optimising a cold spot, where even a large relative speedup produces a negligible absolute improvement, because the operation contributes little to total time.

Check: How do profiling and benchmarking differ in purpose?

Section 5

Algorithmic complexity describes how a program's time and memory requirements grow as its input grows, and understanding it conceptually is essential for performance, because it predicts whether a program that is fast on small data will remain fast on large data. A program whose time grows roughly in proportion to its input size, such as by processing each item once, will scale well, because doubling the input roughly doubles the time, which is manageable. A program whose time grows much faster than its input, such as by comparing every item with every other item, will scale poorly, because doubling the input may quadruple the time, or worse, quickly making large inputs impractical, even if small inputs are fast. That scaling behaviour, more than absolute speed on a single input size, determines whether a program is suitable for large data.

Complexity is often described using Big O notation, which gives a simplified, upper-bound description of growth, ignoring constant factors and small inputs, to focus on the dominant trend as input grows large. For example, linear growth, where time is proportional to input size, is generally good, while quadratic growth, where time is proportional to the square of input size, is often problematic for large inputs, and logarithmic growth, where time grows very slowly with input size, such as in a well-designed search, is excellent. Those descriptions are qualitative guides, not precise predictions, because they ignore constants, such as how fast each operation is, and they ignore data characteristics, such as distribution, which can affect real performance, but they are valuable for comparing algorithms and for spotting scaling problems before they are measured, by reasoning about a program's structure.

Choosing an algorithm with better complexity is often the most effective optimisation, far more effective than tweaking code for a poor algorithm, because a better algorithm changes the growth trend, while tweaking only changes constant factors. For example, replacing a search that checks every item, with a search that uses an indexed structure, can change growth from linear to logarithmic, which for large data is a dramatic improvement, far larger than any code-level optimisation could achieve for the linear search. That is why performance work should start with algorithm and data structure choice, as discussed later, rather than with micro-optimisations, because the former addresses scaling, while the latter only addresses constants, which matter less as input grows, though they still matter for programs where input size is fixed and small.

Complexity analysis should be combined with measurement, not used as a replacement for it, because real programs have constants, overheads, and data characteristics that analysis alone cannot capture. An algorithm with better theoretical complexity may be slower in practice for small inputs, due to setup overhead, or for particular data distributions, due to worst-case behaviour that analysis assumes but that real data may not trigger, or may trigger unexpectedly. Measuring both algorithms on realistic data, with realistic sizes, as discussed for benchmarking, confirms whether a theoretical improvement translates into a practical one, for the program's actual use, rather than assuming it will, based on complexity alone. That combination, of analysis to guide choice and measurement to verify it, is the sound approach to algorithmic performance.

Check: Why is choosing a better algorithm often more effective than tweaking code for a poor algorithm?

Section 6

Common bottlenecks in Python programs fall into a few categories, and recognising them helps you focus optimisation where it matters. Input and output bottlenecks occur when a program spends most time reading or writing files, networks, or other external resources, rather than computing, because external operations are typically much slower than in-memory computation. A program that reads a file line by line, with many small reads, or that writes output in many small pieces, may be slowed by the overhead of each operation, and it can often be improved by reading or writing in larger chunks, by buffering, or by reducing the amount of data read or written, such as by reading only needed columns, rather than entire records. That I/O optimisation, focused on reducing operations and data volume, is often more effective than optimising computation, when I/O dominates, as section timing, discussed earlier, will reveal.

Loop bottlenecks occur when a program performs an expensive operation inside a loop that runs many times, such as searching a long list for each item, or performing a calculation that could be done once outside the loop, rather than repeatedly inside it. That pattern, of repeated work inside a loop, is a classic performance bug, because its cost multiplies by the number of iterations, turning a small inefficiency into a large total cost. Fixing it involves moving invariant work outside the loop, replacing a linear search with a faster lookup, such as a dictionary, or restructuring the loop to avoid repeated work, as discussed for data structures later. That restructuring, guided by profiling that shows a loop's body as a hot spot, is a straightforward optimisation with potentially large benefits, because it addresses work that is repeated unnecessarily.

Repeated work bottlenecks also occur across function calls, where a program recomputes the same result many times, for the same input, because it does not remember previous results. Caching, or memoisation, as discussed later, addresses that by storing results and reusing them, trading memory for time, which is worthwhile when computation is expensive and results are reused frequently, but not when results are rarely reused, or when memory is scarce, in which case caching adds overhead without benefit. Recognising repeated computation, through profiling that shows a function called many times with the same arguments, is the first step to deciding whether caching will help, and measuring its effect, as with any optimisation, confirms whether it actually does, for realistic data.

String and data structure bottlenecks are another category, where a program uses an inefficient way to build or manipulate data, such as building a large string by repeated concatenation in a loop, which can be slow because each concatenation creates a new string, copying all previous content, leading to quadratic growth in time, as discussed for complexity. Using a list to collect parts and joining them once at the end, or using a data structure designed for the operation, such as a dictionary for lookups, rather than a list, can dramatically improve performance for such patterns, and those improvements are often simple to implement once the bottleneck is identified, because they involve changing how data is built or accessed, not changing the program's logic. That simplicity, combined with large potential benefit, makes data structure bottlenecks a high-value target for optimisation, once profiling identifies them.

Check: Why is repeated work inside a loop particularly costly for performance?

Section 7

Data structure choice has a large impact on performance, because different structures support different operations efficiently, and choosing the wrong structure for a program's main operations can make it slow, even if its algorithm is otherwise sound. A list, for example, is efficient for accessing items by position, and for appending at the end, but it is inefficient for checking whether an item is present, or for inserting at the beginning, because those operations may require scanning or shifting many items. A dictionary, by contrast, is efficient for checking presence and for looking up values by key, because it uses hashing to find items quickly, regardless of how many items it contains, in typical cases, but it is less efficient for ordered access, or for operations that require scanning all items in a particular order, depending on how it is used. Choosing between them, based on which operations a program performs most often, is a key performance decision.

Sets share dictionaries' efficiency for presence checking, because they also use hashing, and they are the right choice when a program needs to test membership frequently, or to perform set operations such as union or intersection, rather than storing associated values. Using a list for membership testing, when a set would do, is a common performance mistake, because membership testing in a list grows with list size, while in a set it remains roughly constant, in typical cases, which for large data is a dramatic difference, as discussed for complexity. That difference, between linear and constant-time membership, illustrates why data structure choice matters more than micro-optimisation for large data, because it changes scaling, not just constants.

Other structures, such as queues, stacks, and specialised structures for particular tasks, also have performance characteristics that suit specific patterns. A queue, implemented efficiently, allows fast addition at one end and removal at the other, which suits producer-consumer patterns, as discussed for concurrency, while a list used as a queue, with removal from the beginning, can be slow, because it shifts all remaining items on each removal. Recognising those pattern-structure matches, and using a structure designed for the pattern, rather than forcing a general structure to serve a specific pattern inefficiently, is a performance skill that grows with experience and with familiarity with the standard library's structures, which include efficient implementations for many common patterns.

Data structure choice also affects memory use, because different structures store data with different overheads, such as dictionaries using more memory per item than lists, due to hashing tables, or sets using more memory than lists for the same items, for similar reasons. That memory overhead may be acceptable when it buys a large time improvement, as for membership testing, but it may be problematic for very large datasets, where memory is scarce, and a more memory-efficient structure, even if slightly slower, may be necessary to avoid running out of memory. Balancing time against memory, as discussed for performance generally, is part of data structure choice, and it should be based on measurement, for realistic data sizes, rather than on a blanket preference for speed or for memory efficiency alone.

Check: Why is a set or dictionary much faster than a list for membership testing on large data?

Section 8

Caching, also called memoisation when applied to function results, is a technique for trading memory for time, by storing the result of an expensive operation and reusing it, rather than recomputing it, when the same operation is requested again with the same inputs. That trade is worthwhile when computation is expensive, when results are reused frequently, and when memory is sufficient to store cached results without running out, because in those conditions, caching can dramatically reduce total time, by avoiding repeated work, as discussed for repeated work bottlenecks. When those conditions do not hold, such as when results are rarely reused, or when memory is scarce, caching adds overhead for storing and looking up results, without benefit, and it may even slow a program down, or cause it to run out of memory, if cached results accumulate without limit.

Implementing caching requires deciding what to cache, how to key cached results, and when to invalidate or evict them, because a cache that grows without bound, or that returns stale results after inputs change, can cause problems worse than the repeated computation it was meant to avoid. Keying, which means identifying a cached result by its inputs, must be precise, so that different inputs do not collide to the same key, returning a wrong result, and so that the same inputs always map to the same key, allowing reuse. Invalidation, which means removing cached results when they are no longer valid, such as when underlying data changes, is essential for correctness, because a cache that returns outdated results, presented as current, misleads users and can cause incorrect decisions based on stale data, which is worse than slow but correct computation.

Cache size management is another consideration, because an unbounded cache, storing every result ever computed, will eventually consume all available memory, especially for programs that process many distinct inputs over a long run. Limiting cache size, such as by keeping only the most recently used results, or only results up to a certain count, and evicting older results when the limit is reached, keeps memory use bounded, while still providing benefit for frequently reused results, which are likely to be recent, in many workloads. That size limiting, with a sensible eviction policy, is a standard part of cache design, and it should be chosen based on the program's reuse patterns and memory constraints, rather than left unbounded by default, which is a common caching mistake.

Caching can be applied at several levels, from function-level memoisation, where a single function's results are cached, to data-level caching, where expensive data, such as a file's contents or a database query result, is cached, to system-level caching, where results are stored in a file or in a dedicated cache service, for reuse across runs. Each level has different trade-offs in terms of complexity, persistence, and invalidation, and the appropriate level depends on how expensive the operation is, how often its result is reused, and whether reuse should persist across runs, or only within a single run. Starting with the simplest level that meets the need, such as function-level caching within a run, and moving to more persistent levels only if measurement shows that cross-run reuse would help significantly, keeps caching proportionate to its benefit.

Check: When is caching not worthwhile despite avoiding repeated computation?

Section 9

Lazy evaluation means postponing work until its result is actually needed, rather than doing all work upfront, and it can improve performance, especially for programs that process large data, or that may not need all results. Generators, in Python, are a lazy evaluation mechanism, where values are produced one at a time, on demand, rather than all at once in a list, which saves memory, because only one value needs to be held at a time, and it can save time, because work stops as soon as enough results have been produced, rather than computing all results even if only a few are needed. That laziness is particularly valuable for pipelines, where data flows through several stages, each producing values on demand for the next stage, without materialising intermediate collections that would consume memory and time to build fully, even if later stages only need part of them.

Eager evaluation, by contrast, computes all results upfront, storing them in a collection, which is simpler to reason about, because the collection is complete and can be reused, indexed, and measured, but which can be wasteful, if only part of the collection is actually used, or if the collection is very large, consuming memory that could have been avoided by laziness. Choosing between lazy and eager evaluation involves balancing simplicity and reusability, which favour eager, against memory and early-stopping benefits, which favour lazy, and the right choice depends on how results will be used, whether they will be reused multiple times, whether they need random access, and whether their total size is large compared with available memory. For a program that processes a large file line by line, laziness is usually right, because it avoids loading the whole file, while for a program that needs to sort results, or to access them repeatedly in random order, eager evaluation, with a complete collection, may be necessary, because sorting and random access require all data to be available.

Lazy evaluation also interacts with performance measurement, because a lazy program's work is spread out over time, as values are requested, rather than concentrated upfront, which can make timing confusing if measurement only covers part of the program. For example, timing a function that returns a generator, without actually consuming its values, will measure only the tiny cost of creating the generator, not the cost of producing values, which occurs later, when the generator is iterated.

Another consideration for lazy evaluation is that it can make debugging harder, because errors in producing values occur when values are requested, not when the generator is created, which may be far from the code that created it, in both time and location, making it harder to trace an error to its source. That delayed error reporting, while it is a consequence of laziness, should be considered when choosing lazy evaluation, especially for programs where early error detection is important, such as when validating input, where an error in the first item should be reported immediately, rather than after much other work has been done. Balancing laziness against early error detection, by validating eagerly where it matters, while processing lazily where it helps, is a practical compromise that retains laziness's benefits while avoiding its debugging drawbacks.

Check: Why can timing a generator-creating function alone give a misleadingly low time?

Section 10

Premature optimisation is the practice of optimising code before knowing, through measurement, that it needs optimising, or before knowing where its bottlenecks are, and it is widely regarded as a mistake, because it adds complexity, reduces readability, and risks introducing bugs, for benefits that may be negligible or non-existent, if the optimised code was not actually a bottleneck. That mistake often arises from intuition, where a developer assumes that a particular operation must be slow, because it looks expensive, or because they have heard that it is slow, without measuring whether it actually contributes significantly to total time for realistic data. That intuition, while it may sometimes be correct, is often wrong, as discussed for measurement, because performance depends on data, on context, and on the program's overall structure, in ways that are hard to intuit reliably.

Premature optimisation also harms maintainability, because optimised code is often more complex than straightforward code, using tricks, such as manual loop unrolling, or clever data structures, that are harder to read, harder to test, and harder to change, than the simple code they replace. That complexity cost persists long after any performance benefit, if there was one, has been forgotten, and it makes future changes more error-prone, because a maintainer must understand the optimisation to change the code safely, rather than simply understanding its logic. That long-term cost, of complexity for uncertain benefit, is why premature optimisation is discouraged, even when it does produce a small speedup, because the speedup may not justify the complexity, especially if the program was already fast enough for its purpose, as discussed for performance requirements.

The antidote to premature optimisation is a disciplined workflow: write clear, correct code first, measure its performance on realistic data, identify bottlenecks through profiling, if it is too slow, optimise only those bottlenecks, with the simplest effective change, and measure again to confirm improvement, as discussed throughout this chapter. That workflow ensures that optimisation effort is spent where it matters, that its benefit is verified, and that its complexity is justified by a measured need, rather than by intuition. It also ensures that code remains clear and correct, except where optimisation is genuinely needed, in which case its complexity is documented and justified, so that future maintainers understand why it exists and when it can be simplified, if performance requirements change.

There are exceptions to the rule against premature optimisation, where a performance consideration should be designed in from the start, rather than added later, such as choosing an appropriate algorithm or data structure, as discussed earlier, because changing an algorithm later can be a major rewrite, while choosing a good one initially costs little extra effort, if the choice is based on an understanding of the program's scaling requirements. That design-level performance thinking, about algorithms, data structures, and overall architecture, is not premature optimisation; it is good design, because it addresses scaling and structure, which are hard to change later, while avoiding micro-optimisations, such as tweaking individual operations, which are easy to add later, if measurement shows they are needed, and which should not be done prematurely. Distinguishing between design-level performance, which should be considered early, and micro-optimisation, which should wait for measurement, is a key skill in performance work.

Check: How does design-level performance thinking differ from premature micro-optimisation?

Section 11

A profiler is a tool that records, while a program runs, which functions are called, how often, and how much time each takes, producing a report that shows where time goes in far more detail than whole-program or section timing can. Python's standard library includes a profiler that can be run on a program without changing its code, by invoking the program under the profiler, which then prints a summary sorted by total time, by call count, or by other criteria, allowing you to see at a glance which functions dominate. That summary is the starting point for optimisation, because it replaces guessing with evidence, showing not only slow functions, but also functions that are fast individually yet called so many times that their total cost is large, which is a different optimisation problem, requiring fewer calls rather than faster code.

Reading a profiler report requires understanding its columns, in general terms, without memorising every detail of a particular tool's output. The most important columns are usually the number of calls, the total time spent in a function including calls it made, and the time spent in the function itself excluding its callees, because those three together distinguish a function that is itself slow from one that is merely a caller of slow functions, and from one that is called excessively. A function with high total time but low self time is not itself the problem; its callees are, and optimising it directly would waste effort, while a function with high self time is a genuine hot spot, and a function with a very high call count, even with low total time, may indicate repeated work that could be reduced by caching or by restructuring a loop, as discussed earlier.

Profiling has overhead, because recording every call takes time and memory, so a profiled program will run slower than it would normally, and its timing proportions may differ slightly from an unprofiled run, especially for very short functions where profiling overhead is proportionately large. That overhead does not usually invalidate profiling results, because it affects all functions, and the relative ranking of hot spots generally remains similar, but it does mean that absolute times from a profiled run should not be used as benchmarks, for comparing versions, without also timing unprofiled runs, as discussed for benchmarking. Profiling is for diagnosis, to find where time goes, while benchmarking, with unprofiled timing, is for measurement, to judge overall speed, and keeping those roles separate prevents confusion between diagnostic and benchmark numbers.

Finally, profiling should be done on realistic data, as with all performance measurement, because a profile on small test data may show different hot spots from a profile on real data, if real data has different characteristics, such as larger values, more duplicates, or different distributions, that change which functions dominate. A profile that guides optimisation for test data, but not for real data, can lead to optimising the wrong functions, while real bottlenecks remain untouched, which is a waste of effort and a source of frustration when optimisation does not improve real-world performance as expected. That realistic profiling, combined with a clear understanding of profiler reports and their overhead, makes profiling a powerful, evidence-based tool for performance work, rather than a source of misleading numbers.

Check: How do total time and self time in a profiler report differ in what they reveal?

Section 12

Memory profiling is the counterpart to time profiling, measuring how much memory a program uses, where it is allocated, and whether it grows over time in ways that indicate a problem, such as a leak, where memory is allocated but never freed, even though it is no longer needed. Detecting such growth, through memory measurement over time, is the first step to fixing it, because a leak that is invisible in a short test run may become fatal in a long production run, where memory accumulates for hours or days.

Measuring memory can be done at several levels, from whole-program memory use, reported by the operating system, to detailed allocation tracking, which records which functions or lines allocate the most memory, similar to time profiling but for memory. Whole-program measurement is simple and sufficient for detecting growth over time, by recording memory use at intervals during a run, and checking whether it stabilises or keeps growing, while detailed allocation tracking is needed to find which part of a program is responsible for growth, when growth is detected, because whole-program numbers alone do not say where memory is being allocated, only that it is growing. That two-level approach, of coarse measurement to detect a problem and detailed measurement to locate it, mirrors time profiling, and it keeps memory work proportionate, by using detailed tools only when a problem is confirmed.

Common memory problems include holding references to data that is no longer needed, such as keeping every item processed in a list, when only a summary is needed, which causes memory to grow with input size, even though the program's output does not require all input to be retained. That pattern, of unnecessary retention, can often be fixed by processing data in a stream, as discussed for lazy evaluation, discarding each item after it is processed, rather than accumulating all items, which changes memory growth from proportional to input size to roughly constant, a dramatic improvement for large inputs, with no loss of functionality, if the program's logic does not actually need random access to all items. Recognising unnecessary retention, through memory measurement that shows growth with input, is a key memory optimisation skill, because it addresses a structural problem, not just a code tweak.

Another memory problem is creating large intermediate structures, such as building a huge list as an intermediate step, when a smaller structure, or a streaming approach, would suffice, which wastes both memory and time, because building the intermediate structure takes time as well as space. Replacing an intermediate list with a generator, or with a more compact structure, can reduce both memory and time, though it may make code slightly less straightforward, if later stages need random access to intermediate data that a generator cannot provide, in which case a trade-off must be made, as discussed for lazy versus eager evaluation. That trade-off, between memory saving and code simplicity or random access, should be decided based on measurement, for realistic data sizes, rather than on a blanket preference for streaming, because for small data, intermediate structures are harmless, and their simplicity may outweigh their memory cost.

Check: How does a true memory leak differ from normal memory growth that stabilises?

Section 13

Loop optimisation focuses on the code inside loops that run many times, because even a small inefficiency there multiplies by the number of iterations, becoming a large total cost, as discussed for bottlenecks generally. The first loop optimisation is to move invariant work outside the loop, meaning work whose result does not change between iterations, such as a calculation based only on values that are constant during the loop, or a lookup that returns the same result each time, should be done once before the loop, and its result stored in a variable, rather than recomputed on every iteration. That simple change, which is often obvious once a profiler shows a loop's body as a hot spot, can produce a large speedup with no change to the program's logic, because it eliminates repeated work that was never necessary, and it is a classic example of optimisation that is both simple and effective, when applied to a genuine hot spot identified by measurement.

Another loop optimisation is to reduce work inside the loop by using more efficient operations, such as replacing a linear search inside a loop with a dictionary lookup, as discussed for data structures, or by using a built-in operation that processes many items at once, rather than processing them one by one in Python code, because built-in operations are often implemented in a faster language, and they avoid the overhead of Python-level loop iterations for each item. That replacement, of a Python loop with a built-in or library operation, is a common and effective optimisation, but it should only be applied where profiling shows that the loop is actually a bottleneck, because replacing clear loop code with a clever built-in call can reduce readability, and if the loop was not a bottleneck, that readability loss buys no meaningful speedup, which is a poor trade, as discussed for premature optimisation.

Loop structure itself can also affect performance, such as by choosing the right kind of loop for the task, or by avoiding unnecessary work in loop conditions, such as calling a function to determine a loop's end on every iteration, when its result does not change, which is another form of invariant work that should be moved outside. Similarly, nested loops, where an inner loop runs for every iteration of an outer loop, deserve special attention, because their total work multiplies, and an inefficiency in an inner loop is magnified by the outer loop's iterations, making inner loops particularly important targets for optimisation, when profiling shows them as hot spots. That nested-loop magnification, where a small inner inefficiency becomes a large total cost, is why profiling often highlights inner loops, and why optimising them, even slightly, can have a large overall effect, if they are genuinely hot.

Check: Why should invariant work be moved outside a loop rather than recomputed each iteration?

Section 14

Avoiding repeated attribute lookups and repeated function calls, within hot code, is a micro-optimisation that can help, but only when applied to a genuine bottleneck identified by profiling, because its benefit is usually small compared with algorithmic or data structure changes, and its cost in readability can be significant if applied indiscriminately. The idea is simple: accessing an attribute, such as a method or a value on an object, or calling a function, involves work, such as looking up the attribute or setting up a call, and if that work is repeated many times in a loop, with the same result each time, storing the result in a local variable once, and reusing it, can save that repeated lookup work. That saving, multiplied by many iterations, can become noticeable for a hot loop, but it is typically a small percentage improvement, not a dramatic speedup, and it should not be the first optimisation attempted, when larger improvements, such as a better algorithm, are available.

A related micro-optimisation is to use local variables instead of global variables in hot code, because local variable access is generally faster than global access in Python, due to how the interpreter resolves names, and a loop that repeatedly accesses a global variable can be sped up, slightly, by assigning that global to a local variable before the loop, and using the local inside the loop. That change, like attribute lookup caching, is simple and low-risk, but its benefit is also small, and it should be justified by profiling, showing that the loop is hot and that name lookup contributes measurably to its time, rather than applied speculatively to all loops, where it adds clutter without benefit, and where it may confuse a reader who wonders why a global was copied to a local, without understanding that it was for performance, unless the change is documented.

Another micro-optimisation involves choosing efficient idioms, such as using a comprehension or a built-in function, rather than an explicit loop with appends, for building a collection, because comprehensions and built-ins are often faster, due to their implementation, and they are also more concise and readable, in many cases, making them a rare optimisation that improves both speed and clarity, when used appropriately. That dual benefit makes idiom choice a good first micro-optimisation, when a hot spot involves building a collection, because it does not sacrifice readability for speed, unlike many micro-optimisations, which do. However, even idiom choice should be verified by measurement, for realistic data, because a comprehension that is faster for large data may be similar in speed for small data, where simplicity, rather than speed, should guide choice, and where a straightforward loop may be clearer to a reader unfamiliar with comprehensions.

Check: Why should micro-optimisations only be applied to genuine bottlenecks, not everywhere?

Section 15

Using built-in functions and standard library operations, rather than writing equivalent Python code by hand, is often a performance win, because built-ins are implemented in a faster language, are well-tested, and are optimised for common operations, such as sorting, searching, summing, and transforming collections. A program that sorts a list by writing its own sorting loop in Python will almost certainly be slower than one that uses the built-in sort, which is not only faster, but also more correct, because it has been tested extensively and handles edge cases that a hand-written sort might miss. That combination, of speed and correctness, makes built-ins a preferred choice, whenever they fit a task, and it is a reason to learn the standard library well, so that you recognise when a built-in exists for a task, rather than reinventing it in slower, buggier Python code.

Built-ins also improve readability, in most cases, because a single built-in call, with a clear name, expresses an operation more concisely than a loop that implements it step by step, allowing a reader to understand intent at a glance, rather than tracing loop logic to infer what it does. That readability benefit, combined with speed, makes built-ins a rare optimisation that does not trade clarity for performance, unlike many optimisations, which do, and it is why experienced Python developers prefer built-ins and standard library functions, even when their own loop code would work, because the built-in is both faster and clearer, which is a win on both dimensions that performance work usually trades off against each other.

There are limits to built-in use, because a built-in that does not exactly match a task may require awkward workarounds, such as transforming data into a form the built-in expects, which can offset its speed benefit, or can make code less clear than a straightforward loop that directly expresses the task. In those cases, a hand-written loop, tailored to the task, may be clearer and, after accounting for transformation costs, may be no slower, or even faster, than a forced built-in use, and choosing the loop is then the better decision, despite the general preference for built-ins. That judgement, of when a built-in fits naturally versus when it must be forced, should be based on both readability and measurement, for realistic data, rather than on a blanket rule that built-ins are always better, because blanket rules, in performance as elsewhere, have exceptions that matter in practice.

Standard library modules, beyond built-ins, also offer optimised operations for common tasks, such as efficient data structures, fast text processing, and specialised algorithms, which, like built-ins, are implemented efficiently and tested thoroughly, and which should be preferred to hand-written equivalents, when they fit. Learning which modules exist, and what operations they provide, is part of learning Python well, and it pays off in both performance and correctness, because it prevents reinventing tools that already exist, in slower and less reliable forms. That learning, through reading documentation and through experience, is an ongoing part of Python development, and it complements profiling and measurement, by giving you a toolkit of efficient operations to apply when profiling identifies a bottleneck that a library operation can address.

Check: Why are built-in operations often both faster and clearer than hand-written loops?

Section 16

Input and output optimisation focuses on reducing the cost of reading and writing data, which, as discussed for bottlenecks, often dominates total time for programs that process files, networks, or other external data. One key technique is buffering, where data is read or written in larger chunks, rather than in many small operations, because each operation has overhead, such as a system call, and buffering amortises that overhead over many items, by collecting them into a single operation. Python's file handling buffers by default, in most cases, which means that reading line by line, in Python code, does not necessarily cause a system call per line, because the underlying buffer reads a chunk and serves lines from it, but writing many small pieces, without buffering, or flushing after every piece, can still be slow, and understanding buffering, and avoiding unnecessary flushing, is a practical I/O optimisation, especially for programs that write large outputs.

Batching is another I/O optimisation, where many small requests, such as database queries or network requests, are combined into fewer, larger requests, reducing overhead per item, because each request has fixed costs, such as connection setup, query parsing, or network round-trip time, which are incurred once per request, regardless of how much data it carries, up to a point. A program that queries a database once per item, for thousands of items, will spend most time in request overhead, rather than in actual data retrieval, while one that batches items into a single query, or into a few large queries, will spend far less time in overhead, and will finish much faster, even though it retrieves the same data.

Reducing data volume is a third I/O optimisation, where a program reads or writes only the data it actually needs, rather than entire records or files or by compressing output, where appropriate, to reduce bytes written and transmitted, at the cost of compression time, which may be worthwhile if I/O is the bottleneck, but not if computation is. That volume reduction, guided by an understanding of what data a program actually uses, can dramatically reduce I/O time, especially for large files or for network transfers, where bandwidth is limited, and it is often simpler than other optimisations, because it involves changing what is read or written, rather than how it is processed, though it requires care to ensure that needed data is not accidentally omitted, which would cause incorrect results, a worse outcome than slow but correct processing.

Asynchronous I/O, as discussed for concurrency, is another I/O optimisation, where a program overlaps I/O operations, so that while one operation waits, another proceeds, rather than waiting sequentially for each operation to complete, which can dramatically reduce total time for programs that perform many independent I/O operations, such as downloading many files, as discussed in the concurrency chapter. That overlapping, achieved through threads, processes, or asynchronous tasks, depending on the program's design, is particularly effective when I/O waiting dominates, because it uses waiting time productively, as discussed earlier, and it should be considered for I/O-heavy programs, after simpler optimisations, such as buffering and batching, have been applied, because concurrency adds complexity, and it should only be added when measurement shows that I/O waiting is a genuine bottleneck that simpler optimisations cannot address.

Check: How does batching many small requests into fewer large ones reduce I/O cost?

Section 17

Sorting and searching are fundamental operations whose performance depends heavily on algorithm and data structure choice, and understanding their basics helps you choose efficient approaches for common tasks. Sorting a collection, using a built-in sort, is generally efficient, with time growing slightly faster than linearly with input size, in typical cases, which is good for large data, while a hand-written sorting algorithm, especially a simple one that compares every pair, can grow quadratically, becoming impractical for large inputs, as discussed for complexity. That is why built-in sorting should be preferred, as discussed earlier, not only for speed, but also for correctness, because it handles edge cases, such as equal values and different types, correctly, whereas a hand-written sort may contain subtle bugs that only appear for particular data, in addition to being slower.

Searching for an item in a collection also depends on structure: searching an unsorted list requires checking items one by one, in the worst case, which grows linearly with size, while searching a sorted list, using a binary search that repeatedly halves the search range, grows logarithmically, which is much faster for large data, and searching a dictionary or set, using hashing, grows roughly constantly, regardless of size, in typical cases, as discussed for data structures. Choosing a search method based on whether data is sorted, whether it will be searched many times, and whether it can be stored in a dictionary or set, is therefore a key performance decision, because the same search, performed many times, can dominate total time if done linearly, while a faster method, chosen upfront, can make the whole program dramatically faster, with little extra code, such as by building a dictionary once and reusing it for many lookups, rather than searching a list each time.

Preprocessing, such as sorting data once, or building an index, such as a dictionary mapping keys to values, is a classic trade-off, where upfront work, which takes time and memory, pays off if the preprocessed structure is then used many times, because each subsequent operation becomes much faster, amortising the upfront cost over many uses. That reuse count, which can be estimated from a program's logic or measured from its actual use, is the key factor in deciding whether preprocessing is worthwhile, for a particular program.

Sorting stability and key functions also affect performance and correctness, because sorting by a complex key, such as a function that computes a value for each item, can be expensive if that function is called many times during sorting, though built-in sorting typically calls the key function once per item, caching its result, which is efficient, provided the key function itself is not excessively expensive. Understanding that key caching, and designing key functions to be simple and fast, such as by precomputing values, rather than by performing expensive calculations inside a key function that will be called for every item, can improve sorting performance, especially for large data, where key computation cost multiplies by input size, becoming a significant part of total sorting time, even if the sorting algorithm itself is efficient.

Check: Why is building a dictionary index worthwhile when many searches will be performed?

Section 18

Concurrency can improve performance for programs that wait, as discussed in the concurrency chapter, but it should be applied to performance problems only after simpler optimisations have been considered, because it adds complexity and potential bugs, as discussed there, and because its benefit depends on whether a program's bottleneck is waiting, computation, or something else. For a waiting-heavy program, such as one that downloads many files or queries many services, concurrency, through threads, processes, or asynchronous tasks, can dramatically reduce total time, by overlapping waiting operations, so that total time approaches the longest single wait, rather than the sum of all waits, which is a large improvement when waits are long and numerous. That benefit, however, depends on waits being independent, so that they can overlap, and on the external services being able to handle concurrent requests, without rate limiting or overload, which, if present, may limit concurrency's benefit or even cause failures, if concurrency is too aggressive, as discussed for flow control and for service respect.

For compute-heavy programs, concurrency through processes, as discussed for the interpreter lock, can improve performance by parallelising computation across cores, provided that work can be divided into independent pieces, with minimal communication, because communication overhead, as discussed for inter-process communication, can offset parallel gains, especially for small pieces of work, where communication costs dominate computation, making parallelisation slower than sequential execution, despite using multiple cores. That chunking consideration, of dividing work into pieces large enough to amortise communication, is crucial for compute parallelism, and it should be tuned by measurement, with different chunk sizes and worker counts, to find the configuration where parallel benefit outweighs overhead, for a particular workload, rather than assuming that more workers always help, which they do not, beyond the point where overhead and contention dominate.

Concurrency for performance should also be weighed against its costs in complexity, debugging difficulty, and resource use, as discussed in the concurrency chapter, because a concurrent program that is faster but incorrect, due to a race, or that is faster on one machine but fails on another, due to resource limits, is not a successful optimisation, regardless of its speed on a particular test. That weighing, of speed benefit against complexity and risk, should be based on measurement, showing a significant speedup for realistic data, and on testing, showing correctness under concurrent load, with ordering, shutdown, and error handling verified, as discussed for concurrent testing, before concurrency is adopted as a performance solution, rather than as an experiment that happens to be faster in one run, but that has not been verified for correctness or for stability under varied conditions.

For a program that is both waiting-heavy and compute-heavy, a hybrid design, with asynchronous handling for waiting and a process pool for computation, may be the best overall choice.

Check: Why should algorithm improvement come before concurrency as a performance solution?

Section 19

Readability and performance often trade off against each other, because the clearest code is not always the fastest, and the fastest code is not always the clearest, and managing that trade-off is a key skill in performance work. Clear code, with descriptive names, straightforward logic, and simple structures, is easier to read, test, and maintain, and it is less likely to contain bugs, because its intent is obvious to a reader, who can spot mistakes more easily than in clever, optimised code, where intent is obscured by tricks designed for speed. Optimised code, by contrast, may use less obvious structures, such as precomputed values, cached lookups, or manual loop restructuring, which improve speed, but which make code harder to understand, especially for a reader who does not know that optimisation was deliberate and measured, and who may simplify it back to clearer but slower code, without realising that performance will suffer, unless the optimisation is documented, as discussed for micro-optimisations.

The trade-off should be managed by prioritising readability by default, and sacrificing it for performance only where measurement shows a genuine need, such as in a hot spot identified by profiling, where a speedup is significant and where the program's requirements demand it, because in most code, readability matters more than a small speedup, since most code is not a bottleneck, and its performance, while not optimal, is sufficient, as discussed for performance requirements. That prioritisation, of readability first, performance second, except in proven hot spots, keeps a codebase maintainable, while still allowing performance where it matters, and it prevents the accumulation of premature optimisations, discussed earlier, which collectively degrade readability without meaningfully improving speed, because they are applied everywhere, rather than only where needed.

When performance does require sacrificing readability, the sacrifice should be minimised and documented, by keeping the optimised code as clear as possible, given its optimisation, by adding comments explaining why the less obvious form was chosen, what measurement justified it, and what the clearer form would cost in performance, so that a future maintainer can judge whether the optimisation is still needed, or whether it can be simplified, if performance requirements or data characteristics change.

Another way to manage the trade-off is to isolate optimised code, by encapsulating it in a well-named function, with a clear interface, so that its internal complexity does not spread throughout the program, and so that callers can use it as if it were simple, without needing to understand its optimisation. That encapsulation, which is good design generally, is particularly valuable for performance, because it confines complexity to the place where it is needed, while keeping the rest of the program clear, and it allows the optimised implementation to be changed, or even replaced with a simpler one, if performance requirements change, without affecting callers, provided the interface remains the same. That isolation, of performance-critical code from the rest of a program, is a practical technique for balancing readability and speed, and it should be used whenever optimisation is necessary, to prevent its complexity from contaminating an entire codebase.

Check: How can encapsulating optimised code in a function help balance readability and speed?

Section 20

Performance regression testing means checking, automatically, that a program does not become significantly slower as it evolves, by running benchmarks on every significant change and comparing results to a baseline, as discussed for benchmarking generally. That automation, similar to correctness testing, provides an early warning when a change, such as a new feature, a refactoring, or a dependency update, inadvertently slows a program, allowing the slowdown to be investigated and fixed promptly, while the change is still fresh in the author's mind, rather than discovered later, when users complain, and when the cause is harder to identify, because many changes have accumulated since the slowdown was introduced. That early detection, like early bug detection generally, reduces the cost of fixing performance problems, because it catches them when they are small and their cause is clear, rather than when they have grown into a major, mysterious slowdown.

Designing performance regression tests involves choosing benchmarks that are representative, stable, and sensitive, as discussed for benchmarking, and setting thresholds for acceptable change, such as a maximum percentage slowdown, beyond which a test fails and alerts the author, rather than requiring exact equality, which would fail due to normal variation, even when no real regression occurred. That threshold, chosen based on measurement variation, such as typical run-to-run variation for a benchmark, ensures that regression tests are not flaky, failing randomly due to noise, which would cause developers to ignore them, defeating their purpose, while still catching real regressions, which exceed normal variation by a clear margin. That threshold tuning, based on observed variation, is a practical part of performance regression testing, and it should be revisited as benchmarks and hardware change, to keep tests both sensitive and reliable.

Performance regression tests should be run in a consistent environment, as much as possible, because benchmarks run on different machines, or under different load, are not directly comparable, and a test that compares a result on one machine to a baseline recorded on another may falsely report a regression, or miss a real one, due to hardware differences, rather than code changes. Running regression tests on a dedicated, quiet machine, or in a controlled environment, such as a continuous integration service with consistent hardware, improves comparability, though it cannot eliminate variation entirely, which is why thresholds, rather than exact comparisons, are used, as discussed above. That environmental consistency, combined with repeated runs and typical values, rather than single runs, makes regression testing more reliable, even in shared environments where perfect consistency is unattainable.

A regression test that is ignored because it is flaky is worse than no test, because it creates a false sense of coverage while actually providing none. Documenting benchmark results alongside code changes also helps future maintainers understand why a particular performance decision was made.

Check: Why should performance regression thresholds allow normal variation rather than requiring exact equality?

Section 21

Profiling in production, as opposed to profiling in development, involves measuring a program while it serves real users, with real data and real load, which can reveal bottlenecks that do not appear in development, because production data, traffic patterns, and hardware differ from test conditions, in ways that change which functions dominate, how much memory is used, and where waiting occurs. That production insight is valuable, because a program optimised for development data may still be slow in production, if its real workload differs, such as by having larger inputs, more concurrent users, or different data distributions, that shift bottlenecks to parts of code that were not hot in testing, and only production profiling, or production-like load testing, can reveal those shifts before users complain, though production profiling must be done carefully, to avoid slowing the program itself, or exposing sensitive data in profile reports.

Sampling profilers, which periodically record what a program is doing, rather than recording every function call, are a common production profiling technique, because they have low overhead, typically only a few percent, compared with detailed profilers that record every call and can slow a program significantly, making them unsuitable for continuous production use, though they are fine for short development runs. Sampling works by interrupting the program at regular intervals, recording its current call stack, and aggregating those samples over time, to estimate where time is spent, with accuracy that improves as more samples are collected, though it may miss very short functions that rarely appear in samples, which is acceptable for production profiling, where the goal is to find major hot spots, not to measure every function precisely, and where low overhead matters more than perfect detail, because a profiler that slows production significantly defeats its own purpose, by degrading the service it is meant to improve.

Production profiling data should be handled securely, as discussed for logs generally, because profile reports can reveal internal details, such as function names, file paths, and data patterns, which, while useful for developers, could help an attacker understand a program's structure, if they were exposed, and they should therefore be stored with restricted access, retained only as long as needed for analysis, and never included in user-facing output, such as error pages or API responses, where they could leak internal information. That secure handling, combined with low-overhead sampling, and with a clear purpose, such as investigating a specific slowdown or verifying a performance improvement in production, makes production profiling a practical, responsible tool, rather than a risky intrusion into a live system, and it should be used sparingly, for defined investigations, rather than continuously, unless its overhead and data handling have been carefully assessed and approved for continuous use, in which case it can provide ongoing visibility into production performance, similar to monitoring and logging, which are continuous by design.

Check: Why are sampling profilers preferred for production over detailed call-recording profilers?

Section 22

Data processing performance, for programs that transform large datasets, depends on how data is represented, how it is accessed, and how many times it is passed through, because each pass over data costs time, and each representation has different costs for different operations, as discussed for data structures generally. A program that reads data, transforms it, filters it, and then aggregates it, in separate passes, each materialising an intermediate collection, will be slower and more memory-hungry than one that combines those stages into a single pass, or that streams data through stages without materialising intermediates, as discussed for lazy evaluation, because separate passes repeat work, such as iterating over data multiple times, and intermediate collections consume memory and time to build, even though they are not part of the program's final output, and are only a means to the next stage. That single-pass, streaming design, while it may be slightly less straightforward to write, because stages must be composed carefully, is often the most effective data processing optimisation, especially for large data, where multiple passes and intermediates become prohibitively expensive, in both time and memory.

Representing data efficiently also matters, because a dataset stored as a collection of objects, each with overhead for its type and attributes, uses far more memory, and is slower to process, than the same data stored as simple values in a compact structure, such as a list of tuples or a dictionary with simple keys, though the object representation may be clearer and easier to work with, for small data, where its overhead is negligible. That trade-off, between clarity and efficiency, should be judged by data size, as with other performance trade-offs, because for small data, clarity wins, while for large data, efficiency may be necessary to avoid running out of memory or taking unacceptably long, and a program that starts with clear object representations may need to switch to more compact representations as its data grows, which is a refactoring that should be guided by measurement, showing that representation overhead is actually a bottleneck, rather than assumed, because changing representation affects code throughout a program, and should only be done when its benefit is clear and significant.

Filtering and transformation order can also affect performance, because filtering data early, to reduce its size before expensive transformations, is generally faster than transforming all data and then filtering, because transformation work is wasted on items that will later be discarded, while filtering first ensures that expensive work is only done on items that will actually be used. That ordering principle, of reducing data as early as possible, applies to many data processing pipelines, from simple scripts to complex systems, and it is a straightforward optimisation that is often overlooked, because a pipeline written in logical order, transformation then filtering, may feel natural, even though filtering first, where possible, is more efficient, provided that filtering does not depend on transformed values, in which case transformation must come first, and the ordering must respect data dependencies, not just efficiency, because an incorrect order, chosen only for speed, can produce wrong results, which is worse than slow but correct processing.

Check: Why is filtering data early, before expensive transformations, usually faster?

Section 23

Generators versus lists, as a performance choice, has already been discussed for lazy evaluation, but it deserves a deeper look, because the choice affects not only memory, but also time, reusability, and error handling, in ways that should be weighed for each use. A generator produces values on demand, using little memory, and it can stop early, saving time, when only part of its output is needed, as discussed earlier, but it can only be iterated once, after which it is exhausted, and it cannot be indexed, sorted, or measured for length, without consuming it, which means that a program that needs to reuse values, or to access them randomly, must either materialise them into a list, losing laziness's benefits, or regenerate them, repeating work, which may be more expensive than materialising once, if regeneration is costly and reuse is frequent. That single-use limitation is a key drawback of generators, and it should be considered when choosing between them and lists, because a generator that is accidentally iterated twice, expecting the same values, will produce no values on the second iteration, leading to a bug that is easy to miss in testing, where a generator may be iterated only once, but that appears in production, where reuse is attempted, causing incorrect results or empty output, which is a correctness problem, not just a performance one.

Lists, by contrast, store all values, using more memory, but they can be reused, indexed, sorted, and measured, which makes them more flexible, and for small data, where memory is not a concern, that flexibility outweighs laziness's memory saving, because the program's logic becomes simpler and less error-prone, with a complete collection available for any operation, rather than a one-shot stream that must be handled carefully. That simplicity, for small data, is why lists are the default choice for many programs, and generators should be chosen only when their benefits, of memory saving or early stopping, are actually needed, for large data or for pipelines where only part of the output will be used, as measured, rather than by default, because defaulting to generators everywhere adds complexity, such as handling exhaustion and delayed errors, without benefit, for programs where data is small and reuse is common, which is a form of premature optimisation, in the sense discussed earlier, where a performance technique is applied without a measured need.

Converting between generators and lists is straightforward, and it can be used to get the best of both, such as by using a generator pipeline for processing, and materialising its output into a list only at the end, when the final results are needed for sorting or for multiple uses, which preserves streaming benefits during processing, while providing a reusable collection for final use, at the cost of memory for the final results only, rather than for all intermediates, which is a good compromise for many pipelines. That hybrid approach, of streaming during processing and materialising at the end, is a practical pattern that balances memory, time, and flexibility, and it should be considered whenever a pipeline's final output will be reused, while its intermediates need not be, because it captures most of laziness's benefit, without its single-use drawback for final results, which are the results that matter most for reuse.

Check: Why can a generator's single-use nature cause a bug if it is iterated twice?

Section 24

Startup time, the time from when a program is launched to when it is ready to do useful work, matters for command-line tools, scripts, and other programs that are run frequently, because even a small startup delay, multiplied by many runs, becomes a significant total delay, and because users notice startup delays directly, as waiting before any output appears, even if total run time is short. Startup time is affected by how much code a program imports, how much initialisation it performs, and how much data it loads before its main work begins, and a program that imports many large libraries, or that loads a large configuration or dataset at startup, even when that data is not needed for a particular run, such as when the user only asked for help, will feel slow, even if its main work is fast, because startup dominates its total time for short runs, which are common for command-line tools, where many runs are quick, such as showing help or processing a small file.

Optimising startup time starts with measuring it, by timing a program's launch, with different commands, such as help versus a real task, to see how much time is spent before main work, and with profiling startup, if needed, to identify which imports or initialisations dominate, because, as with other performance work, guessing at startup bottlenecks is unreliable, and measurement may reveal a surprising culprit, such as a single large library import that is only needed for one subcommand, but that is imported for every run, even when that subcommand is not used. That lazy import pattern, where a library is imported only when the subcommand or feature that needs it is actually used, rather than at program startup for all runs, is a standard startup optimisation, because it defers import cost until it is needed, making common, quick runs, such as help, much faster, while only slightly delaying runs that actually need the library, which is a good trade, because help runs are frequent and should be instant, while library-needing runs are already longer and can absorb import time less noticeably.

Another startup optimisation is to avoid loading large data or performing expensive initialisation before it is known to be needed, such as by loading configuration lazily, or by deferring database connections until the first query, rather than at startup, which, like lazy imports, makes startup faster for runs that do not need that data or connection, and it also makes error handling clearer, because a failure to load data or connect will occur when that feature is actually used, rather than at startup for every run, even for runs that would not have needed it, which can be confusing for users who only wanted help, but who receive an error about a database connection they did not intend to use. That lazy initialisation, applied consistently, turns startup into a minimal, fast step, with work deferred until its need is confirmed, which is a general performance principle, of doing work only when needed, applied to startup, where its benefit is most visible to users, because startup is the first experience they have of a program, and a fast startup creates a good impression, even before any real work is done.

Check: Why does importing a large library at startup slow down even a help command?

Section 25

Command-line tools have performance considerations that differ from long-running programs, because they are run frequently, for short durations, and their total time includes startup, as discussed earlier, plus their main work, plus shutdown, and users experience all three as waiting, even though only main work is productive. A tool that takes a second to start, due to heavy imports, and then processes a small file in a tenth of a second, will feel slow, because most of its time is startup, not work, and optimising its main work, even dramatically, will have little effect on user experience, because startup dominates, which is why startup optimisation, discussed earlier, is particularly important for command-line tools, and why profiling a tool should include startup, not just main work, to avoid optimising the wrong part, as discussed for measurement generally, where the slowest stage, not the most obvious one, should be optimised first.

Output performance also matters for command-line tools, because a tool that prints a large amount of output, line by line, with many small writes, or with flushing after every line, can be slowed significantly by output overhead, especially when output is redirected to a file or piped to another program, where each write has a cost, and where buffering, as discussed for I/O, can dramatically improve speed, by collecting output into larger chunks before writing, rather than writing each line separately, with a flush, which forces the system to handle each line as a separate operation, wasting time that could have been saved by buffering, which Python's output handling does by default, unless explicitly disabled, though a tool that explicitly flushes, for progress reporting, should do so sparingly, such as every hundred items, rather than every item, to balance progress visibility against output overhead, as discussed for progress reporting generally.

Tool performance should also be judged by how it scales with input size, as discussed for complexity, because a tool that is fast on a small test file may become slow on a large real file, if its algorithm scales poorly, such as by searching a list for each item, rather than using a dictionary, or by loading an entire file into memory, rather than streaming it, and that scaling failure will only become apparent when users try the tool on real data, which is larger and more varied than test data, leading to complaints that the tool is slow, even though it was fast in testing, which is a common performance surprise for command-line tools, whose test data is often tiny compared with real data, and whose developers may not have tested with large inputs, because they assumed that small-data performance would generalise, which, as discussed throughout this chapter, it often does not, when scaling is poor.

Check: Why does startup time dominate user experience for a short command-line tool?

Section 26

Automating performance testing, as part of a development workflow, means running benchmarks and profile checks automatically, on every change or on a schedule, so that performance regressions are detected early, as discussed for regression testing generally, but with additional considerations for automation, such as ensuring that automated benchmarks run in a consistent environment and that their failures are actionable, rather than flaky, which would cause developers to ignore them, as discussed for regression thresholds. Automated performance testing can be as simple as a script that runs a program on a standard dataset, records its time, and compares it to a baseline, failing if it exceeds a threshold, or as elaborate as a continuous integration setup that runs a suite of benchmarks, records results over time, and graphs trends, allowing developers to see performance evolution, not just pass or fail for a single change, which is valuable for long-term performance maintenance, because gradual slowdowns, each too small to trigger a threshold alone, can accumulate into a significant slowdown over many changes, which only trend analysis reveals, by showing a steady upward slope in benchmark times, even when no single change caused a dramatic regression.

Automated performance tests should be separated from correctness tests, because they have different requirements: correctness tests should be fast, deterministic, and run on every change, while performance tests may be slower, may vary with machine load, and may need a quiet, consistent environment to produce reliable results, which is not always available on every developer's machine or on every continuous integration run, where other jobs may be running concurrently, affecting timing. That separation, by running performance tests on a schedule, such as nightly, or on a dedicated machine, rather than on every commit, balances the need for early detection against the need for reliable, non-flaky results, and it prevents performance tests from slowing down development, by making every commit wait for long benchmarks, which would discourage developers from running tests at all, defeating their purpose, while still providing regular performance feedback, on a timescale that is sufficient to catch regressions before they accumulate too far, though not as immediately as per-commit testing would, which is a reasonable trade-off for most programs, where performance regressions are less urgent than correctness failures, and where a daily check is sufficient to catch them before release, provided that releases are not made without checking the latest performance results.

Finally, automated performance testing should be documented, with benchmarks, datasets, thresholds, and expected results recorded, so that a new developer can understand what is being tested, why, and how to interpret a failure, rather than facing a mysterious performance test that fails intermittently, without explanation, which is likely to be ignored or disabled, as discussed for flaky tests generally. That documentation, combined with a clear process for investigating a performance failure, such as profiling the new version and comparing it to the old, as discussed for regression diagnosis, turns automated performance testing from a bureaucratic check into a practical tool, that developers trust and use, because its failures are meaningful, its results are understandable, and its process for fixing a regression is clear, which is the standard that any automated testing, for correctness or for performance, should meet, to be sustainable over time, rather than becoming a neglected, flaky suite that everyone ignores.

Check: Why should performance tests often run on a schedule rather than on every commit?

Section 27

Interpreting profiler output has pitfalls that can mislead optimisation, even when profiling itself is done correctly, because a profiler report shows where time was spent in a particular run, but it does not, by itself, say what should be optimised, or whether optimisation will help overall, and a naive reading can lead to wasted effort or to incorrect conclusions. One pitfall is focusing on a function with high total time, without checking whether its time is spent in itself or in its callees, as discussed for total versus self time, because optimising a caller, when its callees are the real hot spots, will have little effect, while optimising the callees, or reducing calls to them, will help, and a profiler report that is read without that distinction can lead to optimising the wrong function, with no measurable improvement, despite effort, which is frustrating and which can discourage further performance work, if the developer concludes, incorrectly, that profiling does not help, when in fact the profiler was misread, not misleading.

Another pitfall is over-interpreting small differences, such as a function that takes two percent of total time versus one that takes three percent, because those differences may be within measurement variation, or may be due to profiling overhead, rather than to a real performance difference, and optimising the three percent function, while ignoring a fifty percent function, because the smaller one looks easier to optimise, is a poor prioritisation, driven by ease rather than by impact, which is a common human bias in optimisation, where developers optimise what is familiar or easy, rather than what matters most, as measured. That bias should be countered by sorting profiler output by total time, and by focusing on the top few functions, which together account for most time, rather than on minor functions, even if they are easier to understand or to optimise, because the greatest benefit comes from the largest hot spots, as discussed for bottleneck prioritisation generally, and a small improvement to a large hot spot outweighs a large improvement to a tiny one, in absolute terms, which is what users experience as total time.

A third pitfall is profiling with unrepresentative data, as discussed repeatedly, where a profile on test data shows different hot spots from real data, leading to optimisation for test data that does not help real users, which is a waste of effort and a source of disappointment when real performance does not improve, despite optimisation, because the real bottleneck was never profiled, only the test bottleneck was. That pitfall is avoided by profiling with realistic data, as emphasised throughout this chapter, and by profiling with several representative datasets, if a program's workload varies, such as by data size, by distribution, or by operation mix, to ensure that optimisation addresses hot spots that are consistently hot across workloads, rather than hot only for one particular test case, which may not reflect real use, and which may lead to over-specialisation for test data, at the expense of general performance.

Check: Why is it misleading to optimise a function with high total time without checking its self time?

Section 28

Performance work in teams requires shared benchmarks, shared baselines, and shared understanding of performance goals, because a program's performance is affected by many contributors, each of whom may make changes that individually seem harmless, but that collectively slow a program, as discussed for regression testing, and only a shared, automated performance practice can detect and prevent that collective slowdown, by making performance visible to everyone, not just to a single performance specialist, who cannot review every change for its performance impact, especially in a large codebase where changes are frequent and diverse. That shared practice starts with documenting performance goals, such as acceptable response times, memory limits, and throughput targets, in a place where all contributors can find them, such as a project's readme or contributor guide, so that a contributor knows, before making a change, whether their change must meet a performance requirement, and how to test whether it does, rather than discovering a performance problem only after their change is merged and users complain, which is too late for easy fixing, because the change is already integrated and its author may have moved on to other work.

Shared benchmarks, as discussed for automated testing, are the mechanism by which team performance goals are enforced, because they provide a common, repeatable measurement that all contributors can run, and that continuous integration can run automatically, to check whether a change meets goals, or whether it regresses performance beyond a threshold, as discussed for regression testing. Those benchmarks should be maintained collectively, with new benchmarks added when new features are added, and old benchmarks updated when behaviour changes, so that the benchmark suite remains representative of a program's current workload, rather than becoming outdated, testing only old features while new features, which may be slow, go untested, which is a common failure mode for benchmark suites that are not maintained alongside code, similar to test suites that are not maintained, which gradually lose coverage and usefulness, even though they still pass, because they no longer test what matters most for current users.

Performance review, as part of code review, should also be a team practice, where reviewers ask, for a change that affects a hot spot, or that adds a new feature with performance implications, whether it was benchmarked, whether its performance was compared to a baseline, and whether its complexity is justified, as discussed for optimisation review generally, rather than accepting performance claims without evidence, or rejecting performance concerns as premature, without measurement, because both extremes, of uncritical acceptance and of blanket dismissal, harm team performance practice, by either allowing regressions or by discouraging legitimate performance work, and a balanced review, focused on evidence and on proportion, as discussed earlier, helps a team maintain both performance and maintainability, by ensuring that performance is considered, but not over-prioritised, for every change, in proportion to its actual impact, as measured, not as guessed.

Check: Why do teams need shared benchmarks rather than relying on individual developers' timing?

Section 29

Response time, for programs that interact with users or with other programs over a network, is a performance metric that deserves special attention, because users experience response time directly, as waiting, and because slow response times can cause users to abandon a task, retry a request, or perceive a program as broken, even if it eventually completes correctly, which is a user experience failure, not just a performance statistic. Response time is affected by many factors, including computation time, I/O waiting, network latency, and queuing, where requests wait behind other requests, and optimising response time therefore requires understanding which of those factors dominates, for a particular program and workload, through measurement, as discussed for section timing and profiling, rather than assuming that computation is the bottleneck, when in fact queuing or network waiting may dominate, especially under load, where many requests compete for limited resources, causing each request to wait longer, even though its own computation is fast, which is a classic performance problem for servers under load, where response time degrades non-linearly as load approaches capacity, a phenomenon that can be surprising to developers who test only with low load, where response time is good, and who are then surprised when production response time collapses under real load, because queuing effects were not anticipated or tested.

Measuring response time should be done from the user's perspective, meaning from when a request is sent to when a complete response is received, rather than from when a program starts processing a request, because the former includes queuing, network, and other delays that the user actually experiences, while the latter excludes them, giving an optimistic picture that does not match user experience, and that can hide queuing problems. That user-perspective measurement, combined with internal timing, to distinguish processing time from waiting time, gives a complete picture of response time, showing both where time is spent and what users actually experience, which is necessary for effective response time optimisation, because optimising only internal processing, when waiting dominates, will not improve user experience, despite improving internal metrics, which is a common performance mistake, where a team optimises what it measures internally, rather than what users experience externally.

Finally, response time goals should be set based on user expectations and on the program's purpose, rather than on arbitrary targets, because different programs have different response time needs: an interactive tool that must feel instant needs response times in the range of fractions of a second, while a batch report that users expect to take minutes can tolerate longer response times, provided it progresses visibly and completes by its deadline, as discussed for performance requirements generally. Setting goals based on user needs, measuring response time from the user's perspective, and optimising the dominant factor, whether computation, I/O, queuing, or network, through the techniques discussed throughout this chapter, forms a practical approach to response time performance, ensuring that optimisation effort improves user experience, not just internal metrics, which is ultimately what performance work is for, in user-facing programs, where speed is experienced, not just measured.

Check: Why should response time be measured from when a request is sent, not from when processing starts?

Section 30

A performance checklist, summarising the practices in this chapter, helps ensure that performance work is systematic, evidence-based, and proportionate, rather than intuitive, scattered, or premature, and it can be used during development, during code review, and before release, to check whether performance has been considered appropriately for a program's context. The first checklist area is requirements: have performance requirements been stated, in terms of acceptable time, memory, and responsiveness, for realistic data and realistic load, and does the program meet them, as measured, not as assumed? If requirements have not been stated, performance work lacks a target, and it is impossible to know whether optimisation is needed, or when it is sufficient, which leads to either under-optimisation, where a program remains too slow, or over-optimisation, where effort is wasted on speed that users do not need, at the cost of readability and maintainability, as discussed for premature optimisation.

The second checklist area is measurement: has performance been measured, with a baseline, with repeated runs, on realistic data, using whole-program timing, section timing, and profiling, as appropriate, to identify bottlenecks, rather than guessing at them? If measurement has not been done, optimisation is speculative, and it may target the wrong parts, wasting effort while real bottlenecks remain, which is a common performance failure, where developers optimise familiar code, rather than measured hot spots, because they did not measure first, or because they measured only on small test data, which does not reflect real workloads, leading to optimisations that do not help real users, despite improving test performance, which is a misleading success that can only be avoided by realistic, repeated measurement, as emphasised throughout this chapter.

The third checklist area is optimisation: have optimisations been applied only to genuine bottlenecks, starting with algorithm and data structure choice, then with I/O, caching, and loop improvements, as appropriate, and have they been verified by measurement, to confirm that they actually helped, and by how much, without introducing regressions in correctness, memory, or readability, that outweigh their speed benefit? If optimisation was applied indiscriminately, or without verification, it may have added complexity without benefit, or may have introduced bugs, and it should be reviewed, with measurement evidence, to decide whether it should be kept, simplified, or removed, because an unverified optimisation is a liability, not an asset, in a codebase, especially if its purpose is undocumented and its benefit is unknown to future maintainers, who may be afraid to change it, even when it is no longer needed.

A checklist used consistently, rather than only when performance is already a problem, helps prevent performance regressions before they occur, by making measurement and verification a routine part of development. That routine use, integrated into code review and release processes, turns performance from a reactive firefight into a proactive practice.

Check: What three areas does the performance checklist in this section cover?

Section 31

Sorting large data efficiently requires attention to algorithm, to key computation, and to memory, because sorting is a common operation whose cost grows faster than linearly, and whose intermediate memory use can be large, especially for data that does not fit comfortably in memory, where sorting must be done in chunks, with external merging, rather than in a single in-memory sort, which would fail or swap excessively, slowing dramatically, as discussed for memory scaling generally. For data that fits in memory, built-in sorting, as discussed earlier, is usually the best choice, because it is efficient, correct, and well-tested, and hand-written sorting should be avoided, except for educational purposes, or for specialised cases where a built-in sort cannot express the required order, in which case a custom key function, designed to be simple and fast, as discussed for key functions, is preferable to a custom sorting algorithm, because key computation, while it can be expensive, is typically far less expensive than a poor sorting algorithm, and because built-in sorting's efficiency, in terms of comparisons and moves, is hard to match with hand-written code, especially for large, varied data, where edge cases, such as equal keys or already sorted data, can cause a naive algorithm to perform very poorly, while a built-in sort handles them gracefully, through adaptive techniques that detect existing order and exploit it, rather than sorting from scratch, which is a significant advantage for real data, which is often partially sorted, such as by date or by identifier, even if not fully sorted by the desired key, and which a naive algorithm would treat as random, wasting work that an adaptive sort saves. Finally, sorting large data should be tested with realistic sizes and distributions, because sorting performance depends on data characteristics, such as how sorted data already is, how many duplicates exist, and how expensive key computation is, all of which vary by dataset, and a sorting approach that is fast for random test data may be slow for real, partially sorted data with many duplicates, or vice versa, depending on its algorithm's adaptivity and its key handling, which is why benchmarking sorting, on realistic data, as discussed for benchmarking generally, is essential, rather than assuming that a built-in sort will always be fast enough, or that a particular key function will always be cheap enough, because those assumptions, while often true, can fail for specific datasets, and only measurement on realistic data reveals whether sorting is actually a bottleneck, and whether precomputation, chunking, or a different key design would help, for that dataset, which is the evidence-based approach to sorting performance that this chapter advocates throughout, for all performance work, not just for sorting.

That evidence-based approach also protects against a common trap: choosing a sorting strategy because it sounds sophisticated, rather than because measurement shows it helps for the data at hand. A simpler sort with a well-chosen key often outperforms a clever custom algorithm, especially when the custom code is harder to test and harder to maintain over time.

Check: Why can precomputing sort keys be faster than computing them during sorting?

Section 32

Performance testing in continuous integration, as discussed for regression testing generally, involves running benchmarks automatically, when code changes, and comparing results to baselines, to detect regressions early, but it also involves practical challenges, such as environment variation, flaky results, and long run times, which must be managed to make automated performance testing sustainable, rather than a source of frustration that developers learn to ignore, as discussed for flaky tests generally. Environment variation, where benchmarks run on shared machines with varying load, can cause timing variation that swamps real performance changes, making it hard to distinguish a genuine regression from noise, and that variation can be reduced, though not eliminated, by running benchmarks multiple times, by using typical values rather than single runs, by setting thresholds that allow normal variation, as discussed for regression thresholds, and by running benchmarks on dedicated or consistent hardware, where possible, rather than on general-purpose shared runners, whose load varies unpredictably, though dedicated hardware may not be available for small projects, in which case thresholds must be set more generously, to avoid false alarms, at the cost of sensitivity to small regressions, which is a trade-off that should be documented, so that developers understand why a performance test passed despite a small slowdown, or failed despite no code change, due to environment variation, rather than assuming that the test is broken, when in fact it is reflecting real, if noisy, measurement conditions. Finally, automated performance testing should be integrated into development workflow in a way that is actionable, meaning that when a performance test fails, it should be clear what changed, which benchmark regressed, by how much, and how to investigate, through profiling, as discussed for regression diagnosis, rather than merely reporting that performance failed, without context, which leaves developers guessing at the cause, and which may lead them to revert a change unnecessarily, or to ignore the failure, because they do not know how to fix it, and that actionability, achieved through clear reporting, through links to benchmark results and to profile comparisons, and through documentation of how to investigate a performance failure, as discussed for regression testing generally, turns automated performance testing from a gate that blocks changes, into a tool that helps developers understand and improve performance, which is its purpose, and which is essential for its long-term adoption, because developers will only maintain a test suite that helps them, not one that merely obstructs them with unexplained failures, especially for performance, where failures are often noisy and where investigation requires profiling skill that not every developer may have, and where clear guidance and tooling can make the difference between a performance test suite that is valued and one that is abandoned.

In practice, a small suite of benchmarks covering the most important workloads is more valuable than a large suite covering everything, because a small suite runs quickly, is easier to keep reliable, and is more likely to be run and acted upon by the whole team.

Check: Why do flaky performance tests erode trust in automated performance testing?

Section 33

String operations have performance characteristics that deserve attention, because text processing is common, and inefficient string handling can dominate a program's time, especially for large text, as discussed for data processing generally, where building, searching, and transforming text are frequent operations, each with different costs depending on how they are done. Building a large string by repeated concatenation in a loop, as discussed earlier, is a classic inefficiency, because each concatenation creates a new string, copying all previous content, leading to quadratic time growth, as the string grows, while collecting parts in a list and joining them once at the end, using a join operation, is linear, because it allocates the final string once, with its total size known in advance, and copies each part only once, which for large strings is a dramatic improvement, and which is also clearer, in many cases, because it separates collecting parts from combining them, making code easier to read, in addition to being faster, which is a rare win, as discussed for built-in operations, where speed and clarity coincide, rather than trading off, as they often do for other optimisations.

Searching text also has performance considerations, because searching for a substring in a large text, using a simple search, can be efficient, if the search is implemented well, as built-in string search typically is, but searching for many patterns, or searching with complex patterns, such as regular expressions, can be expensive, especially if patterns are compiled repeatedly, rather than once and reused, or if a pattern causes excessive backtracking, where it tries many combinations before failing, which can be dramatically slow for certain inputs, even though it is fast for typical inputs, a phenomenon that should be considered when using regular expressions for performance-critical text processing, by compiling patterns once, by testing them with realistic, including worst-case, inputs, and by preferring simpler string operations, such as containment checks or splitting, where they suffice, rather than using a regular expression for a simple task, which adds complexity and potential performance pitfalls, for no benefit, when a simpler operation would do, as discussed for choosing the simplest tool that fits, generally, for performance as for other aspects of programming.

Transforming text, such as changing case, replacing substrings, or formatting values, also has costs that multiply with text size and with the number of transformations, and a program that transforms the same text many times, or that creates many intermediate strings during transformation, can waste time and memory, compared with one that transforms once, or that uses efficient formatting, such as joining parts, rather than repeated concatenation during formatting, which, like building strings generally, can become quadratic if done naively in a loop. That transformation cost, while it may seem minor for small text, becomes significant for large text, such as processing a large log file or a large document, where each transformation is applied to many lines, and where a small per-line inefficiency multiplies by the number of lines, becoming a large total cost, which profiling, as discussed earlier, will reveal as a hot spot in text processing code, guiding optimisation to the transformation that dominates, rather than to all transformations indiscriminately, which would be a poor use of effort, as discussed for bottleneck prioritisation generally.

Check: Why is joining list parts once faster than repeated string concatenation in a loop?

Section 34

File input and output patterns, for formats such as CSV and JSON, have performance implications that go beyond general I/O optimisation, because parsing and generating those formats involves work that can dominate processing time, especially for large files, where parsing each record, validating its fields, and converting its values from text to numbers or dates, can be more expensive than reading the file itself, as discussed for I/O bottlenecks generally, where computation and I/O both contribute, and where optimising only I/O, such as by buffering, may not help much, if parsing dominates, in which case optimisation should focus on parsing, such as by using a faster parser, by avoiding unnecessary conversions, or by processing only needed fields, rather than parsing and converting every field for every record, when only a few fields are actually used, which is a common waste in data processing programs that read entire records, convert all fields, and then use only a subset, for their final output, while a program that reads only needed fields, or that defers conversion until a field is actually used, can be significantly faster, especially for wide records with many fields, where most fields are unused, and where conversion cost, multiplied by the number of records, becomes large, even though each conversion is individually cheap.

JSON parsing, in particular, can be expensive for large, nested structures, because parsing involves building Python objects for every value, including nested dictionaries and lists, which consumes both time and memory, and a program that parses a huge JSON file into memory, only to extract a small part of it, wastes time building objects that will never be used, while a streaming JSON parser, which reads values incrementally, without building the entire structure, can be far more efficient, for such selective extraction, though it is more complex to use, and it should only be adopted when measurement shows that JSON parsing is actually a bottleneck, and that selective extraction would help, rather than by default, because for most programs, where JSON data is modest in size and fully used, a standard parser is simpler and fast enough, and streaming adds complexity without benefit, which is a trade-off that should be judged by measurement, as with other performance decisions, where simplicity wins unless a measured bottleneck justifies complexity, as discussed for premature optimisation generally, where adding complexity without a measured need is discouraged, even when the complexity is a well-known technique, such as streaming parsing, which is powerful but not always necessary.

CSV parsing, similarly, can be optimised by using the standard library's CSV module, which is efficient and correct, rather than by splitting lines manually, which, as discussed for CSV handling generally, is not only slower in some cases, due to manual handling of quoting and escaping, but also incorrect for complex CSV data, with quoted commas or line breaks, and a program that manually parses CSV, for performance reasons, may actually be slower, once quoting handling is added correctly, than the standard module, which is optimised for that task

Check: Why can manual CSV parsing be both slower and buggier than using the standard CSV module?

Section 35

Energy efficiency is becoming an important performance consideration as programs run on battery-powered devices, on shared servers where energy costs money, and on large-scale systems where energy use has a real environmental cost. It is closely related to time performance, because a program that uses less processor time, less memory, and less input and output generally uses less energy. The relationship is not exact, however, because energy also depends on how a program waits. A program that polls repeatedly, checking for work in a tight loop, consumes processor time and energy even while idle. A program that waits for events, sleeping until work arrives, uses far less energy for the same responsiveness. For a program that spends most of its time waiting, such as a background tool or a mobile app, that waiting behaviour can dominate total energy use, even if its actual computation is fast. Users notice battery drain directly, so an efficient waiting design is a user-visible quality issue, not just an engineering detail.

Many techniques that improve time performance also improve energy efficiency, because they reduce total work. Caching avoids recomputation, batching groups input and output operations so fixed costs are shared, and lazy evaluation avoids processing data that is never used. Each of these saves energy as well as time, though there are trade-offs: caching uses memory, and memory itself consumes some energy, but for expensive operations the saving from avoiding recomputation usually far outweighs the memory cost. Batching may delay individual results slightly, but it reduces the number of operations overall. The same measurement discipline applies: optimise work that measurement shows is significant, rather than guessing, and verify that a change actually reduces work rather than merely moving it elsewhere.

Measuring energy directly is harder than measuring time, because it needs hardware support such as battery monitoring or power measurement tools, which are not available on every system. Energy use also varies with hardware, system load, and other programs running at the same time, making it difficult to attribute energy precisely to one program on a shared machine. For most Python programs, the practical starting point is therefore to reduce time and work, which are measurable and which generally correlate with lower energy use, and to pay particular attention to waiting behaviour. For programs where energy is a primary requirement, such as a mobile app, direct measurement with platform tools should be attempted with realistic workloads. A program that is fast in time may still be energy-hungry if it polls or keeps hardware awake, so both time and, where possible, energy should be checked.

That focus on waiting behaviour also applies to servers that wake repeatedly for trivial checks, and to tools that poll a file or a queue instead of watching for changes. Replacing polling with event-driven waiting is often a simple change with a large energy saving, because it removes work that was never useful in the first place.

Check: Why does polling for work waste energy compared with event-driven waiting?

Section 36

A case study shows how the techniques in this chapter combine in practice. Imagine a program that generates a weekly report from several CSV files by reading each file, converting its rows, filtering them, and then aggregating the results, and that takes far longer than expected, so users complain that reports are late. The first step is to measure, timing the whole program on realistic data and timing its major stages separately, rather than guessing where time goes. Suppose stage timing reveals that reading and converting dominate, while filtering and aggregating are fast. That immediately focuses attention on input handling rather than on aggregation logic, which might have been suspected but which measurement shows is not the bottleneck. It also gives a baseline against which later optimisations can be judged, by comparing stage times before and after each change on the same realistic data.

Profiling the reading and converting stage might reveal that a function converting date strings to dates is called for every row, even for rows that will later be filtered out, because filtering happens after conversion. Converting dates is expensive, due to parsing, and multiplied by many rows it dominates total time, while filtering, which could have removed many rows cheaply, is done too late, wasting conversion work on rows that are ultimately discarded. Moving filtering before conversion, where filtering does not depend on converted dates, or converting dates only for rows that pass filtering, could roughly halve conversion time if filtering removes half the rows. That is a large improvement achieved not by optimising date conversion itself, but by restructuring the pipeline to avoid unnecessary work. It is also a correctness check: the restructuring is only safe if filtering does not depend on converted values, and output before and after must be compared on realistic data to confirm results are unchanged. A faster but incorrect program is a failure, regardless of its speed.

After restructuring, the program should be re-measured to confirm improvement and re-profiled to see whether a new bottleneck has emerged, such as reading itself, which might then be improved by buffering or by reading only needed columns. This cycle of measure, profile, optimise, and re-measure continues until performance requirements are met, at which point optimisation should stop rather than continuing indefinitely at the cost of readability. The case study's lesson is general: measurement prevents wasted effort on fast stages, restructuring to avoid work often beats micro-optimising work that should not have been done, and verification after every change protects correctness. That disciplined cycle is the approach this chapter advocates for all performance work.

The same cycle applies at smaller scale to everyday scripts, not only to report generators. Timing a script, finding its slowest stage, and removing unnecessary work there will usually beat tuning code that measurement shows is already fast enough for its purpose.

Check: In the case study, why was moving filtering before conversion more effective than optimising conversion itself?

Section 37

Scalability describes how a program's performance changes as its workload grows, whether in data size, number of users, or number of requests. It is related to algorithmic complexity, but broader, because it also involves system factors such as hardware limits, network capacity, and shared resources. A system that works well for ten users may fail for ten thousand, due to contention for a shared database, memory limits, or network bandwidth, none of which are captured by algorithmic complexity alone. A database lock that is rarely contended at low load can become a severe bottleneck at high load, when many requests compete for it, causing response times to degrade dramatically even though each request's computation is fast in isolation. Scalability testing with increasing load is therefore essential for programs expected to grow, because it reveals bottlenecks that only appear at scale, before users experience them in production.

Two broad scaling strategies exist. Vertical scaling means making a single machine larger, with more processor, memory, or faster storage. It is the simplest strategy because it requires no change to a program's architecture, only to its hardware, and it can handle moderate growth, but hardware has finite limits and becomes prohibitively expensive at very large scale. Horizontal scaling means distributing work across many machines, each handling part of the workload. It can scale further by adding machines, but it requires a program designed for distribution, with data partitioning, communication between machines, and handling of partial failures where one machine fails while others continue. That adds significant design, testing, and operational complexity. The right choice depends on growth expectations, budget, and team expertise, and should be made deliberately rather than by default.

A program that starts with vertical scaling can transition more easily to horizontal scaling later if it is designed with clear boundaries, such as stateless components that can be distributed, rather than being tightly coupled to single-machine resources such as shared memory or local files. At the same time, over-engineering for infinite scale from the start is a form of premature optimisation: it adds complexity that may never be needed if growth is modest. The balanced approach is to design for reasonable growth, keep future distribution possible, and revisit the scaling strategy as actual growth data becomes available. Load testing, where a program is subjected to increasing workload with realistic data and user behaviour while response time, throughput, and resource use are measured, shows where bottlenecks appear and at what workload degradation begins. That evidence supports capacity planning and scaling decisions far better than speculation.

Capacity planning also benefits from this evidence. Knowing the workload at which degradation begins lets a team decide in advance when to add hardware, when to split work across machines, and when to set limits that protect the system, rather than reacting after users have already been affected.

Check: How do vertical and horizontal scaling differ, and when is each appropriate?

Section 38

Real-time performance, for programs that must respond within strict deadlines, such as control systems, interactive games, or live data processing, differs from general performance because it is not enough for a program to be fast on average. It must be fast consistently, with worst-case response times below a deadline, even under load. A single missed deadline can cause a failure, such as a missed control action, a dropped frame, or a lost data sample, which in real-time contexts can have serious consequences, far beyond a slow response in a batch program where a delay merely means waiting longer. A program that is fast on average but occasionally very slow, due to a rare expensive operation, may fail its real-time requirements even though a benchmark reporting only averages looks excellent. Worst-case times and percentile times, such as the time below which most responses fall and the maximum time observed, are therefore essential metrics for real-time work, because they reveal variability, not just central tendency.

Designing for real-time performance involves favouring operations with predictable, bounded times, even if their average time is slightly higher, because predictability outweighs average speed when deadlines define success. A program that meets deadlines consistently with slightly higher average time is a successful real-time program, while one that is faster on average but misses deadlines occasionally is a failed one, regardless of its average speed. That trade-off should be made explicitly rather than by defaulting to average-optimised operations whose worst case only becomes apparent under worst-case testing or in production. In interactive programs without strict deadlines, the same thinking still helps: users experience variability, not just averages, and a program that is usually fast but occasionally very slow feels worse than one that is consistently moderately fast, because unexpected delays are more frustrating than predictable waiting.

Testing real-time performance requires stress testing with worst-case data, high load, and concurrent activity to trigger rare slow paths, and measuring percentiles and maxima over long runs rather than averages over short runs. It should also be done on production-like hardware with production-like load, because real-time behaviour depends on contention for processor, memory, and input and output from other programs. A program that meets deadlines on a quiet test machine may miss them on a busy production machine. Scheduling also matters: giving real-time tasks higher priority than background tasks prevents them being delayed by less critical work. Together, bounded operations, worst-case testing, and sensible scheduling determine whether deadlines are met in a real system, not a benchmark in isolation, and that system-level view is the ultimate test of real-time performance.

For Python programs in particular, remember that worst-case behaviour can come from the runtime as well as from your own code, such as a large allocation or a blocking call at an awkward moment. Testing under realistic load is therefore the only reliable way to confirm that deadlines will be met in practice.

Check: Why is worst-case response time more important than average time for real-time programs?

Section 39

Communicating performance results to colleagues, users, or managers requires presenting measurements with context, so that they can be understood and acted upon, rather than merely reporting a single number. A time without an explanation of what was measured, on what data, on what hardware, and with what variation is misleading. A result that looks impressive, such as a large speedup, may actually be due to a change in data, hardware, or measurement method rather than to a code change, if context is not provided. Decisions based on such numbers can be wrong, such as adopting a change that does not actually help, or rejecting one that does because its measurement was not comparable to a baseline. Performance communication should therefore always include, at minimum, what was measured, how it was measured, on what data, on what hardware, how many times it was run, and what variation was observed, so that a reader can judge whether a result is reliable and comparable to other results.

A precise-looking number measured on unrepresentative data is less useful than a rough number measured on realistic data with clear context. Precision without context is not accuracy, and a time reported to three decimal places can create a false sense of certainty when measurement variation and differences between test and real conditions mean the true value is only approximate. Results should therefore be presented as ranges with variation and caveats, rather than as single exact figures presented as guarantees. Readers who are told that a result is an estimate with stated uncertainty are less likely to be disappointed or to make wrong decisions when real performance differs from the estimate. Honest, contextualised reporting may feel less impressive than a single dramatic number, but it builds trust over time, whereas oversold numbers damage trust when they fail to match reality.

Good communication is therefore part of performance work, not an afterthought done hastily after measurement is complete. Measurement should be designed from the start to be communicable, with context, variation, and caveats recorded alongside results, so that a program's performance history can be understood and trends can be seen as work progresses. That discipline matters for small scripts and large systems alike, because performance decisions affect users, budgets, and timelines, and they should be based on trustworthy evidence. Clear, honest reporting supports good decisions and sustained investment in performance work, while misleading numbers, even when produced in good faith, lead to poor decisions with real costs. Present results so that someone else could repeat the measurement and check your conclusion: that is the standard performance reporting should meet.

A useful habit is to keep benchmark results with their context in the same place, such as a short note alongside the code or in a team document, so that a result found months later can still be interpreted correctly and compared fairly with newer measurements.

Check: Why should performance results always be reported with context, variation, and caveats?

Section 40

This chapter has covered performance and profiling basics, from measuring before optimising, through profiling, benchmarking, algorithmic complexity, data structures, caching, and input and output optimisation, to concurrency, readability trade-offs, regression testing, and communication. Its lessons can be summarised as a disciplined, evidence-based approach: start with requirements, measure to identify bottlenecks, apply optimisations only to genuine bottlenecks, beginning with algorithm and data structure choice, and verify improvements by re-measurement on realistic data before considering the work complete. Optimising speculatively, based on intuition or on microbenchmarks for cold spots, wastes effort and harms readability without meaningfully improving user experience. The goal is not maximum speed but sufficient speed for a program's purpose, achieved through a balance of time, memory, and responsiveness, judged by requirements and by measurement.

Performance must also be maintained, not just achieved once. A program's performance can gradually degrade through accumulated changes, each individually harmless but collectively significant, which is a common failure mode for programs that are not monitored. They only discover slowdowns when users complain, by which time the cause is hard to identify among many changes. A program with regular performance measurement, baselines, and regression tests detects slowdowns early, when they are small and their cause is clear, allowing prompt fixing with minimal disruption. That early, continuous practice is cheaper, safer, and more effective than reactive fixing after users complain, and it applies to performance as it does to correctness, security, and other quality attributes, which are all best maintained through continuous measurement and testing rather than occasional effort when problems become visible.

Finally, keep performance work proportionate. Optimise where measurement shows a genuine need, keep the simpler and more readable approach where it is fast enough, and document optimisations so future maintainers understand why they exist and whether they are still needed as data, requirements, and hardware evolve. That balance, between speed and readability, between early design and over-engineering, and between average and worst-case thinking, is the heart of performance engineering. If you carry one habit from this chapter into every program you write, let it be this: measure first, change one thing at a time, verify the result, and stop when requirements are met. That habit will help you write programs that are not only fast enough, but also maintainable, reliable, and trustworthy in their performance over time.

As you move on to later chapters, carry this discipline with you. Type hints will help you catch errors early, design patterns will give you reusable structures, and clean code practices will keep optimised code readable. Performance work sits alongside those skills, not apart from them, because a fast program that is hard to understand, hard to test, or hard to change is not a good program, even if its benchmark numbers look impressive in isolation. Measure, verify, and document, and your programs will stay fast enough as they grow.

Check: What disciplined workflow summarises this chapter's approach to performance?

Section 41

Memory profiling complements time profiling, because a program can be fast enough in time yet use so much memory that it fails, slows down through swapping, or crowds out other programs on a shared machine. Time profilers show where time goes, but they do not show where memory goes, which objects are large, which allocations grow over time, or whether memory is being held after it is no longer needed. Memory profiling tools address those questions by tracking allocation sizes, by showing which parts of a program allocate the most, and by comparing memory use over time to reveal growth. A program whose memory grows steadily during a long run, even though its workload is steady, may be holding references to objects it no longer needs, which is a memory leak in the practical sense, even if the language's memory management eventually reclaims some of it. Detecting that growth early, through memory measurement on realistic long runs, prevents failures that short tests never reveal.

Common memory problems have recognisable patterns. Holding an ever-growing list of results, logs, or cached items without a size limit will eventually exhaust memory, as discussed for unbounded caches. Keeping references to large objects in global state, or in objects that live for the whole program, prevents them being reclaimed even after their useful life ends. Building many intermediate objects, such as large temporary lists or strings during data processing, can cause peak memory use far higher than the final result requires, even if those intermediates are eventually reclaimed. Each pattern is fixed differently: bounding caches and logs, clearing references when they are no longer needed, and processing data in chunks or with generators rather than building complete intermediate collections. Measurement guides which fix matters, because a program may have several memory uses, only one of which dominates.

Memory optimisation, like time optimisation, should preserve correctness and readability, and should be verified by re-measurement. Reducing peak memory by processing in chunks, for example, changes a program's structure, so its output must be compared before and after to confirm it is unchanged. Reducing memory by reusing objects or by using more compact representations can make code harder to read, so it should only be done where measurement shows memory is actually a problem. For most programs, the simplest memory practices suffice: avoid unbounded growth, prefer generators for large streams, and release large objects when they are done. Those habits, combined with occasional memory profiling on realistic workloads, keep memory use under control without adding complexity where it is not needed.

Long-running programs deserve particular attention here, because memory growth that is invisible in a short test becomes obvious over hours or days, and a program that passes every short test can still fail in production after running long enough for its leak to exhaust available memory.

Check: Why can a program that is fast enough still fail because of memory use?

Section 42

Loop optimisation deserves careful attention because loops are where much of a program's work happens, and small per-iteration costs multiply by the number of iterations. The most effective loop optimisation is usually to reduce the number of iterations, or to move work out of the loop, rather than to micro-optimise the loop body itself. Work that does not change between iterations, such as computing a value that depends only on constants, should be computed once before the loop, not recomputed every time. Filtering that removes items cheaply should happen before expensive per-item work, as in the case study earlier. A loop that searches a list repeatedly for each item, turning a linear process into a quadratic one, should be restructured to use a set or dictionary for lookups, changing the overall complexity rather than merely speeding up each lookup. Those structural changes, guided by measurement, usually dwarf gains from micro-optimisation.

Within a loop body, simple practices help without harming readability. Avoiding repeated attribute lookups or repeated function calls for values that could be stored in a local variable once can help for very hot loops, but it should only be done where profiling shows the loop is actually a bottleneck, because it makes code slightly harder to read. Using built-in operations that process a whole collection at once, rather than an explicit loop that processes one item at a time in Python, is often faster, because the built-in operation's inner loop runs in optimised code rather than in interpreted Python. That is why, for example, joining strings, summing numbers, or filtering with a comprehension can be faster than an equivalent explicit loop, while also being shorter and clearer. The key is to recognise when a built-in or library operation expresses the whole loop's work, and to use it, rather than writing the loop by hand out of habit.

Finally, beware of loop optimisations that sacrifice correctness or clarity for tiny gains. Unrolling a loop by hand, or combining several loops into one to save iterations, can introduce bugs and make code much harder to follow, for a saving that measurement may show is negligible. The discipline from earlier sections applies: profile first, identify the hot loop, consider structural changes such as reducing iterations, moving invariant work out, or using a built-in operation, and then verify by re-measurement that the change actually helped on realistic data. A loop that is not a bottleneck should be left in its clearest form, because its readability matters more than its speed, and because future maintainers will thank you for code they can understand at a glance, even if it is not the fastest conceivable version.

That structural view also helps when reading other people's code, because a loop that looks harmless at first glance may hide a quadratic pattern, and spotting it early, before data grows, can prevent a performance incident that only appears when the program is already in production use.

Check: What is usually more effective than micro-optimising a loop body, and why?

Section 43

Web applications have performance characteristics that differ from scripts and batch programs, because they serve many requests, often concurrently, and because users experience performance as response time for each request. A web request typically involves several stages: receiving the request, reading or computing data, possibly querying a database or another service, rendering a response, and sending it back. Optimising a web application therefore means looking at all stages, not just computation, because a slow database query, a slow external service, or a large response can dominate response time even when the application's own code is fast. Profiling and timing per stage, as discussed for stage timing generally, reveals which stage matters for a given endpoint, and different endpoints may have different bottlenecks, so each important endpoint should be measured separately rather than assuming one optimisation helps them all.

Caching is especially valuable for web applications, because many requests repeat the same work, such as rendering a popular page or computing a result that changes slowly. Caching responses, or caching expensive computed values, can dramatically reduce response time and load, provided invalidation is handled correctly so users do not see stale data. Database query efficiency is another common web bottleneck: fetching more rows than needed, fetching the same data repeatedly for each item in a list, or missing an index on a frequently queried field can each make an endpoint slow, and fixing the query pattern often helps more than optimising application code. Response size also matters: a large response takes longer to send, especially over slow networks, so sending only needed data and compressing responses where appropriate improves perceived performance for users on slow connections.

Finally, web performance must be judged under load, not just for a single request in isolation. An endpoint that responds quickly when tested alone may slow dramatically when many users request it at once, due to contention for a database, for a shared resource, or for processor time, as discussed for scalability. Load testing with realistic concurrent users reveals that behaviour before production does, and it informs decisions about caching, query optimisation, and capacity. Monitoring in production, with response time percentiles and error rates per endpoint, then shows whether performance stays acceptable as usage grows and code changes. That combination, of per-endpoint measurement, targeted optimisation, load testing, and ongoing monitoring, is the web application version of the disciplined performance practice this chapter advocates, adapted to the request-and-response world where users judge performance by how quickly each page or result arrives.

Security and correctness still come first for web applications, of course, and a caching or query optimisation that serves stale data to the wrong user, or that exposes data it should not, is not an optimisation at all, however much it improves response times in a benchmark.

Check: Why should each important web endpoint be measured separately rather than optimised together?

Section 44

Database performance is a frequent bottleneck for data-driven programs, because a program that is fast in its own code can still be slow if it waits for database queries. The most common database performance problems are patterns, not individual slow queries in isolation. Fetching data one row at a time in a loop, where each iteration triggers a separate query, multiplies query overhead by the number of rows and is far slower than fetching needed data in fewer, larger queries. Fetching far more columns or rows than a program actually uses wastes database time, network time, and memory. Repeating the same query many times for the same data, instead of fetching it once and reusing it, wastes work that caching or restructuring could avoid. Each of these patterns is a structural problem, fixed by changing how a program talks to the database, rather than by micro-optimising the program's own code, and each is revealed by measuring query counts and query times, not just total program time.

Indexes are the database's main tool for making queries fast, and understanding them at a conceptual level helps Python developers write better data access code. An index lets a database find matching rows quickly, without scanning every row, much as an index in a book lets a reader find a topic without reading every page. Querying on an indexed field is therefore often much faster than querying on an unindexed field, especially for large tables. Indexes are not free, however: they use storage, and they slow down writes, because each index must be updated when data changes. Adding an index for every field would therefore be counterproductive. The right indexes are those on fields frequently used for filtering, joining, or sorting, judged by measuring actual query patterns on realistic data, not by guessing which fields might be queried.

Finally, database performance should be tested with realistic data volumes, because a query that is instant on a small test database with a few hundred rows may be slow on a production database with millions of rows, where scanning becomes expensive and where missing indexes matter far more. Test databases should therefore be populated with representative volumes, or performance should be checked in a staging environment with realistic data, before a data access change is deployed. That realistic testing, combined with query measurement, pattern fixes, and thoughtful indexing, gives a program database performance that scales with its data, rather than degrading sharply as data grows. It also illustrates a general lesson: when a program depends on an external system, performance work must include that system, not just the program's own code, because the slowest stage, wherever it lives, determines overall performance.

That realistic-volume testing also applies to indexes themselves, because an index that helps on a large table may be irrelevant on a small one, and only testing at representative scale shows whether an index actually improves the queries your program really runs.

Check: Why is fetching data one row at a time in a loop usually far slower than fetching it in fewer queries?

Section 45

Performance trade-offs are unavoidable, and good performance work means making them deliberately rather than by accident. The most common trade-off is speed against readability: an optimisation that makes code faster but much harder to understand should only be kept if measurement shows it matters for a genuine bottleneck, and it should be documented so future maintainers understand why it exists. Another is speed against memory, as with caching, where faster responses cost storage, or with precomputation, where faster later work costs memory now. A third is speed against correctness risk: a clever optimisation that changes results in edge cases is not an optimisation at all, and should be rejected even if it is faster on typical data, because correctness outranks speed. Naming these trade-offs explicitly, when deciding on an optimisation, helps a team agree on whether it is worth keeping, and helps future maintainers judge whether it is still worth its cost as requirements change.

Time against development effort is another trade-off that is easy to overlook. An optimisation that saves a small amount of user time but takes days to implement and test may not be worth it, compared with a feature or a bug fix that users need more urgently. Performance work should therefore be prioritised by impact: how many users are affected, how much time is saved, and how often the slow path runs, weighed against the effort and risk of the change. A slow path that runs once a day for one user matters less than a slightly slow path that runs thousands of times a day for many users, even if the first is slower in absolute terms. That impact-based prioritisation, grounded in measurement of real usage where possible, ensures performance effort goes where it helps most, rather than where it is most interesting technically, which is a common trap for enthusiastic optimisers who enjoy squeezing speed from a favourite function while a more impactful bottleneck elsewhere is ignored.

Finally, trade-offs should be revisited over time, because what was once a good trade can become a bad one as data, hardware, and requirements change. An optimisation that was essential when data was large and hardware was slow may become unnecessary overhead when hardware improves or when data patterns change. A cache that was valuable when a computation was expensive may become wasteful when the computation becomes cheap. Periodic review, prompted by regression tests, monitoring, or simply by revisiting documented optimisations during refactoring, keeps trade-offs current. Documenting why an optimisation exists, what it costs, and what measurement justified it makes that review possible, because a future maintainer can check whether the original justification still holds. That documentation habit turns performance trade-offs from hidden assumptions into explicit, revisable decisions, which is a mark of mature performance practice.

Check: Why should performance trade-offs be documented and revisited over time?

Section 46

Performance anti-patterns are recurring mistakes that make programs slow, and recognising them helps you avoid them before profiling is even needed. One is premature optimisation, already discussed, where code is made complex for speed before measurement shows a need, wasting effort and harming readability. Another is optimising the wrong thing, where a developer tunes a function they suspect is slow while the real bottleneck, revealed only by profiling, lies elsewhere, such as in input and output or in a database query. A third is repeated work, where the same computation, query, or file read is performed many times for the same data, instead of once with the result reused. A fourth is quadratic behaviour hidden in innocent-looking code, such as searching a list inside a loop, concatenating strings in a loop, or checking membership in a list repeatedly, each of which turns a linear task into a quadratic one as data grows. Learning to spot these patterns by reading code, as well as by profiling, makes you a faster debugger of performance problems, because you can often guess where to measure first, while still verifying your guess by measurement before changing code.

Another anti-pattern is ignoring data growth, where a program is tested only on small data and then fails on large data, because its complexity, its memory use, or its database pattern does not scale, as discussed for scalability. Related to it is ignoring worst-case inputs, where a program is fast on typical data but very slow on adversarial or unusual data, such as already sorted data for a naive sort, or a pattern that causes excessive backtracking in a regular expression. Testing with large and with unusual data, in addition to typical data, catches those failures early. A further anti-pattern is measuring once, on one machine, and treating the result as a universal truth, when timing varies with hardware, load, and data, and when a single run can be misleading. Repeated measurement, with realistic data and with variation reported, avoids that trap, as discussed for benchmarking.

Avoiding anti-patterns is largely a matter of habit: measure before optimising, think about complexity and data growth, look for repeated work and hidden quadratic patterns, test with realistic and with worst-case data, and report measurements with context. Those habits, practised consistently, prevent most performance problems from being introduced in the first place, which is far cheaper than fixing them after users complain. When a performance problem does appear, the same habits guide its diagnosis: profile to find the bottleneck, check for a known anti-pattern in that area, fix the pattern rather than micro-optimising around it, and verify by re-measurement. That cycle, of prevention through habit and diagnosis through measurement, is more reliable than relying on optimisation skill alone, because it catches problems systematically, rather than depending on individual intuition, which, as this chapter has shown repeatedly, is often wrong about where time actually goes.

Check: What is a hidden quadratic anti-pattern, and why does it only become obvious as data grows?

Section 47

If you are new to performance work, a simple first-steps approach will take you a long way without needing advanced tools. Start by defining what fast enough means for your program: a response time users will accept, a total run time that fits its schedule, or a throughput that handles expected load, as discussed for performance requirements. Without that definition, you cannot know when to stop optimising, and you risk either stopping too early, leaving users with a slow program, or optimising indefinitely, wasting effort on speed nobody needs. Next, measure your program on realistic data, timing the whole program and its major stages, using simple timing around each stage. That alone often reveals the bottleneck, because one stage usually dominates, and it gives you a baseline to compare later changes against, which is essential for knowing whether a change actually helped.

Then, look at the dominant stage and ask three questions: is it doing unnecessary work that could be removed, is it using an algorithm or data structure with poor scaling for your data size, and is it repeating work that could be done once and reused? Those three questions cover most performance problems, from filtering after expensive conversion, to searching a list inside a loop, to recomputing a value for every item. Fix the answer that measurement shows matters most, one change at a time, and re-measure after each change to confirm it helped and did not break correctness, by comparing output before and after on the same data. If a change does not help, revert it, because keeping a change that does not help only adds complexity. That loop, of measure, ask, change, and verify, is the core of performance work, and it requires no advanced tools, only timing, realistic data, and discipline.

Finally, keep your changes readable and documented, especially if you make a change whose reason is not obvious from the code alone. Add a short comment explaining what measurement showed and why the change was made, so a future reader understands that the less obvious version exists for a measured reason, not by accident. That documentation also helps you revisit the change later, if data or requirements change. As you gain experience, you can add profilers, benchmarks, and regression tests to this basic workflow, as described elsewhere in this chapter, but the basic workflow itself remains the foundation: define fast enough, measure on realistic data, fix the dominant bottleneck by removing unnecessary work or improving scaling, verify by re-measurement, and stop when requirements are met. Mastering that foundation will make every advanced tool you later learn more useful, because you will already know what to measure and why.

If your first measurement surprises you, and it often will, treat that surprise as valuable information, because it shows that your mental model of the program was wrong, and correcting that model will make your next performance judgement, and your next program design, noticeably better.

Check: What three questions should you ask about the dominant stage when starting performance work?

Section 48

Performance tools, at a conceptual level, fall into several kinds, each answering a different question, and knowing which tool answers which question helps you choose the right one. Timers answer how long something takes, for a whole program or for a stage, and they are the simplest and most widely used tool. Profilers answer where time goes within a program, by function or by line, and they are the right tool when you know a program is slow but not which part is responsible. Benchmarking tools answer how fast a small piece of code is, in isolation, with repeated runs and statistical summaries, and they are the right tool for comparing two implementations of the same small task. Memory profilers answer where memory goes and whether it grows over time, as discussed earlier. Load testing tools answer how a program behaves under concurrent load, by simulating many users or requests and measuring response times and throughput as load increases. Each tool has a purpose, and using the wrong tool, such as benchmarking a small function when the real problem is a slow database query, wastes time and can mislead.

Choosing a tool also involves practical considerations: how easy it is to use, how much it slows the program while measuring, and how trustworthy its results are. A profiler that slows a program dramatically may distort which parts appear slow, though it usually still identifies the right bottleneck in relative terms, which should then be confirmed with simpler timing. A benchmark that runs on a noisy shared machine may give variable results, which should be handled with repeated runs and typical values, as discussed for benchmarking. A load test that uses unrealistic user behaviour may give misleading capacity estimates, which should be avoided by modelling realistic behaviour as closely as possible. Those practical limits do not make tools useless, but they mean results should be interpreted with judgement, cross-checked with another method where possible, and reported with context, as discussed for communication.

Finally, tools are aids, not substitutes for thinking. A profiler can tell you where time goes, but it cannot tell you whether that work is necessary, whether a better algorithm exists, or whether a restructuring would remove the work entirely, and those judgements require understanding the program's purpose and its data, which only a developer can provide. Similarly, a benchmark can tell you which of two implementations is faster, but it cannot tell you whether either is fast enough, or whether the code being benchmarked is even on the hot path. Use tools to inform decisions, not to make them for you, and combine tool results with reasoning about complexity, about data growth, and about user requirements, as this chapter has emphasised throughout. That combination, of tool evidence and developer judgement, is what makes performance work reliable, rather than a mechanical exercise in running tools and applying their suggestions uncritically, which can lead to optimisations that look good in a tool's output but do not help real users, because they address a cold spot or a test-only workload, rather than a genuine bottleneck in real use.

Check: Which tool answers where time goes within a program, and which answers how a small piece of code performs in isolation?

Section 49

Performance myths persist because they sound plausible, and recognising them protects you from wasted effort. One myth is that a faster language or a faster machine will fix a slow program, when in fact a program with quadratic complexity or with repeated database queries will still be slow, just slightly less slow, on faster hardware, because its scaling problem remains, and it will become slow again as data grows. Another myth is that micro-optimisation, such as tweaking a single line, will transform performance, when most programs have a bottleneck elsewhere, and micro-optimising a cold spot, as discussed earlier, yields negligible overall gain. A third myth is that caching always helps, when an unbounded or poorly invalidated cache can waste memory, serve stale data, or add complexity that outweighs its benefit, especially for data that is rarely reused. A fourth myth is that more concurrency always means more speed, when concurrency adds overhead, contention, and complexity, and when it only helps if work is actually parallelisable and if shared resources are not the real bottleneck, as discussed for concurrency concepts.

Another myth is that performance and readability are always opposed, when many of the most effective optimisations, such as using a built-in operation, choosing a better data structure, or removing unnecessary work, also make code clearer, because they express intent more directly. A related myth is that optimisation should be left until the end, when in fact algorithm and data structure choices, made early, determine scaling, and changing them late can require major refactoring, while micro-optimisation can safely be left until measurement shows a need. The distinction is between early design for scaling, which is wise, and early micro-optimisation, which is wasteful, and confusing the two leads either to premature micro-optimisation or to neglected scaling design, both of which are harmful. A final myth is that a single benchmark proves a program is fast, when a benchmark on one machine, with one dataset, and with one workload cannot prove performance in general, and when realistic, repeated, and varied testing is needed to support a performance claim, as discussed for benchmarking and for communication.

Debunking myths matters because myths drive decisions: a team that believes faster hardware will fix scaling will buy hardware instead of fixing code, a team that believes caching always helps will add caches everywhere instead of fixing query patterns, and a team that believes micro-optimisation transforms performance will spend weeks tuning cold spots while users wait on a real bottleneck. Replacing myths with measurement, with complexity thinking, and with the disciplined workflow of this chapter leads to better decisions, because it grounds performance work in evidence about a specific program, its data, and its users, rather than in general slogans that may be true in some cases but false in others. When you hear a performance claim, ask what was measured, on what data, and whether the bottleneck it addresses is a genuine one for your program, and let the answers, not the slogan, decide whether to act.

Check: Why will faster hardware not fix a program whose real problem is quadratic scaling?

Section 50

Putting it all together, a performance plan for a new program or for improving an existing one can be stated as a short sequence that draws on everything in this chapter. First, define requirements in user terms: how fast must it be, for whom, and under what load, so that you know when performance work is done. Second, design for scaling from the start by choosing algorithms and data structures whose complexity fits expected data growth, and by avoiding known anti-patterns such as hidden quadratic behaviour, repeated work, and unbounded growth. Third, build the clearest correct version first, without micro-optimisation, because a clear version is easier to test, easier to profile, and easier to optimise later in the right places, once measurement shows where they are. Fourth, measure on realistic data, with stage timing and, where needed, profiling, to find genuine bottlenecks, and record baselines so later changes can be judged.

Fifth, optimise bottlenecks one at a time, starting with restructuring to remove unnecessary work, then with algorithm or data structure improvements, then with caching or batching where appropriate, and only last with micro-optimisation of hot code, verifying each change by re-measurement and by output comparison to protect correctness. Sixth, test under load and with large and unusual data, to catch scalability and worst-case problems that typical small tests miss, and set regression tests or monitoring so future changes that slow the program are detected early. Seventh, document optimisations and report results with context, variation, and caveats, so colleagues can understand, verify, and revisit them as conditions change. That seven-step plan is not a rigid ritual but a checklist to adapt to each program's size and purpose: a small script may need only the first few steps, while a large system will need them all, with more formality.

Finally, remember that performance is a means, not an end. A program exists to serve users, and performance matters because it affects their experience, their time, and their trust. A program that is fast but incorrect, insecure, or unmaintainable has failed, however impressive its timings, just as a program that is correct but unusably slow has failed its users in a different way. The goal of this chapter has been to help you hold both truths together: to write Python programs that are correct first, and then fast enough, through measurement, through thoughtful design, and through disciplined, verified optimisation, without sacrificing readability or reliability along the way. Carry that balance into your next programs, measure before you optimise, and let evidence, not intuition, guide your performance decisions, and you will build programs that users experience as responsive, reliable, and trustworthy, which is the real reward of performance work well done.

Keep this plan somewhere you can find it again, such as in your own notes or in a team document, and return to it whenever a program feels slow, because having a plan to follow prevents the most common performance mistake of all, which is guessing, changing code at random, and hoping the result is faster.

Check: What are the main steps of the performance plan summarised in this section?

Practical Task

Take a small Python script you have written, or write a short script that processes a realistic amount of data, such as reading a CSV file with several thousand rows, filtering it, and producing a summary. Time the whole script and time its major stages separately, using simple timing, on realistic data. Identify the slowest stage, profile it if you can, and try one restructuring change, such as filtering before an expensive conversion or replacing a list search inside a loop with a set lookup. Re-measure after the change, compare the output before and after to confirm it is unchanged, and write a short note stating what you measured, what you changed, how much it helped, and whether the result now meets a performance goal you define in advance. If the change does not help, revert it and note why, as that is also a valid performance finding.

Chapter Questions

1. Why should performance work start with measurement rather than with intuition about which part of a program is slow?
2. How does choosing a data structure with the wrong scaling behaviour affect a program as its data grows?
3. When is caching a good optimisation, and what risks do unbounded or poorly invalidated caches introduce?
4. Why is re-measurement after each optimisation change essential, and what should be compared besides speed?
5. How do you decide when performance work is finished for a program, rather than optimising indefinitely?