Chapter 34: Concurrency Concepts
Section 1
Concurrency is the idea of making progress on more than one task at the same time, or at least of organising work so that tasks that are waiting do not block tasks that could be running. A program that downloads several files one after another, waiting for each download to finish before starting the next, is sequential. A concurrent program might start several downloads and make progress on whichever one is ready, so that waiting time is used productively. That distinction, between doing tasks one after another and interleaving them, is the foundation of concurrency, and it appears in many kinds of programs, from web servers to data tools.
It is important to distinguish concurrency from parallelism, although the two are related and often confused. Concurrency is about structure: a program is concurrent if it is organised to handle multiple tasks, even if only one task is actually running at any instant, with the program switching between tasks when one waits. Parallelism is about execution: a program is parallel if multiple tasks are actually running at the same time, on different processors or cores. A concurrent program may or may not be parallel, depending on how it is run, and a parallel program is usually concurrent in structure as well. Keeping those definitions separate helps you reason about what a particular approach can and cannot achieve.
Concurrency is useful mainly when tasks spend time waiting. Waiting happens when a program reads from a disk, downloads from a network, waits for a user, or waits for another program to respond. During that waiting time, the processor is idle, at least with respect to that task, and another task could use it. A program that interleaves waiting tasks can therefore finish sooner, in wall-clock time, than a sequential program, even if the total amount of work is the same, because it wastes less time waiting. That benefit is most dramatic for tasks that wait a lot, such as network operations, and least dramatic for tasks that are constantly computing, where there is little waiting time to exploit.
Concurrency also introduces complexity. When tasks interleave, their order of execution becomes less predictable, and they may interfere with each other if they share data or resources. A program that was correct when run sequentially may become incorrect when run concurrently, because two tasks may update the same value at the same time, or one task may read data while another is halfway through changing it. Managing that interference, through careful design and through synchronisation mechanisms, is the central challenge of concurrency, and it is why concurrent programs require more thought than sequential ones.
Finally, concurrency is not always the right solution. For a simple task, or for a task that is already fast enough, adding concurrency may add complexity without meaningful benefit, and it may introduce bugs that are hard to find. The decision to use concurrency should be based on a real need, such as handling many waiting tasks or responding to users while doing background work, and on an understanding of the costs, in complexity and in debugging difficulty. That judgement, about when concurrency helps and when it hurts, is the most important concurrency skill, and it is the theme of this chapter.
Check: What is the difference between concurrency and parallelism?
Section 2
Sequential programs are easy to reason about, because they do one thing at a time, in a predictable order. Each step finishes before the next begins, so the state of the program at any point is determined by the steps that have already run. That predictability makes sequential programs easier to write, test, and debug, and it is why most programs start sequential. When a sequential program is correct, its correctness is usually clear, because its execution follows a single, understandable path from start to finish.
The limitation of sequential programs appears when they must wait. A program that downloads a file, for example, spends most of its time waiting for the network, doing nothing useful, while the processor sits idle. If it must download ten files, it will wait ten times in succession, and its total time will be roughly the sum of all waiting times. That is wasteful, especially when the waiting tasks are independent, meaning that one download does not depend on another's result. Independent waiting tasks are the classic case where concurrency helps, because they can be interleaved so that waiting time for one is used to progress another.
Another limitation appears in interactive programs. A program that must respond to user input, while also performing a long task, cannot do both sequentially without freezing. If it starts the long task and does not return to check for input until the task finishes, the user will experience the program as unresponsive, even though it is working correctly. A concurrent design, where the long task runs in the background while the interface remains responsive, solves that problem, by separating the work that must be responsive from the work that can take time. That separation is a common reason to introduce concurrency, even when total speed is not the main concern.
Sequential programs also have a simplicity advantage that should not be underestimated. Adding concurrency to solve a waiting problem may be unnecessary if the waiting problem can be solved more simply, such as by batching requests, by caching results, or by optimising the sequential code. Before introducing concurrency, it is worth asking whether a simpler change would achieve most of the benefit with less complexity. That question, applied consistently, prevents concurrency from being added where it is not needed, and it keeps programs as simple as their requirements allow.
Finally, many programs are partly sequential and partly concurrent, with a sequential core for most work and concurrent handling only where waiting or responsiveness demands it. That hybrid approach is often the most practical, because it confines complexity to the parts of the program where it is needed, while keeping the rest simple and predictable. Designing a program with that hybrid structure, rather than making everything concurrent, is a sign of mature judgement about concurrency, and it is a pattern you will see in many real systems.
Check: Why does a sequential program that downloads ten files waste time compared with a concurrent one?
Section 3
Threads are one way to achieve concurrency in Python, and they are best understood as separate paths of execution within a single program. A program with multiple threads has multiple sequences of steps running, interleaved by the system, sharing the same memory and the same variables. That sharing makes threads convenient for tasks that need to communicate or share data, because they can simply read and write shared variables, without needing to pass messages between separate processes. It also makes threads risky, because shared data can be changed by one thread while another is reading it, leading to inconsistent results if care is not taken.
Threads are well suited to tasks that spend time waiting, such as network operations, file operations, or waiting for user input. When one thread waits, another thread can run, using the processor productively, as discussed earlier. That is why threads are commonly used for programs that handle many network connections, or that download many files, where waiting dominates. For tasks that are constantly computing, threads offer less benefit in Python, for reasons related to how Python executes code, which will be discussed later in this chapter, but they can still be useful for structuring a program, even if they do not speed up computation.
Creating and managing threads involves deciding how many threads to use, what each thread should do, and how they should communicate and synchronise. Too few threads may not exploit waiting time fully, while too many threads can overwhelm a system, by consuming memory and by spending more time switching between threads than doing useful work. A common pattern is to use a pool of worker threads, with a fixed number, that take tasks from a queue, process them, and return results. That pattern, which will be discussed in more detail later, balances concurrency against resource use, and it is a standard way to structure threaded programs.
Threads also require care with shared state. If two threads update the same counter, or append to the same list, without synchronisation, the result may be incorrect, because the updates can interleave in unexpected ways. That problem, known as a race condition, is central to threaded programming, and it is discussed in detail later in this chapter. Avoiding race conditions requires either avoiding shared mutable state, where possible, or protecting it with synchronisation mechanisms, such as locks, which ensure that only one thread accesses shared data at a time. That protection adds complexity, but it is necessary for correctness.
Finally, threads in Python are a concept-level tool for this chapter, not a full tutorial. You should understand what threads are, when they help, and what risks they bring, and you should be able to recognise a threaded design and its potential problems. Writing complex threaded programs requires further study, including hands-on practice with small, controlled examples, and an understanding of the specific threading tools Python provides. This chapter gives you the conceptual foundation for that further study, by explaining the ideas that underlie all threaded programming, whatever the language or library.
Check: Why are threads well suited to tasks that spend time waiting?
Section 4
Processes are another way to achieve concurrency, and they differ from threads in a fundamental way: each process has its own memory, separate from other processes, and it runs independently. A program that uses multiple processes is really several programs running at once, each with its own variables and state, communicating only through explicit channels, such as pipes, files, or messages. That separation makes processes safer in one respect, because one process cannot accidentally corrupt another's memory by writing to a shared variable, but it also makes communication more explicit and sometimes more cumbersome, because data must be sent between processes rather than simply shared.
Processes are well suited to tasks that are computationally intensive, because each process can run on a different processor core, in parallel, without the limitations that affect threads in Python for computation, which will be discussed later. A program that processes many independent items, such as converting many images or analysing many files, can divide the items among processes, each of which works on its share, and then combine the results. That division, known as parallel processing, can reduce total time significantly for compute-heavy work, provided that the work of dividing, communicating, and combining does not outweigh the benefit of parallelism.
Processes also offer isolation as a benefit. If one process crashes, due to a bug or a bad input, other processes may continue, and the main program can detect the failure and respond, such as by retrying the failed item or reporting the error. That isolation makes multi-process programs more robust to failures in individual tasks, compared with a threaded program, where a crash in one thread can bring down the whole program. Isolation is particularly valuable when processing untrusted or unpredictable data, where a failure in one item should not affect others.
The costs of processes include higher overhead and more complex communication. Starting a process takes more time and memory than starting a thread, because a new process needs its own memory space and its own interpreter state. Communicating between processes also takes more effort, because data must be serialised, sent, and deserialised, rather than simply shared. For small tasks, or for tasks that need frequent, fine-grained communication, that overhead may outweigh the benefit of parallelism, and threads or a sequential approach may be better. Choosing between processes and threads therefore involves weighing isolation and parallelism against overhead and communication complexity.
Finally, like threads, processes are covered here at concept level. You should understand what processes are, how they differ from threads, when they are appropriate, and what their costs are. Writing multi-process programs requires further study of Python's process tools, including how to start processes, how to communicate between them, and how to handle their completion and failure. The conceptual foundation in this chapter, about independence, isolation, and communication, will help you understand those tools when you meet them, and will help you decide when processes are the right choice for a task.
Check: How does a process differ from a thread in terms of memory sharing?
Section 5
Asynchronous programming, often called asyncio in Python, is a third approach to concurrency, based on a single thread that switches between tasks cooperatively, at points where a task waits. Instead of the system switching between threads at arbitrary moments, an asynchronous program is written with tasks that explicitly pause, or await, when they wait, allowing another task to run until it too waits. That cooperative switching, within a single thread, avoids many of the risks of threads, because a task cannot be interrupted in the middle of an operation, except at the explicit pause points it defines. That predictability makes asynchronous programs easier to reason about, in some respects, than threaded programs.
Asynchronous programming is well suited to programs that handle many waiting tasks, such as a web server handling many simultaneous connections, or a tool that fetches data from many network sources. In those programs, the number of tasks may be large, and using a thread per task would consume significant memory and switching overhead, while using a single asynchronous thread with many tasks is more efficient, because tasks are lightweight and switching is cheap. That efficiency, for high numbers of waiting tasks, is the main advantage of asynchronous programming, and it is why it is widely used for network servers and clients.
Writing asynchronous code requires a different style from sequential code, because tasks must be written to pause at waiting points, using a special syntax, and the program must be run by an event loop, which is the component that schedules tasks and resumes them when their waiting ends. That style takes some learning, and it can be confusing at first, because a program that looks sequential, with pauses, is actually concurrent, with many tasks interleaved by the loop. Understanding the event loop conceptually, as a scheduler that runs tasks until they wait and then switches to another ready task, is the key to understanding asynchronous programming, even without mastering its full syntax yet.
Asynchronous programming is not a full tutorial topic in this chapter, by design. This chapter covers it at concept level only: what it is, when it helps, how it differs from threads and processes, and what its event loop does. A full asynchronous tutorial would cover writing asynchronous functions, awaiting operations, creating and gathering tasks, handling timeouts and cancellation, and integrating asynchronous code with synchronous code, among other topics. Those topics are substantial, and they deserve dedicated study after the concepts here are clear. Rushing into asynchronous syntax without understanding its concepts leads to confusion, because the syntax only makes sense in light of the event loop model.
Finally, asynchronous programming, like threads and processes, is a tool with trade-offs. It excels for many waiting tasks, but it is less suited to computationally intensive tasks, because a single thread cannot parallelise computation, and a long computation that never pauses will block the event loop, preventing other tasks from running. Recognising that limitation, and choosing asynchronous programming for waiting-heavy work while choosing processes for compute-heavy work, is part of the judgement this chapter aims to develop. That judgement, about matching the concurrency model to the task, matters more than mastering any single model's syntax in isolation.
Check: How does cooperative switching in asynchronous programming differ from thread switching?
Section 6
The Global Interpreter Lock, often abbreviated as GIL, is a feature of the standard Python implementation that affects how threads behave, and understanding it conceptually is important for choosing between threads and processes. In simple terms, the GIL is a lock that ensures only one thread executes Python code at a time, even on a computer with multiple processor cores. That means that, for pure Python computation, multiple threads do not run in parallel; they take turns executing, with the GIL passed between them. That turn-taking still allows threads to be useful for waiting tasks, because a thread that waits releases the GIL, allowing another thread to run, but it limits their benefit for computation, where threads are constantly trying to execute and must wait for the GIL.
The GIL exists for reasons related to memory management and to the simplicity of Python's implementation, and discussing those reasons in detail is beyond this chapter, but the practical consequence is what matters for concurrency decisions. If a task is waiting-heavy, such as downloading files or handling network connections, threads are effective, because waiting threads do not need the GIL, and the GIL is available to threads that are ready to work. If a task is compute-heavy, such as performing many calculations in Python, threads offer little speedup, because only one thread can compute at a time, and the overhead of switching between threads may even slow the program down compared with a sequential version.
Processes do not share the GIL, because each process has its own interpreter and its own lock, so multiple processes can compute in parallel on different cores. That is why processes are the recommended approach for compute-heavy parallelism in Python, as discussed earlier. Asynchronous programs, which run in a single thread, also do not parallelise computation, for the same reason: a single thread, with or without an event loop, can only execute one piece of Python code at a time. Understanding the GIL therefore clarifies why the three concurrency models differ in their suitability for computation versus waiting, and it prevents the common misconception that adding threads will automatically speed up any program.
It is important not to overstate the GIL's impact. Many real programs are waiting-heavy, or they spend time in operations that release the GIL, such as certain library operations that are implemented outside Python, and for those programs threads work well. The GIL is not a reason to avoid threads entirely; it is a reason to understand what kind of task a thread is doing, and whether that task will benefit from threading. A threaded program that downloads files will usually be faster than a sequential one, despite the GIL, because downloading is waiting, not computation. A threaded program that performs pure Python calculations will usually not be faster, and that distinction should guide design.
Finally, the GIL is specific to the standard Python implementation, and other implementations may handle threading differently, but for the purposes of this book, which focuses on standard Python, the GIL's practical consequences as described here are the relevant ones. That lesson, more than any specific detail about the lock itself, is what this section aims to teach.
Check: Why do threads help waiting-heavy tasks despite the GIL, but not compute-heavy tasks?
Section 7
A race condition occurs when the correctness of a program depends on the timing or ordering of concurrent tasks, and different orderings produce different, sometimes incorrect, results. The classic example is two tasks updating the same counter: each task reads the counter, adds one, and writes it back. If both tasks read the same value before either writes, they will both write the same incremented value, and the counter will increase by one instead of two, even though two increments occurred. That lost update is a race condition, because the result depends on the race between the tasks to read and write, and the correct result requires a specific ordering that is not guaranteed.
Race conditions are insidious because they may not appear every time a program runs. A program with a race condition may produce correct results most of the time, when tasks happen to interleave favourably, and incorrect results only occasionally, when they interleave unfavourably. That intermittency makes race conditions hard to reproduce, hard to test for, and hard to debug, because a test that passes today may fail tomorrow, with no change to the code. A program that is usually correct but occasionally wrong is, in practice, incorrect, because its users cannot trust its output, and its failures may occur at the worst possible moment, such as under heavy load when timing changes.
Avoiding race conditions starts with design: minimise shared mutable state, where possible, so that tasks do not interfere with each other. If each task works on its own data, and results are combined only at the end, there is no shared state to race over, and the program is safe by construction. That design, sometimes called share-nothing, is the simplest way to avoid races, and it should be preferred where it fits the task. For example, a program that processes independent files can give each task its own file and its own result, combining results only after all tasks finish, with no shared counter or list being updated concurrently.
When shared state cannot be avoided, it must be protected with synchronisation, such as a lock, which ensures that only one task accesses the shared data at a time, for a complete operation. A lock turns a read-modify-write sequence into an atomic operation, meaning that it cannot be interleaved with another task's access, so the lost update problem disappears. Using locks correctly requires care: every access to the shared data must use the same lock, the lock must be held for the whole operation, not just part of it, and locks must be released reliably, even if an error occurs. Those requirements, while straightforward in principle, are easy to get wrong in practice, especially as programs grow.
Finally, race conditions can occur not only with threads, but also with processes that share files or other resources, and with asynchronous tasks that share data, although the mechanisms differ. In all cases, the underlying problem is the same: concurrent access to shared, mutable state without proper coordination. Recognising that problem, designing to minimise shared state, and protecting what remains, are the core skills for writing correct concurrent programs, whatever the concurrency model. A program that is correct under concurrency is one where shared state has been considered explicitly, not one where it was left to chance.
Check: Why is a race condition hard to reproduce and test for?
Section 8
Locks are the most basic synchronisation mechanism for protecting shared data in concurrent programs. A lock is like a key to a room: only the task holding the key can enter, and other tasks must wait outside until the key is returned. In code, a task acquires a lock before accessing shared data, performs its operation, and then releases the lock, allowing another task to proceed. That simple protocol ensures that operations on shared data are atomic, meaning that they complete without interleaving, which prevents race conditions for the protected data. Locks are a foundational concept, and understanding them helps you understand more advanced synchronisation mechanisms, which build on similar ideas.
Using locks correctly requires discipline. Every access to a particular piece of shared data must use the same lock, because a lock only protects data if all tasks agree to use it. If one task uses the lock and another accesses the data directly, without the lock, the protection is ineffective, and a race can still occur. That agreement must be maintained as code evolves, which is why shared data and its lock should be kept together, ideally encapsulated in a single object or module, so that a future maintainer cannot easily access the data without the lock. Encapsulation turns a convention into a structure that is harder to violate accidentally.
Locks also introduce a risk of deadlock, which occurs when two or more tasks wait for each other forever, each holding a lock that the other needs. For example, one task holds lock A and waits for lock B, while another holds lock B and waits for lock A, and neither can proceed. Deadlock is a serious problem, because the affected tasks stop making progress, and the program may appear hung, without an obvious error. Avoiding deadlock requires careful design, such as always acquiring locks in the same order, or avoiding holding multiple locks at once where possible. Those rules, while simple to state, require discipline to follow as programs grow more complex.
Another consideration is lock granularity: how much data a single lock protects, and how long it is held. A coarse lock, protecting a large amount of data for a long operation, is simple and safe, but it reduces concurrency, because tasks must wait even when they access different parts of the data. A fine-grained lock, protecting small pieces of data for short operations, allows more concurrency, but it is more complex and more prone to deadlock, because more locks are involved. Choosing granularity involves balancing safety, simplicity, and performance, and for most programs, starting with coarse locks and refining only if performance demands it is a sensible approach.
Finally, locks are not the only synchronisation mechanism, and they are not always the best one. Queues, discussed next, allow tasks to communicate without directly sharing data, which can avoid locks altogether. Other mechanisms, such as semaphores, events, and conditions, provide different ways to coordinate tasks, each suited to particular patterns. Understanding locks first, as the simplest mechanism, gives you a foundation for understanding those others, and it teaches the central lesson of synchronisation: shared data must be accessed in a coordinated way, or correctness cannot be guaranteed.
Check: What is deadlock, and how can acquiring locks in a consistent order help avoid it?
Section 9
Queues are a synchronisation and communication mechanism that allow concurrent tasks to pass work and results to each other safely, without directly sharing mutable data. A queue is a first-in, first-out structure: tasks put items into the queue, and other tasks take items out, in order, with the queue handling synchronisation internally, so that puts and takes are safe even when performed concurrently. That built-in safety makes queues a preferred way to structure concurrent programs, because they replace ad-hoc shared data and locks with a well-defined, tested mechanism for communication.
A common pattern using queues is the producer-consumer pattern. Producer tasks generate work items, such as files to process or URLs to fetch, and put them into a queue. Consumer tasks, often a pool of workers, take items from the queue, process them, and put results into another queue, or store them in a safe way. The queue decouples producers from consumers: producers do not need to know how many consumers there are, or how fast they work, and consumers do not need to know where work comes from. That decoupling makes the program easier to understand and to scale, because adding more consumers, up to a point, increases throughput without changing the producer logic.
Queues also provide backpressure, which is a way to prevent producers from overwhelming consumers. A queue with a maximum size will block a producer that tries to put an item into a full queue, until a consumer takes an item and frees space. That blocking prevents unbounded growth of queued work, which could consume memory and overwhelm the system, and it naturally balances production and consumption rates. Without backpressure, a fast producer and a slow consumer could cause a program to accumulate a huge backlog, leading to memory problems or to stale work that is no longer relevant by the time it is processed.
Signalling completion is another aspect of queue-based design. Consumers need to know when no more work will arrive, so that they can finish, rather than waiting forever for another item. That signalling is often done with a special sentinel item, a value that means end of work, placed into the queue after all real work items, with one sentinel per consumer, so that each consumer receives the signal and can exit cleanly. Designing that shutdown protocol carefully, and testing it, is part of building a correct queue-based program, because a program that never shuts down, or that shuts down while work remains, is incorrect even if its processing logic is sound.
Finally, queues illustrate a broader design principle for concurrency: prefer communication over shared state. Instead of tasks writing to a shared list, with locks to protect it, they can send results through a queue, where the queue handles synchronisation, and a single task collects results in order. That principle, of structuring concurrent programs around messages and queues rather than around shared mutable data, reduces race conditions, simplifies reasoning, and makes programs easier to test, because each task's behaviour can be understood in terms of what it receives and what it sends, rather than in terms of global state changes.
Check: How does a bounded queue provide backpressure between producers and consumers?
Section 10
Shared state is data that multiple concurrent tasks can access, and managing it safely is the central challenge of concurrent programming, as discussed throughout this chapter. Shared state can take many forms: a variable, a list, a dictionary, a file, a database, or any resource that tasks both read and write. The danger arises not from sharing itself, but from mutable sharing, where tasks change shared data while others are reading or changing it, leading to race conditions, inconsistent reads, and lost updates, as discussed earlier. Understanding the forms shared state takes, and the strategies for managing it, is essential for designing correct concurrent programs.
One strategy is to avoid shared mutable state altogether, by giving each task its own copy of data, or by making data immutable, meaning that once created, it cannot be changed, only replaced with a new version. Immutable data is safe to share, because tasks can read it without fear that another task will change it underneath them. That safety is why functional programming styles, which favour immutability, are often recommended for concurrent programs, even in languages like Python that do not enforce immutability. Designing with immutability in mind, where practical, simplifies concurrent reasoning dramatically, because shared data becomes read-only, and read-only sharing is safe.
Another strategy is to confine shared state to a single task. In that design, only one task, often called an owner or a manager, is allowed to access a particular piece of data, and other tasks communicate with that owner through queues or messages, asking it to perform operations on their behalf. That confinement ensures that all accesses to the data are sequential, because they all go through a single task, even though the program as a whole is concurrent. That pattern combines the benefits of concurrency, in overall throughput, with the simplicity of sequential access to shared data, and it is a powerful design for programs with complex shared state.
A third strategy is to protect shared state with synchronisation, as discussed for locks, where tasks coordinate their access so that operations are atomic and consistent. That strategy is necessary when shared mutable state cannot be avoided, such as when tasks must update a common counter or a shared cache, but it should be used with care, because synchronisation adds complexity and deadlock risk, as discussed earlier. Keeping protected state small, well-encapsulated, and accessed through clear operations, rather than allowing arbitrary access from many places, makes synchronisation manageable and less error-prone.
Finally, shared state includes external resources, not just variables in memory. A file that multiple tasks write to, a network connection shared by threads, or a database accessed concurrently, are all shared state, and they require similar care: either access must be serialised through a single owner, or it must be protected with appropriate synchronisation, or the resource itself must provide safe concurrent access. Recognising external resources as shared state, and designing their access accordingly, prevents a class of bugs where a program is correct in memory but incorrect in its interaction with the outside world, which is just as important for real programs.
Check: Why is immutable data safe to share between concurrent tasks?
Section 11
A worker pool is a fixed group of threads or processes that take tasks from a queue and execute them, and it is a standard way to manage concurrency without creating an unbounded number of workers. Instead of starting a new thread for every task, which could overwhelm the system if there are thousands of tasks, a pool reuses a limited number of workers, each of which processes tasks one after another from a shared queue. That reuse limits resource use, because only the pool size number of workers exist at any time, while still allowing many tasks to be processed concurrently, up to the pool's capacity.
Choosing a pool size involves balancing throughput against resources. A pool that is too small may not exploit waiting time fully, because workers spend time waiting while other tasks queue up, even though the system could handle more concurrent work. A pool that is too large may waste memory, increase switching overhead, and overwhelm external resources, such as a server that receives too many simultaneous requests. The right size depends on the task: waiting-heavy tasks can support larger pools, because workers spend most time waiting, while compute-heavy tasks benefit from a pool size close to the number of processor cores, because more workers than cores cannot compute in parallel.
Pools also simplify shutdown and error handling. When all tasks have been submitted, the pool can be told to finish existing tasks and then shut down, waiting for workers to complete before the program exits. That orderly shutdown ensures that no task is lost because the program exited while workers were still running, which is a common bug in ad-hoc threaded programs that start threads and then exit without waiting. A pool's shutdown protocol, with a clear signal that no more tasks will arrive and a wait for completion, turns a fragile pattern into a reliable one.
Error handling in pools deserves attention. If a worker task raises an exception, that exception should be captured and reported to the task's submitter, rather than crashing the worker or being lost silently. Pool designs typically store a task's result or exception, allowing the submitter to retrieve it later, when the task completes. That retrieval pattern, where a submitted task returns a handle that can be checked for completion and result, is a useful abstraction, because it separates submitting work from collecting results, allowing a program to submit many tasks and then collect their results in order, or as they complete.
Finally, pools illustrate a general principle: concurrency should be managed, not unbounded. Creating workers on demand, without limit, is simple to code but risky in production, because a burst of tasks can exhaust resources. A bounded pool, with a queue for waiting tasks and backpressure when the queue fills, as discussed earlier, provides that management, and it is a pattern that applies to threads, processes, and asynchronous tasks alike. Managed concurrency, with clear limits, is more robust than unmanaged concurrency, even if it requires a little more setup.
Check: Why is a fixed-size worker pool safer than starting a new thread for every task?
Section 12
A future is a handle representing a task that may not have finished yet, allowing a program to submit work, continue with other work, and later check whether the task is done and retrieve its result. Futures abstract away the details of how a task is executed, whether by a thread, a process, or an asynchronous task, and they provide a uniform way to ask about completion, to wait for a result, and to handle exceptions. That abstraction is valuable, because it lets a program express concurrency in terms of tasks and results, rather than in terms of low-level thread or process management, which is more error-prone.
Submitting a task to a pool typically returns a future immediately, without waiting for the task to finish. The submitting program can then submit more tasks, do other work, and later collect results, either by waiting for a specific future, or by waiting for any future to complete, or by waiting for all futures to complete. Those collection patterns suit different needs: waiting for all is useful when a program needs every result before proceeding, while waiting for any is useful when a program can act on the first result, or when it wants to process results as they become available, rather than in submission order.
Futures also handle exceptions in a structured way. If a task raises an exception, that exception is stored in its future, rather than crashing the worker or the submitting program immediately. When the submitter retrieves the future's result, the exception is raised at that point, allowing the submitter to handle it with normal error handling, such as logging it, retrying the task, or reporting it to the user. That deferred exception handling is important, because it preserves the separation between executing a task and handling its outcome, and it prevents a single failing task from bringing down a whole pool.
Timeouts can be applied when waiting for a future, so that a program does not wait forever for a task that has hung or that is taking unexpectedly long. A timeout specifies how long to wait, after which the program can decide what to do, such as cancelling the task, if cancellation is supported, or proceeding without its result and reporting that it timed out. Timeouts are a practical safeguard for concurrent programs, especially those that depend on external resources, such as networks, where a task may hang indefinitely if a remote system does not respond. Choosing a timeout requires judgement, balancing patience for slow but legitimate work against the need to detect hangs promptly.
Finally, futures encourage a style of concurrent programming that is easier to reason about than raw threads, because tasks are expressed as units of work with clear inputs and outputs, and their execution is managed by a pool or an event loop. That style, sometimes called task-based concurrency, is the modern preferred approach in many languages, including Python, because it hides low-level details while preserving the benefits of concurrency. Understanding futures conceptually, even without mastering a particular library's syntax, helps you recognise task-based designs and appreciate why they are preferred to manual thread management.
Check: How does a future let a program submit work and collect its result later?
Section 13
Timeouts and cancellation are essential tools for managing concurrent tasks that may hang, take too long, or become unnecessary. A timeout sets a limit on how long a task, or a wait for a task, may take, after which the program takes alternative action, such as reporting a failure, using a default result, or trying another approach. Without timeouts, a program that waits for a task that never completes will hang forever, wasting resources and, in an interactive program, leaving the user waiting indefinitely. Timeouts turn an indefinite wait into a bounded one, with a clear outcome when the bound is exceeded.
Cancellation is related but distinct: it is a request to stop a task that is already running, because its result is no longer needed, or because it has exceeded its time limit. Cancellation is cooperative in many systems, meaning that a task must check for cancellation requests at suitable points and stop cleanly, rather than being forcibly terminated from outside, which could leave shared state inconsistent or resources unreleased. That cooperativeness means that cancellation must be designed into a task, with checks at points where it is safe to stop, such as between steps or while waiting, rather than assumed to work automatically for any task.
Designing tasks for cancellation requires thinking about cleanup. A task that opens files, holds locks, or uses network connections should release those resources when cancelled, just as it would on normal completion or on error. If cancellation leaves resources held, subsequent tasks may fail or hang, waiting for a resource that will never be released. That cleanup requirement is similar to error handling generally, and it is why tasks should be written with clear resource management, such as context managers that release resources automatically, even when a task is cancelled or raises an exception.
Timeouts and cancellation interact with user experience. In an interactive program, a long task that can be cancelled by the user, such as by pressing a cancel button, feels responsive, because the user retains control, even if the task itself is slow. In a server, a timeout on handling a request prevents a single slow request from consuming resources indefinitely, protecting other users from being affected by one slow task. In both cases, timeouts and cancellation are not just technical features; they are part of designing a program that behaves well under adverse conditions, such as slow networks or heavy load.
Finally, timeouts should be chosen with care, based on the task's normal duration and on the consequences of timing out too early or too late. A timeout that is too short will cause legitimate, slow tasks to fail unnecessarily, frustrating users and wasting work already done. A timeout that is too long will allow hung tasks to consume resources for too long before being detected. Measuring normal task durations, and setting timeouts generously above them, while still catching genuine hangs, is a practical approach, and it should be revisited as task durations change over time.
Check: Why is cancellation usually cooperative rather than forcible termination?
Section 14
Deadlock, livelock, and starvation are three failure modes of concurrent programs, each involving tasks that fail to make progress, but for different reasons. Deadlock, discussed earlier, occurs when tasks wait for each other in a cycle, each holding a resource that another needs, so none can proceed. Livelock occurs when tasks are not blocked, but they keep changing state in response to each other without making progress, such as two tasks repeatedly backing off and retrying in a way that always collides. Starvation occurs when a task is perpetually denied a resource because other tasks are always prioritised ahead of it, so it waits indefinitely, even though the system as a whole is making progress.
Deadlock prevention, as discussed, involves acquiring resources in a consistent order, or avoiding holding multiple resources at once, so that a cycle cannot form. Another prevention technique is to use timeouts when acquiring resources, so that a task that cannot acquire a resource within a limit releases the resources it already holds and retries later, breaking a potential deadlock cycle. That timeout-and-retry approach can prevent deadlock, but it can also cause livelock, if tasks repeatedly acquire, time out, release, and retry in a synchronised pattern that always collides. Adding randomness to retry delays, so that tasks do not retry in lockstep, is a common way to break such patterns and avoid livelock.
Starvation prevention involves fairness in scheduling and resource allocation. If a resource is allocated on a first-come, first-served basis, or with a queue that preserves order, tasks will eventually receive it, even if they must wait. If allocation is based on priority, without any ageing, where a task's priority increases the longer it waits, low-priority tasks may starve indefinitely, because higher-priority tasks keep arriving ahead of them. Fairness mechanisms, such as queues, ageing, or limits on how long a task can hold a resource, ensure that every task eventually progresses, even under heavy load, which is important for programs that must serve all users, not just high-priority ones.
Detecting these failure modes is difficult, because they may occur only under specific timing conditions, and they may not produce error messages, only a lack of progress. A program that appears hung, or that responds very slowly to some requests while others proceed normally, may be suffering from deadlock, livelock, or starvation, and diagnosing which one requires understanding the program's resource allocation and scheduling. Logging resource acquisition, task states, and waiting times can help diagnose such problems, by revealing which tasks are waiting, for what, and for how long, turning a mysterious hang into a analysable pattern.
Finally, the best defence against these failure modes is design simplicity. Programs with fewer shared resources, fewer locks, and clearer task structures are less likely to deadlock, livelock, or starve, because there are fewer opportunities for cycles, collisions, or unfair allocation. Starting with a simple design, such as a queue-based producer-consumer pattern with a single lock or no locks, and adding complexity only when needed, reduces the risk of these failure modes, while still achieving concurrency where it matters. Simplicity, as in sequential programming, is a virtue in concurrent programming too, and it is often the difference between a program that works reliably and one that fails mysteriously under load.
Check: How do deadlock, livelock, and starvation differ from each other?
Section 15
Thread safety means that code behaves correctly when called concurrently by multiple threads, and it is a property that must be designed for, not assumed. A function is thread-safe if it can be called by several threads at once, without race conditions, corrupted data, or incorrect results, even when those threads share data through the function. Some functions are thread-safe by nature, because they only use local variables and do not modify shared state, so concurrent calls cannot interfere. Others are thread-safe because they protect shared state with locks or other synchronisation, ensuring that concurrent calls are serialised where necessary. Still others are not thread-safe, and calling them concurrently can cause failures that are hard to reproduce.
Libraries and data structures vary in their thread safety, and documentation should be consulted before using them concurrently. A data structure that is safe for concurrent reads, but not for concurrent writes, may be suitable for a program where threads only read shared data, but not for one where they also write. A library function that uses internal shared state, such as a cache, may need external synchronisation, even if its interface looks simple. Assuming thread safety without checking documentation is a common source of concurrent bugs, because a program may work in testing, where concurrency is low, and fail in production, where concurrency is higher and timing differs.
Immutable data structures, discussed earlier, are inherently thread-safe for reading, because they cannot be changed, so concurrent reads cannot interfere. That is one reason why immutability is valued in concurrent programming: it removes a whole class of thread safety concerns, by design. Similarly, data structures that are designed for concurrent use, with internal synchronisation, can be used safely without external locks, but they may have performance costs, because internal synchronisation adds overhead to every operation. Choosing between immutable, concurrent, and ordinary data structures involves balancing safety, performance, and simplicity, based on the program's needs.
Thread safety also applies to external resources, such as files, network connections, and databases, which may or may not support concurrent access. A file opened by multiple threads, with each thread writing at different positions, may produce interleaved, corrupted output, unless writes are synchronised or each thread writes to its own file. A database connection shared by threads may need synchronisation, or each thread may need its own connection, depending on the database library's guarantees. Those external thread safety considerations are just as important as in-memory data structures, because real programs interact with external resources constantly.
Finally, testing thread safety is challenging, because race conditions may only appear under specific timing, as discussed earlier. Testing with high concurrency, with many threads and many operations, increases the chance of exposing a race, but it does not guarantee that a program is thread-safe, because a race may still require a particular interleaving that a test did not produce. Code review, focusing on shared state and synchronisation, is therefore an important complement to testing for thread safety, because it can identify potential races that testing might miss. A combination of careful design, documentation checking, stress testing, and review gives the best assurance of thread safety.
Check: Why should you check a library's documentation for thread safety before using it concurrently?
Section 16
An event loop is the core of asynchronous programming, and understanding it conceptually is more important than memorising its syntax. An event loop is a scheduler that runs in a single thread, maintaining a list of tasks that are ready to run and a list of tasks that are waiting for something, such as a network response or a timer. In each iteration, the loop picks a ready task, runs it until it pauses, either because it waits or because it completes, and then picks another ready task. When a waiting task's event occurs, such as data arriving, the loop marks it ready, so that it will be run in a future iteration. That cycle, of running ready tasks and waiting for events, continues until all tasks complete or the loop is stopped.
The event loop's single-threaded nature has important consequences. Because only one task runs at a time, tasks cannot interfere with each other in the middle of an operation, except at pause points, where they explicitly yield control. That means that shared data accessed between pauses is safe from interleaving, which simplifies reasoning about race conditions, compared with threads, where interleaving can occur at any point. However, it also means that a task that performs a long computation without pausing will block the loop, preventing other tasks from running, even if they are ready, because the loop cannot switch away from a task that does not yield. That blocking behaviour is a key limitation of asynchronous programming, and it is why long computations should be moved to another thread or process, or broken into pieces that pause periodically.
Event loops also handle timers, network events, and other asynchronous operations through a unified mechanism: tasks register interest in an event, and the loop wakes them when the event occurs. That unification allows a single loop to manage many different kinds of waiting, from network reads to timed delays, without needing separate threads for each. It is also why asynchronous programs can handle many simultaneous connections efficiently: each connection is a task waiting for data, and the loop only runs tasks when their data arrives, rather than dedicating a thread to each connection that spends most time waiting.
Understanding the event loop helps you understand asynchronous errors and debugging. If a program appears hung, it may be because the loop is blocked by a task that never pauses, or because no task is ready and the loop is waiting for an event that will never occur. If a task never runs, it may be because it was never scheduled, or because it is waiting for an event that was never registered correctly. Those diagnoses, based on the loop's model, are more useful than guessing, and they illustrate why conceptual understanding precedes syntax mastery for asynchronous programming.
Finally, event loops are not unique to Python; similar loops exist in other languages and in user interface frameworks, where they handle user input, timers, and redraws. Recognising the event loop pattern across contexts helps you transfer understanding from one domain to another, and it shows that asynchronous programming is not a Python-specific trick, but a general approach to managing many waiting tasks in a single thread. That generality is part of its value as a concept, even before you write substantial asynchronous code.
Check: Why does a long computation that never pauses block an event loop?
Section 17
Asynchronous tasks are units of work managed by an event loop, and understanding how they are created, combined, and awaited is key to asynchronous design, even at concept level. A task is created from an asynchronous function, which is a function written to pause at waiting points, and it is scheduled by the loop to run when ready. Multiple tasks can be created and run concurrently, in the sense that the loop interleaves them, even though only one runs at a time. That task model allows a program to express many independent waiting operations, such as fetching data from several sources, as separate tasks, rather than as a single sequential sequence that waits for each in turn.
Combining tasks is a common pattern. A program might create several tasks, each fetching data from a different source, and then wait for all of them to complete, collecting their results in order or as they finish. Waiting for all tasks is useful when a program needs every result before proceeding, such as when combining data from several sources into a single report. Waiting for the first task to complete is useful when a program has several ways to obtain the same result, such as trying several servers, and it only needs the fastest response. Those combination patterns, of waiting for all, for any, or for tasks as they complete, are fundamental to asynchronous design, and they appear in thread and process pools as well, in the form of futures, as discussed earlier.
Handling task results and exceptions follows a similar pattern to futures. If a task raises an exception, that exception is stored with the task, and it is raised when the task's result is retrieved, allowing the program to handle it with normal error handling. If a task is cancelled, it should clean up and stop, as discussed for cancellation generally, and its cancellation should be reported to the program that awaited it, so that the program can decide what to do next, such as retrying, using a default, or reporting a failure.
Timeouts for asynchronous tasks work similarly to timeouts for futures: a program can wait for a task with a time limit, after which it can cancel the task or proceed without its result. That timeout pattern is important for network operations, where a server may not respond, and a program should not wait forever. Choosing timeouts for asynchronous tasks, as for other concurrent tasks, requires balancing patience for slow responses against the need to detect failures promptly, and it should be based on measured normal durations, not on guesses.
Finally, asynchronous tasks should be designed to be independent where possible, with minimal shared mutable state, just as threaded tasks should be. Independence simplifies reasoning, reduces race conditions, and makes tasks easier to test, because each task can be understood on its own. When tasks must share state, that sharing should be confined to pause points, where it is safe, or protected by asynchronous synchronisation mechanisms, which are analogous to locks but designed for the event loop model. That discipline, of independence first and careful sharing second, applies to all concurrency models, and it is especially important for asynchronous tasks, where the illusion of sequential code can hide concurrent sharing if not considered explicitly.
Check: Why is waiting for all tasks useful when combining data from several sources?
Section 18
Choosing between threads, processes, and asynchronous programming depends on the nature of the task, and a clear decision framework helps make that choice systematically. The first question is whether the task is waiting-heavy or compute-heavy. Waiting-heavy tasks, such as network operations, file operations, or handling many simultaneous connections, benefit from threads or asynchronous programming, because those models exploit waiting time efficiently. Compute-heavy tasks, such as heavy calculations or data processing in Python, benefit from processes, because processes can parallelise computation across cores, while threads and asynchronous programs, in a single interpreter, cannot.
The second question is how many concurrent tasks are involved. For a small number of tasks, such as a handful of downloads, threads are simple and effective, because the overhead of threads is manageable and their programming model is familiar. For a very large number of tasks, such as thousands of simultaneous network connections, asynchronous programming is more efficient, because asynchronous tasks are lightweight compared with threads, and a single event loop can manage many tasks without the memory and switching overhead of many threads. For a moderate number of compute-heavy tasks, processes are appropriate, because their number can be matched to the number of cores, balancing parallelism against overhead.
The third question is about communication and shared state. If tasks need to share complex data frequently, threads are convenient, because they share memory, though that convenience brings race risks that must be managed. If tasks are independent, with little communication, processes are attractive, because their isolation prevents interference, and communication can be limited to sending inputs and collecting results through queues. If tasks communicate through messages and need to handle many waiting operations, asynchronous tasks, with their queue and message patterns, may be the clearest design, because they combine lightweight tasks with explicit communication.
The fourth question is about existing code and libraries. Some libraries are designed for threads, some for processes, and some for asynchronous use, and using a library in a model it was not designed for can cause problems, such as blocking an event loop with a synchronous library call, or using a non-thread-safe library concurrently without protection. Checking a library's concurrency model, and choosing a concurrency approach that matches the libraries a program depends on, is a practical consideration that can outweigh theoretical preferences. A program that must use a synchronous, blocking library may be better suited to threads or processes than to asynchronous programming, even if its tasks are waiting-heavy, because blocking calls would stall an event loop.
Finally, simplicity should be a tie-breaker. If several models could work, choose the simplest one that meets the task's needs, because simpler concurrent programs are easier to write correctly, test, and maintain. A sequential program that is fast enough needs no concurrency at all. A threaded program that handles a handful of downloads is simpler than an asynchronous program for the same task, if the task count is small. An asynchronous program that handles thousands of connections is simpler, in resource terms, than a threaded program for the same task, if the task count is large. Matching the model to the task, with simplicity as a guide, leads to concurrent programs that are both effective and maintainable.
Check: What four questions help choose between threads, processes, and asynchronous programming?
Section 19
Inter-process communication, often abbreviated as IPC, is how processes, which do not share memory, exchange data and coordinate their work. Common IPC mechanisms include pipes, which allow one process to send a stream of bytes to another; queues, which allow processes to send discrete messages, with synchronisation handled by the queue; and shared files, where processes write and read data through the filesystem. Each mechanism suits different patterns: pipes suit streaming data from one process to another, queues suit task distribution and result collection, and files suit persistent data that must survive beyond a single run.
Data sent between processes must be serialised, meaning converted from Python objects into a byte format that can be transmitted, and then deserialised back into objects on receipt. That serialisation step has costs: it takes time, it uses memory for the byte representation, and it may not support every kind of object, such as open files or complex objects with internal state that cannot be easily represented as bytes. Designing IPC with serialisation in mind means choosing data formats that are simple and serialisable, such as basic values, lists, dictionaries, and strings, rather than complex objects that are difficult to transmit. That simplicity also makes IPC easier to debug, because transmitted data can be inspected as bytes or as text.
Coordination between processes, beyond data exchange, also needs IPC. A main process may need to know when worker processes have finished, whether they succeeded or failed, and whether any need to be restarted. That coordination is often handled through queues for results, through exit codes that indicate success or failure, and through signals or messages that indicate shutdown or cancellation. Designing coordination explicitly, with clear protocols for starting, finishing, failing, and shutting down workers, prevents a multi-process program from hanging, losing results, or leaving orphan processes running after the main program exits.
Performance considerations for IPC include the overhead of serialisation and transmission, which can outweigh the benefit of parallelism if tasks are small and communication is frequent. A program that divides work into tiny pieces, each requiring IPC to send and receive, may spend more time communicating than computing, and may be slower than a sequential program. Dividing work into larger chunks, with less frequent communication, reduces IPC overhead and improves the benefit of parallelism. That chunking decision, of how much work to assign per IPC exchange, is a key design choice for multi-process programs, and it should be based on measurement, not on assumption.
Finally, IPC security deserves mention. Processes communicating through pipes or queues on the same computer are generally safe from external interference, but processes communicating over a network, or through files in shared locations, may be exposed to other users or to tampering. Validating data received through IPC, just as with any external input, and restricting access to communication channels to authorised processes, are basic security habits for multi-process programs. Those habits, while beyond the scope of this concept chapter, are part of designing robust concurrent systems, and they should be kept in mind as programs grow beyond a single computer.
Check: Why must data sent between processes be serialised, and what does that cost?
Section 20
Testing concurrent programs is more difficult than testing sequential programs, because their behaviour depends on timing and interleaving, which can vary from run to run. A test that passes once may fail on another run, not because the code changed, but because tasks interleaved differently, exposing a race condition that was previously hidden. That non-determinism makes concurrent testing less reliable as a sole guarantee of correctness, and it means that a passing test suite does not prove a concurrent program correct, only that it was correct for the interleavings that occurred during those tests. That limitation should be understood, so that testing is complemented by careful design and review.
One strategy for testing concurrency is to increase the chance of exposing races, by running tests many times, with high concurrency, and with varied timing, such as by introducing small, random delays into tasks. That stress testing, while it cannot prove correctness, can reveal races that a single, low-concurrency test would miss, because it explores more interleavings. Stress tests should be run as part of a test suite, but they should be recognised as probabilistic: a stress test that passes a hundred times may still miss a race that occurs only under a rare interleaving, so passing stress tests should increase confidence, not create certainty.
Another strategy is to make concurrency deterministic for testing, by controlling scheduling or by testing components in isolation. For example, a task's logic can be tested sequentially, without concurrency, to confirm that it produces correct results for given inputs, and then its concurrent interaction, such as queue handling or lock usage, can be tested separately, with controlled, deterministic scenarios. That separation, between logic testing and concurrency testing, makes tests more reliable, because logic tests are deterministic, and concurrency tests focus on the specific interactions that could race, rather than on the whole program at once.
Code review is particularly important for concurrent programs, because a reviewer can reason about shared state, locks, and ordering in a way that tests cannot fully automate. A review checklist for concurrency might ask: what shared state exists, how is it protected, what locks are acquired and in what order, what happens if a task fails or is cancelled, and how does the program shut down cleanly? Answering those questions systematically, for each concurrent component, catches design flaws that testing might miss, and it forces the author to think explicitly about concurrency, rather than assuming it works because tests passed.
Finally, logging and observability help diagnose concurrent problems in production, where tests cannot reproduce them. A concurrent program that logs task starts, completions, failures, and waiting times, with enough detail to reconstruct an interleaving, gives maintainers evidence to diagnose a failure that occurred only once, under load. That logging should be designed with concurrency in mind, such as by including task identifiers in log messages, so that interleaved logs from different tasks can be distinguished. Observability, combined with stress testing, deterministic component testing, and review, forms a layered approach to concurrent quality, where no single layer is sufficient, but together they provide reasonable assurance.
Check: Why does a passing test suite not prove a concurrent program correct?
Section 21
Two common patterns organise concurrent work: data parallelism and pipeline parallelism, though both are often simply called parallel patterns. In data parallelism, the same operation is applied to many independent pieces of data at once, such as converting many files or analysing many records. Each worker handles a different piece, and results are combined at the end. That pattern suits tasks where data can be divided cleanly, and where workers do not need to communicate while processing, because independence makes the program simple and safe.
Pipeline parallelism, by contrast, divides work into stages, with each stage performed by a different worker, and data flowing from one stage to the next through queues. For example, one stage might read files, the next might transform their contents, and the next might write results. That pattern suits tasks where work naturally falls into stages, and where different stages take different amounts of time, because stages can proceed concurrently on different items, like an assembly line. A pipeline can be more efficient than data parallelism for staged work, but it is also more complex to balance, because a slow stage can bottleneck the whole pipeline.
Choosing between those patterns depends on the task's structure. If the task is a single operation repeated many times, data parallelism is usually simpler and more effective. If the task is a sequence of distinct operations, each applied to many items, a pipeline may be more natural, because it mirrors the task's stages and allows each stage to be optimised separately. Some programs combine both patterns, using data parallelism within a pipeline stage, for example by having several workers for a slow stage, to balance the pipeline. That combination, while powerful, adds complexity and should only be used when simpler patterns prove insufficient.
Both patterns benefit from the queue and pool ideas discussed earlier. Data parallelism can be implemented with a pool of workers taking items from a queue, while pipeline parallelism can be implemented with a queue between each pair of stages. Those shared building blocks mean that learning queues and pools gives you the tools for both patterns, and that a program can be restructured from one pattern to another without redesigning its basic communication mechanisms. That reusability of patterns is a practical advantage of thinking in terms of tasks, queues, and workers, rather than in terms of low-level threads or processes.
Finally, both patterns should be measured, not assumed to be faster. A data-parallel program that divides work poorly, or a pipeline that is bottlenecked by one stage, may be slower than a sequential program, once overhead is counted. Measuring throughput, with different numbers of workers and different chunk sizes, reveals whether a pattern helps and how it should be tuned. That measurement, applied to a real workload, turns a pattern from a theoretical idea into a practical design decision, grounded in evidence rather than in enthusiasm for concurrency.
Check: How do data parallelism and pipeline parallelism differ in how they divide work?
Section 22
Flow control, also called backpressure in queue contexts, is the management of how fast work is produced compared with how fast it is consumed, and it is essential for stable concurrent programs. Without flow control, a fast producer can overwhelm a slow consumer, causing unbounded growth in queued work, memory exhaustion, or stale results that are no longer relevant by the time they are processed. With flow control, production slows when consumption cannot keep up, keeping the system stable, even under bursts of work, and ensuring that resources are used efficiently rather than wasted on work that will never be completed in time.
Bounded queues, discussed earlier, are one form of flow control, because they block producers when full, forcing them to wait until consumers free space. Another form is limiting the number of outstanding tasks, such as by only submitting new tasks when previous tasks complete, which keeps the number of in-flight tasks within a chosen limit. A third form is dropping or shedding load, where a program under heavy load refuses new work, with a clear error, rather than accepting work it cannot complete in reasonable time. That shedding, while it may seem harsh, is better than accepting work and then failing slowly, because it gives the requester a clear signal to retry later, rather than a long, uncertain wait.
Flow control also applies to network servers, where a server that accepts connections faster than it can handle them will accumulate connections, consume memory, and eventually fail, affecting all users. A server with flow control, such as a limit on simultaneous connections, or a queue with a maximum size, will refuse or delay new connections when overloaded, protecting existing connections and maintaining quality for users already being served. That protection, of existing work at the expense of new work, is a deliberate trade-off, and it is often the right one for a system that must remain responsive under load.
Designing flow control requires choosing limits based on measurement and on the system's resources. A queue size, a connection limit, or a task limit that is too small will underuse the system, rejecting work that could have been handled, while a limit that is too large will allow overload before flow control engages. Measuring a system's capacity, under realistic load, and setting limits generously above normal load but below overload, is a practical approach, and those limits should be revisited as the system or its workload changes. Flow control limits are not fixed truths; they are tuning parameters that reflect a system's current capacity.
Finally, flow control should be visible in monitoring and logging. A program that is shedding load, or that is frequently blocking producers due to a full queue, should log that fact, so that operators can see that the system is overloaded and can take action, such as adding capacity or reducing demand. Silent flow control, where work is dropped or delayed without any record, hides overload and makes it hard to diagnose why users experience failures or delays. Visible flow control, with counts of dropped, delayed, and queued work, turns overload from a mystery into a measurable condition that can be managed.
Check: Why is refusing new work sometimes better than accepting work a system cannot complete in time?
Section 23
External shared resources, such as files, databases, and network services, introduce concurrency challenges beyond in-memory shared state, because they are shared not only by tasks within a program, but potentially by other programs as well. A file written by multiple tasks, or by multiple programs, can become corrupted if writes interleave, just as a shared variable can, but the consequences may be more lasting, because a corrupted file persists after the program exits. Similarly, a database updated concurrently by several tasks can produce inconsistent data if transactions are not used correctly. Recognising external resources as shared state, and managing their concurrent access, is therefore essential for real programs.
Files are a common external resource, and their concurrent access patterns vary. Reading the same file concurrently is usually safe, because reads do not change the file, though a file being written by another task may be read in a partially updated state, leading to inconsistent reads. Writing to the same file concurrently is unsafe without coordination, because writes from different tasks can interleave at arbitrary points, producing a mixture that belongs to neither task. Coordinating file writes, either by giving each task its own file, by serialising writes through a single task, or by using file locking where available, prevents that corruption, and the choice among those approaches depends on whether the file must be a single, combined output or can be split.
Databases provide their own concurrency mechanisms, such as transactions, which group a set of operations into an atomic unit that either completes fully or not at all, and locking, which prevents conflicting updates. Using those mechanisms correctly, by grouping related operations into transactions and by handling transaction failures, such as conflicts that require retry, is part of concurrent database programming. A program that updates a database from multiple tasks without transactions may produce inconsistent data, even if each update is individually correct, because a task may read data that another task is halfway through changing, or two tasks may update the same record based on stale information.
Network services, such as web APIs, also have concurrency considerations. A service may limit how many simultaneous requests it accepts, or how many requests a client may make in a given period, known as rate limiting. A concurrent program that sends many requests at once may exceed those limits, causing requests to be rejected, or it may overload the service, affecting other users. Respecting rate limits, by pacing requests, using backoff when rejected, and limiting concurrency to a level the service can handle, is both polite and practical, because it avoids rejected requests and service overload, which would slow the program more than pacing does.
Finally, external resources often have their own documentation about concurrent use, which should be consulted before designing concurrent access. A file format, a database, or a service may guarantee safe concurrent access under certain conditions, or may require specific patterns, such as one writer at a time, or a particular transaction isolation level. Following those documented guarantees, rather than assuming safe concurrency, prevents a class of bugs that are hard to diagnose, because they involve interaction between a program and an external system, rather than within the program alone. External concurrency, like internal concurrency, rewards explicit design over assumption.
Check: Why is writing to the same file from multiple tasks unsafe without coordination?
Section 24
User interfaces and concurrency are closely linked, because interfaces must remain responsive while work happens in the background. A graphical interface runs an event loop, similar to the asynchronous event loop discussed earlier, which handles user input, redraws, and timers. If a long task runs on that loop's thread, the interface freezes, because the loop cannot handle input or redraw until the task finishes. That freeze is experienced by users as unresponsiveness, even if the program is working correctly, and it is one of the most common complaints about poorly designed interactive programs. Moving long work to a background thread, process, or asynchronous task, while keeping interface updates on the loop thread, solves that problem.
Background work must communicate with the interface carefully. A background task that updates interface elements directly, from a non-interface thread, can cause errors or inconsistent displays, because interface toolkits are often not thread-safe, and they expect all interface updates to happen on their own thread. The safe pattern is for background tasks to send results or progress updates through a queue or a message mechanism, which the interface loop then processes on its own thread, updating the display safely. That pattern, of background work plus queued interface updates, is a standard design for responsive interfaces, and it mirrors the queue-based patterns discussed for general concurrency.
Progress reporting from background work to an interface also needs care. A background task that reports progress too frequently, such as for every item processed, can overwhelm the interface loop with updates, making it sluggish, even though the background work itself is fast. Reporting progress periodically, such as every hundred items or every second, balances responsiveness against overhead, giving users a sense of progress without flooding the interface. That periodic reporting, combined with a cancel option that signals the background task to stop, as discussed for cancellation, makes long tasks feel controllable and transparent, rather than mysterious and uninterruptible.
Cancellation from an interface deserves special attention, because users expect a cancel button to stop work promptly, but background tasks may be in the middle of an operation that cannot be stopped instantly, such as writing a file or waiting for a network response. Designing cancellation to be checked at safe points, and to clean up partial work, as discussed earlier, ensures that cancelling does not leave the program in an inconsistent state, such as a half-written file presented as complete. Communicating cancellation status to the user, such as showing that cancellation is in progress, prevents confusion when stopping takes a moment, rather than happening instantly.
Finally, even command-line interfaces benefit from concurrency for responsiveness, though less visibly than graphical interfaces. A command-line program that processes a long job while also responding to user input, such as a request for status or for cancellation, needs a concurrent design, with input handling separate from processing. That design, while simpler than a graphical interface, follows the same principles: keep the input path responsive, do long work in the background, communicate through queues, and handle cancellation cleanly. Responsive design, in any interface, is a concurrency design problem, and it rewards the same careful thinking about tasks, communication, and cleanup.
Check: Why should background tasks send interface updates through a queue rather than updating the interface directly?
Section 25
Web servers are a classic example of concurrent programs, because they must handle many clients at once, each of which may wait, send data slowly, or disconnect unexpectedly. A sequential server, handling one client at a time, would be unusable for more than a few clients, because a slow client would block all others. A concurrent server, using threads, processes, or asynchronous tasks, can handle many clients simultaneously, making progress on whichever clients are ready, while others wait. That concurrency is what allows a server to serve many users with reasonable responsiveness, even on modest hardware, and it is why web server design is a central topic in concurrency.
Thread-based servers assign a thread to each connection, or use a pool of threads to handle connections from a queue. That design is conceptually simple, because each connection's handling code can be written sequentially, as if it were the only connection, with the server's concurrency handled by the thread system. Its limitation, as with threads generally, is resource use: each thread consumes memory, and a server handling thousands of connections with a thread each may consume significant memory and switching overhead, even though most connections are idle, waiting for data. Thread pools mitigate that by limiting threads, but they still require a thread per active connection, up to the pool size.
Process-based servers assign a process to each connection, or use a pool of processes, gaining isolation, so that a crash in handling one connection does not affect others, at the cost of higher overhead per connection and more complex communication, if connections need to share state. That isolation is valuable for servers handling untrusted clients, where a failure in one connection should not bring down the server, but it is less efficient for large numbers of connections, for the same resource reasons as threads, only more so, because processes are heavier than threads.
Asynchronous servers use a single event loop, with a task per connection, handling many connections efficiently, because tasks are lightweight and the loop only runs tasks when their data arrives. That efficiency is why asynchronous servers are popular for handling many simultaneous, mostly idle connections, such as chat servers or real-time services, where connections spend most time waiting. Their limitation, as with asynchronous programming generally, is that long computations block the loop, so compute-heavy request handling must be moved to threads or processes, or broken into pausing pieces, to avoid stalling all connections.
Finally, web servers illustrate that concurrency choice depends on workload. A server handling a few, compute-heavy requests may suit processes, for parallelism and isolation. A server handling many, waiting-heavy connections may suit asynchronous tasks, for efficiency. A server handling a moderate number of connections, with straightforward handling code, may suit threads, for simplicity. There is no single best server concurrency model; there is a best model for a given workload, and understanding the trade-offs discussed in this chapter helps you choose and, if needed, combine models, such as an asynchronous front end with a process pool for computation, to match a real workload's mix of waiting and computing.
Check: Why can a sequential web server not serve many clients acceptably?
Section 26
Task parallelism and data parallelism, introduced earlier, deserve a clearer distinction, because they describe different ways to divide work, with different implications for design. Task parallelism divides a program into different tasks, each performing a different operation, which run concurrently. For example, one task might fetch data, another might process it, and another might write results, all running at once on different items, as in a pipeline. Data parallelism, by contrast, divides data into pieces, and applies the same operation to each piece concurrently, such as converting each file in a folder with the same conversion function, using several workers. That distinction, between different operations and same operation on different data, shapes how a program is structured.
Task parallelism suits programs where work naturally falls into different kinds of operations, especially when those operations have different resource needs or different durations. A program that reads from a network, transforms data, and writes to a database, for example, has three distinct tasks, each of which can proceed concurrently on different items, improving throughput compared with performing all three operations sequentially for each item. Task parallelism also suits programs where operations must run continuously, such as a program that monitors a folder, processes new files, and reports results, with each operation running as a long-lived task, rather than as a one-off job.
Data parallelism suits programs where the same operation is repeated many times, on independent data, and where the main challenge is throughput, not coordination between different operations. A program that analyses many independent records, converts many images, or checks many files, is data-parallel, because each record, image, or file can be processed independently, and results can be combined at the end. Data parallelism is often simpler to implement than task parallelism, because workers are identical, they do not need to coordinate with each other except to receive work and return results, and scaling is a matter of adding more workers, up to the limits of resources and overhead.
Some programs combine both forms. A pipeline stage, which is task-parallel in the sense that stages perform different operations, may itself be data-parallel, with several workers performing the same stage operation on different items, to balance a slow stage. That combination, while powerful, adds complexity in scheduling, load balancing, and result ordering, because results from parallel workers within a stage may complete out of order, and the next stage may need them in order, or may need to handle out-of-order results gracefully. Designing for out-of-order completion, where it matters, is part of combining parallelism forms, and it should be considered explicitly, rather than assumed away.
Finally, the distinction between task and data parallelism helps with performance reasoning. Data parallelism scales with the amount of independent data: more data means more parallel work, up to resource limits. Task parallelism scales with the number of distinct operations that can overlap: more stages means more potential overlap, but only up to the number of stages, after which adding more tasks does not help. Understanding which form dominates a program's work helps you predict how it will scale, where its bottlenecks will be, and whether adding concurrency will help or merely add overhead. That prediction, grounded in the program's structure, is more useful than a generic belief that parallelism always speeds things up.
Check: How does task parallelism differ from data parallelism in what is divided?
Section 27
Concurrent bugs are often called Heisenbugs, a play on words suggesting that they change behaviour when observed, because adding logging, running under a debugger, or changing timing can alter interleavings and hide or reveal a race. That elusiveness makes concurrent debugging particularly frustrating: a bug that occurs in production, under load, may disappear when you try to reproduce it in development, with less load and different timing, leading to a cycle of attempted fixes that cannot be verified because the bug cannot be reproduced reliably. Understanding that Heisenbugs are a consequence of non-determinism, not of mysterious forces, helps you approach them systematically, rather than with superstition.
Systematic debugging of concurrent bugs starts with narrowing the problem, as with sequential bugs, but with attention to shared state and timing. Identifying which shared data is involved, which tasks access it, and in what order, turns a vague hang or incorrect result into a hypothesis about a race, a deadlock, or a missed synchronisation. That hypothesis can then be tested, by adding targeted logging that records accesses to the shared data, with task identifiers and timestamps, so that an interleaving can be reconstructed from logs, even if the bug does not reproduce on demand. That logging, designed to be lightweight so that it does not itself change timing too much, is a key tool for concurrent debugging.
Another technique is to force particular interleavings, by introducing delays, barriers, or controlled scheduling in tests, to make a suspected race occur reliably. For example, if you suspect that two tasks race to update a counter, you can insert a delay between reading and writing in a test version, to ensure that both tasks read before either writes, reproducing the lost update consistently. That forced interleaving turns a Heisenbug into a deterministic bug, which can then be fixed and verified, because the test will fail consistently before the fix and pass consistently after it. That technique, of making timing explicit in tests, is powerful for concurrent code, and it complements stress testing, which relies on random timing to expose races probabilistically.
Code review, as discussed earlier, is also a debugging tool for concurrency, because a reviewer can spot a missing lock, an incorrect lock order, or an unprotected shared access, by reasoning about the code, without needing to reproduce a failure. A review focused on concurrency, with a checklist of shared state, synchronisation, shutdown, and error handling, often finds bugs that testing missed, because it considers all possible interleavings, not just those that occurred in tests. That reasoning, while it requires effort, is more reliable than hoping that tests will happen to produce a problematic interleaving.
Finally, preventing concurrent bugs is better than debugging them, and prevention comes from design: minimise shared state, use queues and pools rather than ad-hoc threads, protect what shared state remains, and test components in isolation before integrating them concurrently. A program designed with those principles will have fewer Heisenbugs, and those it does have will be easier to diagnose, because its concurrency structure is clear. Debugging skill is valuable, but design skill, which reduces the need for debugging, is more valuable still, and it is the skill this chapter aims to develop.
Check: Why are concurrent bugs sometimes called Heisenbugs?
Section 28
Beyond locks, several synchronisation primitives help coordinate concurrent tasks, each suited to a particular pattern. An event is a simple flag that one task sets to signal that something has happened, and other tasks wait for, such as a shutdown signal, a data-ready signal, or a start signal for a group of tasks. Events are useful for one-to-many signalling, where one task's action should wake several waiting tasks, without those tasks needing to poll repeatedly, which would waste processor time. Using an event for shutdown, for example, allows worker tasks to wait efficiently until the main task signals that no more work will arrive, at which point they can exit cleanly.
A semaphore is a counter that limits how many tasks may access a resource at once, generalising a lock, which allows only one task. A semaphore with a count of three, for example, allows up to three tasks to use a resource simultaneously, such as a pool of database connections, where three connections are available, and a fourth task must wait until one is released. Semaphores are useful for managing limited resources, where the limit is greater than one, and where tasks should block rather than fail when the resource is exhausted. Using a semaphore correctly requires acquiring before use and releasing after use, reliably, even on error, just as with locks, but with a count rather than a binary state.
A barrier is a synchronisation point where a group of tasks must all arrive before any may proceed, useful for phased computations where each phase depends on all tasks completing the previous phase. For example, in a simulation with several workers, each worker might compute its part of a step, then wait at a barrier until all workers have finished that step, before any proceeds to the next step, ensuring that no worker races ahead using incomplete data from others. Barriers simplify phased programs, by making phase boundaries explicit, rather than requiring manual coordination through events or counters, which is error-prone.
A condition is a more advanced primitive that allows tasks to wait for a particular condition to become true, such as a queue becoming non-empty, or a value reaching a threshold, while holding a lock, and to be notified when the condition may have changed. Conditions are the building blocks for many higher-level synchronisation structures, including queues themselves, and they are useful when waiting depends on a state that changes over time, rather than on a simple event. Understanding conditions conceptually, as wait-and-notify with a lock, helps you understand how queues and other structures work internally, even if you rarely use conditions directly in application code.
Finally, choosing among these primitives should be guided by the pattern you need: events for signalling, semaphores for limited resources, barriers for phased synchronisation, and conditions for state-dependent waiting, with locks as the foundation for protecting shared data. Using the primitive that matches the pattern, rather than forcing every coordination problem into locks, leads to clearer, more correct concurrent programs, because the primitive's semantics match the problem's requirements. That matching, of primitive to pattern, is a key skill in concurrent design, and it grows with practice in recognising common coordination patterns.
Check: When would a semaphore be more appropriate than a simple lock?
Section 29
Coroutines are the building blocks of asynchronous programming, and understanding them conceptually helps you understand asynchronous code, even without writing a full asynchronous program yet. A coroutine is a function that can pause its execution, at a point where it waits, and later resume from exactly where it paused, with its local variables intact. That pause-and-resume ability is what allows an event loop to interleave coroutines: when one coroutine pauses to wait for data, the loop runs another coroutine, and when data arrives, the first coroutine resumes, continuing as if it had never paused, except for the time that passed. That model, of functions that can pause, is the essence of asynchronous programming.
Coroutines differ from ordinary functions, which run from start to finish without pausing, and from threads, which are paused by the system at arbitrary points. A coroutine pauses only at explicit points, where it awaits an operation, which makes its execution predictable: between pauses, it runs without interruption, so shared data accessed in that interval is safe from interleaving, as discussed earlier. That predictability is a major advantage of coroutines, because it allows a programmer to reason about a coroutine's code sequentially, between pauses, while still achieving concurrency across coroutines, through the loop's interleaving at pauses.
Writing coroutines requires a style where waiting operations are marked as awaitable, and where code that calls such an operation must itself be a coroutine, or must be run in a way that bridges synchronous and asynchronous code. That requirement, sometimes called async contagion, means that introducing asynchronous code into a program can spread, as functions that call asynchronous functions must themselves become asynchronous, up to the program's entry point, which is run by the event loop. That spread is a real cost of asynchronous programming, and it should be considered when deciding whether to adopt it, because it can require restructuring a significant part of a program, not just adding a few asynchronous functions in isolation.
Coroutines also interact with error handling and cancellation in ways that follow the task model discussed earlier. If a coroutine raises an exception, that exception propagates to the task that awaited it, where it can be handled normally. If a coroutine is cancelled while paused, it resumes with a cancellation signal, allowing it to clean up and stop, as discussed for cancellation generally. Those behaviours, which mirror ordinary function error handling, make coroutines easier to reason about than callback-based asynchronous programming, where errors and cancellation are handled through separate mechanisms that are harder to follow.
Finally, coroutines are a concept, not just a Python feature. Similar pause-and-resume functions exist in other languages, and the event loop model, with coroutines as tasks, is a general pattern for managing waiting work in a single thread. Understanding coroutines conceptually, as pausable functions scheduled by a loop, gives you a mental model that transfers across languages and frameworks, and it prepares you for a full asynchronous tutorial, where you will write coroutines, await operations, and combine tasks, with that model in mind. That preparation, of concept before syntax, is the approach this chapter has taken throughout, and it is particularly valuable for asynchronous programming, where syntax without concept is confusing.
Check: How does a coroutine's explicit pause make its execution more predictable than a thread's?
Section 30
The simplest concurrency decision is often to not use concurrency at all, and recognising when sequential code suffices is a valuable skill. A program that is fast enough sequentially, that handles only a few tasks, or that is run infrequently, does not need concurrency, and adding it would only add complexity, risk, and maintenance burden, for no meaningful benefit. That judgement, of sufficiency, should be based on measurement: if a sequential program meets its performance and responsiveness requirements, with realistic data and realistic load, then it is sufficient, and concurrency is an unnecessary optimisation, in the sense discussed in earlier chapters about premature optimisation.
When concurrency is needed, starting with the simplest model that meets the need is wise. For a handful of waiting tasks, a small thread pool may be simpler than a full asynchronous design, because its code can be written sequentially, with threads handling interleaving. For a compute-heavy task with independent data, a process pool with a simple map operation, where the same function is applied to many items, may be simpler than a custom pipeline with queues and workers. Choosing the simplest model that fits, rather than the most sophisticated model available, keeps concurrent programs understandable and maintainable, which is especially important because concurrent bugs are harder to find than sequential ones.
Simplicity also applies within a concurrent design. A program with a single queue, a fixed pool, and clear task boundaries is simpler than one with multiple queues, dynamic pool sizing, and complex coordination, and it will usually be easier to debug and to scale, because its behaviour is predictable. Adding complexity, such as dynamic sizing or prioritised queues, should be justified by a measured need, such as a bottleneck that simpler designs cannot address, rather than by anticipation of a problem that may never occur. That restraint, of adding complexity only when measured need demands it, is a general engineering principle, and it applies with particular force to concurrency, where complexity multiplies risk.
Finally, sequential-first thinking helps you learn concurrency incrementally. Writing a correct sequential version first, with clear task boundaries and testable functions, gives you a baseline to compare a concurrent version against, and it ensures that the program's logic is correct before concurrency is added. Adding concurrency to a correct sequential program, in small steps, with tests after each step, turns a risky rewrite into a series of safe, verifiable changes, as discussed for refactoring generally. That incremental approach, from sequential to concurrent, is the safest path to a correct concurrent program, and it is a fitting summary of this chapter's advice: understand the task, choose the simplest concurrency model that helps, design for safety, and measure the result.
A sequential baseline also makes it easier to spot whether a concurrent version actually improves performance, because its timings can be compared directly with the simpler version.
Check: Why is writing a correct sequential version first a good starting point for a concurrent program?
Section 31
State machines offer a way to think about concurrent tasks clearly, by modelling each task as moving through a set of states, such as waiting, running, done, or failed, with transitions triggered by events. That model helps you reason about a task's lifecycle, because at any moment a task is in exactly one state, and its next state depends on the event that occurs, not on hidden variables scattered through code. For a concurrent program with many tasks, a state machine diagram, even an informal one drawn on paper, can reveal missing transitions, such as what happens if a task fails while waiting, or what happens if a shutdown signal arrives while a task is running, which are easy to overlook in code alone.
State machines also help with shared state, because a task's state can be stored in a single, well-defined place, rather than being inferred from several flags that may become inconsistent. For example, a download task might have states for queued, downloading, paused, completed, and failed, with clear transitions between them, and with actions associated with each transition, such as cleaning up a partial file when moving to failed. That clarity makes it easier to protect state transitions with synchronisation, because there is a single point where state changes, rather than many places where flags are set and cleared independently, which can race.
In asynchronous programming, tasks naturally have states, managed by the event loop, such as pending, running, waiting, done, and cancelled, and understanding those states helps you understand task lifecycle, including what it means for a task to be cancelled while waiting, or for a task to complete with an exception. In threaded and process programs, tasks also have lifecycles, though they may be less explicit, because threads and processes are managed by the operating system rather than by a loop. Making those lifecycles explicit, through state variables or through futures that report completion, helps you manage tasks consistently, whatever the concurrency model.
State machines can also model resources, not just tasks. A connection pool, for example, might model each connection as idle, in use, or broken, with transitions when a connection is borrowed, returned, or found to be faulty. That model helps you design pool behaviour, such as what to do when all connections are in use, or when a broken connection is returned, because each case corresponds to a state transition that can be handled explicitly. Modelling resources as state machines, like modelling tasks, turns implicit assumptions into explicit design, which is easier to test and to review.
Finally, state machines are a design tool, not a requirement for every concurrent program. A simple program with a few tasks and a queue may not need a formal state machine, and imposing one would add unnecessary complexity. A program with many tasks, complex lifecycles, or tricky shutdown behaviour, by contrast, benefits greatly from explicit state modelling, because it forces you to consider every transition, including error and cancellation transitions that are easy to forget. Using state machines where they clarify, and not where they clutter, is a judgement that improves with experience, and it is a useful addition to your concurrent design toolkit.
Check: How does modelling a task as a state machine help reveal missing transitions?
Section 32
Ordering and determinism are concerns in concurrent programs because, by default, concurrent tasks complete in an unpredictable order, depending on timing, load, and scheduling. A program that processes items concurrently and writes results in completion order will produce output in a different order each run, which may be acceptable if order does not matter, but which is confusing if users expect results in input order, or if downstream processing assumes a particular order. Designing for ordering, where it matters, requires explicit mechanisms, such as tagging each task with its input position and sorting results by that tag before output, rather than assuming that tasks will complete in submission order, which they will not, in general.
Determinism, meaning that the same input always produces the same output, is a stronger requirement than ordering, and it is harder to achieve in concurrent programs, because timing variations can affect not only order but also which task wins a race, or which error occurs first when several tasks fail. A program that is deterministic sequentially may become non-deterministic concurrently, if its result depends on task interleaving, such as when tasks update shared state in an order-dependent way. Achieving determinism concurrently requires eliminating order-dependent shared updates, for example by giving each task its own result and combining results in a fixed order, rather than by letting tasks update a shared result in completion order.
There are cases where non-determinism is acceptable, such as when a program searches for any solution, rather than a specific one, or when it processes independent items and order truly does not matter for its purpose. In those cases, accepting non-deterministic completion order simplifies design, because no sorting or ordering mechanism is needed, and it may improve performance, because results can be output as soon as they are ready, rather than waiting for earlier tasks to finish. Accepting non-determinism should be a deliberate decision, documented so that users and maintainers know that output order may vary, rather than an accidental property that surprises users who assumed a stable order.
Testing deterministic versus non-deterministic behaviour also differs. A deterministic concurrent program can be tested by comparing its output with expected output, just as a sequential program can, because its output should be the same every run, despite internal concurrency. A non-deterministic program cannot be tested that way, because its output order varies; instead, its tests must check that its output, as a set, is correct, regardless of order, or that its output satisfies certain properties, such as containing all expected items, rather than matching a fixed sequence. Designing tests to match the program's determinism guarantees, rather than assuming determinism where it does not exist, prevents flaky tests that fail intermittently due to order variation alone.
Finally, ordering mechanisms add overhead, because sorting results, buffering out-of-order results until earlier ones arrive, or waiting for tasks in order, all take time and memory. If ordered output is essential for users, the overhead is justified. If it is merely convenient, but unordered output would still be usable, accepting unordered output may be the better trade-off, especially for large workloads where ordering overhead is significant. That trade-off, between determinism and performance, is a practical design decision that should be made explicitly, not by default.
Check: How can a concurrent program produce results in input order despite tasks completing out of order?
Section 33
Resource pools manage a limited set of reusable resources, such as database connections, network connections, or file handles, by lending them to tasks and reclaiming them when tasks finish. A pool is created with a fixed number of resources, tasks borrow a resource when they need it, use it, and return it to the pool, where it becomes available for another task. That reuse avoids the cost of creating a new resource for every task, which can be expensive, such as opening a new database connection, while also limiting the total number of resources in use, which protects the resource's provider, such as a database server, from being overwhelmed by too many simultaneous connections.
Pools interact with concurrency in several ways. A task that needs a resource but finds the pool empty must wait, which is a form of backpressure, as discussed earlier, or it may fail with a timeout, if waiting would take too long. That waiting should be handled gracefully, with a clear error or a retry, rather than hanging indefinitely, because a pool that is exhausted due to a leak, where resources are borrowed but never returned, will cause all subsequent tasks to wait forever. Preventing leaks, by ensuring that borrowed resources are always returned, even on error or cancellation, through reliable cleanup mechanisms, is therefore essential for pool correctness, and it is similar to lock release discipline discussed earlier.
Pool size tuning, like worker pool tuning, involves balancing throughput against resource limits. A pool that is too small will cause tasks to wait unnecessarily, reducing throughput, even though the system could handle more concurrent resource use. A pool that is too large may overwhelm the resource provider, causing it to slow down or reject connections, which also reduces throughput, despite more resources being available. Measuring throughput with different pool sizes, under realistic load, reveals the sweet spot for a given workload and provider, and that measurement should be revisited as workload or provider capacity changes.
Pools also need health checking, because a pooled resource may become broken while idle, such as a database connection that timed out, or a network connection that was closed by the other end. A pool that lends a broken resource without checking it will cause the borrowing task to fail, even though a fresh resource would have succeeded. Checking a resource before lending it, or discarding resources that fail and replacing them with fresh ones, improves pool reliability, at the cost of a small overhead per borrow. That health checking, like other pool maintenance, is part of designing a robust pool, rather than a simple list of resources.
Finally, pools illustrate a general concurrency principle: shared resources should be managed centrally, with clear borrowing and returning protocols, rather than accessed ad hoc by many tasks. Central management makes it possible to enforce limits, to detect leaks, to check health, and to log usage, all in one place, rather than scattering those concerns across every task that uses the resource. That centralisation, while it adds a layer of indirection, simplifies reasoning about resource use under concurrency, and it is a pattern that applies to many kinds of shared resources, from connections to threads to memory buffers.
Check: Why must borrowed pool resources always be returned, even when a task fails?
Section 34
Caching is a technique for speeding up programs by storing results of expensive operations and reusing them, rather than recomputing them, and it interacts with concurrency in interesting ways. A cache shared by concurrent tasks can improve performance dramatically, if many tasks request the same data, because only the first task needs to compute it, while others reuse the cached result. However, a shared cache is also shared mutable state, with all the race and consistency issues discussed earlier, plus additional issues specific to caching, such as what happens when cached data becomes stale, or when two tasks simultaneously try to compute and store the same missing value, known as a cache stampede.
A cache stampede occurs when many tasks request the same uncached value at once, and each task, finding it missing, starts computing it independently, wasting work and potentially overwhelming the underlying resource, such as a database or a network service. Preventing stampedes requires coordination, such as allowing only one task to compute a missing value, while others wait for its result, rather than all computing in parallel. That coordination, often implemented with a lock per cache key, or with a future that represents the in-progress computation, turns a stampede into a single computation with shared results, which is both more efficient and less load on the underlying resource.
Cache invalidation, deciding when cached data is no longer valid and should be recomputed, is another concurrency challenge, because data may change while tasks are using cached copies, leading to stale reads. A cache that never invalidates may serve outdated data indefinitely, which is incorrect for data that changes. A cache that invalidates too aggressively may lose its benefit, because tasks constantly recompute values that have not actually changed. Choosing an invalidation strategy, such as time-based expiry, where cached values expire after a period, or explicit invalidation when data changes, involves balancing freshness against performance, and it should be based on how frequently data changes and how harmful stale data would be for the program's purpose.
Thread safety for caches, as for other shared structures, requires protecting cache operations with synchronisation, or using a cache designed for concurrent use, with internal synchronisation. A cache that is safe for concurrent reads but not for concurrent writes, for example, may be suitable for a read-heavy workload, but not for one where tasks frequently add new values. Checking a cache implementation's concurrency guarantees, and testing it under concurrent load, as discussed for thread safety generally, prevents cache-related races that can cause incorrect results, crashes, or corrupted cache state.
Finally, caching should be measured, not assumed to help. A cache that is rarely hit, because tasks request different values each time, adds overhead for storing and looking up values, without benefit, and may even slow a program down. A cache that is frequently hit, but whose values are expensive to compute, can dramatically speed a program up. Measuring hit rates, computation costs, and overall throughput, with and without caching, under realistic workloads, reveals whether caching helps for a particular program, and how large a cache should be, before it consumes excessive memory for diminishing returns. That measurement, like other performance decisions, grounds caching in evidence rather than in hope.
Check: What is a cache stampede, and how can coordination prevent it?
Section 35
Graceful shutdown is how a concurrent program stops cleanly, finishing work in progress, releasing resources, and exiting without losing data or leaving tasks hanging. A program that is simply terminated, without shutdown handling, may lose results that were computed but not yet saved, leave files half-written, or leave background tasks running as orphans, consuming resources after the main program has exited. Graceful shutdown, by contrast, involves a sequence: stop accepting new work, allow existing work to finish, or cancel it cleanly if it cannot finish promptly, save any pending results, release resources such as connections and files, and then exit, with a clear log message that shutdown completed.
Designing graceful shutdown starts with a shutdown signal, which may come from a user interrupting the program, from a system requesting termination, or from the program itself deciding that its work is done. That signal should be communicated to all tasks, through an event, a queue sentinel, or a cancellation mechanism, so that each task knows to stop accepting new work and to finish or clean up its current work. Tasks should check for shutdown signals at safe points, such as between items, rather than only at the start, so that a long-running task can stop promptly, rather than continuing for a long time after shutdown was requested.
Timeouts for shutdown are also important. A task that ignores shutdown signals, or that takes a very long time to finish its current work, should not prevent shutdown forever, because a program that cannot stop is as problematic as one that cannot start. A shutdown timeout, after which remaining tasks are forcibly stopped or abandoned, with a log message noting that they did not finish cleanly, ensures that shutdown completes within a reasonable time, even if some tasks misbehave. That timeout should be generous enough to allow normal tasks to finish, while still bounding shutdown time, and it should be tested, by simulating a slow or stuck task and confirming that shutdown still completes.
Saving state during shutdown deserves attention, especially for programs that process data in batches or that maintain a queue of work. If a program is shut down while work remains in a queue, that work should either be completed before exit, saved for a later run, or explicitly discarded with a log message, depending on the program's purpose. Silently discarding queued work is a form of data loss, even if the program itself exits cleanly, because work that users expected to be done will not be done, and they may not realise it. Designing shutdown to handle queued work explicitly, by completing, saving, or reporting it, prevents that silent loss.
Finally, graceful shutdown should be tested, not just designed. Running a program, interrupting it at various points, and checking that files are consistent, that no orphan tasks remain, and that a rerun produces correct results, confirms that shutdown handling works in practice, not just in theory. That testing, like other concurrent testing, may need to be repeated with different timing, because shutdown behaviour can depend on when interruption occurs relative to task progress. A program that shuts down gracefully under test, at several interruption points, is a program that can be stopped safely in production, which is an important quality for long-running concurrent programs.
Check: Why should shutdown include a timeout, rather than waiting indefinitely for tasks to finish?
Section 36
Concurrency anti-patterns are common mistakes that seem helpful but cause problems, and recognising them helps you avoid them in your own programs. One anti-pattern is creating an unbounded number of threads or processes, one per task, without a pool or limit, which can exhaust memory and overwhelm external resources when task count spikes, as discussed earlier. That pattern is tempting because it is simple to code, but it is fragile in production, where bursts of work are common, and it should be replaced with a bounded pool and a queue, which provide backpressure and resource limits.
Another anti-pattern is sharing mutable state without synchronisation, on the assumption that races are unlikely, or that a particular operation is atomic when it is not. In Python, even a simple operation such as incrementing a counter involves several steps, as discussed earlier, and it is not atomic without protection, so assuming atomicity leads to lost updates and inconsistent results that appear only intermittently. That anti-pattern often arises from testing with low concurrency, where races do not occur, and it is prevented by designing for shared state explicitly, with locks, queues, or immutability, rather than by hoping that timing will be favourable.
A third anti-pattern is using concurrency for a task that is already fast enough sequentially, or for a task where concurrency does not help, such as pure Python computation with threads, where the GIL prevents speedup, as discussed earlier. That anti-pattern adds complexity, debugging difficulty, and potential bugs, for no benefit, and it is a form of premature optimisation, where a solution is applied before a problem is measured. Avoiding it requires measuring first, as discussed throughout this book, and only adding concurrency when measurement shows a real need, such as waiting time or responsiveness, that concurrency can address.
A fourth anti-pattern is ignoring shutdown, cancellation, and error handling in concurrent code, focusing only on the happy path where all tasks succeed and complete normally. A program that handles success well but hangs on shutdown, loses results on cancellation, or crashes when one task fails, is incomplete, because real concurrent programs encounter failures, cancellations, and shutdowns regularly. Designing for those cases from the start, with clear protocols for each, as discussed in this chapter, prevents a program that works in a demo but fails in production, where adverse conditions are the norm rather than the exception.
Finally, an anti-pattern to avoid is mixing concurrency models without understanding their interaction, such as calling blocking, synchronous code from an asynchronous task, which blocks the event loop, or sharing a non-thread-safe object between threads without protection, because it was designed for single-threaded use. Mixing models can be necessary, such as using a thread pool for blocking work inside an asynchronous program, but it should be done deliberately, with an understanding of how the models interact, and with explicit boundaries, such as running blocking work in a separate pool and communicating results through futures or queues. Unthinking mixing, by contrast, produces programs that are hard to reason about and prone to subtle bugs.
Check: Why is creating an unbounded number of threads per task an anti-pattern?
Section 37
Measuring the benefit of concurrency requires comparing a concurrent program's performance with its sequential baseline, using realistic workloads and meaningful metrics, rather than assuming that concurrency always helps. The most basic metric is elapsed time, or wall-clock time, for a complete job, measured for both sequential and concurrent versions, with the same input data and the same conditions, so that the comparison is fair. If the concurrent version is not significantly faster, or if it is slower, then concurrency may not be helping for that workload, and its complexity may not be justified, regardless of theoretical expectations.
Speedup, the ratio of sequential time to concurrent time, is a useful summary of benefit, but it should be interpreted carefully. A speedup close to the number of workers suggests that parallelism is working well, with little overhead, while a speedup far below the number of workers suggests overhead, bottlenecks, or a sequential portion that cannot be parallelised. That sequential portion is important: if a program spends a significant fraction of its time in sequential setup, result combination, or other non-parallelisable work, then even infinite workers cannot reduce total time below that sequential fraction, a principle known as Amdahl's law in its qualitative form. Understanding that principle helps you set realistic expectations for concurrency, and to focus optimisation on sequential bottlenecks, where it may have more impact than adding more workers.
Other metrics matter besides elapsed time. Throughput, such as items processed per second, is useful for servers and pipelines that handle a continuous stream of work, rather than a single job. Latency, such as time to handle a single request, matters for interactive programs, where users care about responsiveness for their own request, not just overall throughput. Resource use, such as processor, memory, and network utilisation, matters for efficiency and for cost, especially in environments where resources are limited or charged for. A concurrent program that is faster in elapsed time but uses far more memory may be a poor trade, if memory is scarce, while one that improves latency for users may be valuable even if its throughput is unchanged.
Measurement should be repeated, because concurrent performance varies with timing, load, and system conditions, and a single measurement may be misleading. Running each version several times, with the same data, and reporting typical results, rather than the best result, gives a more reliable comparison. It is also useful to measure with different numbers of workers, to find the point where adding workers stops helping or starts hurting, due to overhead or resource contention, as discussed for pool sizing.
Finally, measurement should include correctness, not just speed. A concurrent program that is faster but occasionally produces incorrect results, due to a race, is worse than a slower, correct sequential program, because its speed is illusory if its output cannot be trusted. Testing correctness under concurrent load, with stress tests and with checks for result completeness and ordering, as discussed earlier, should accompany performance measurement, so that speedup is not achieved at the cost of correctness. A concurrent program that is both faster and correct, as verified by testing and measurement, is the goal, and it is achieved by combining design, testing, and measurement, not by any one alone.
Check: In qualitative terms, what does Amdahl's law say about a program with a large sequential portion?
Section 38
Chunking is the division of work into pieces of a suitable size for concurrent processing, and choosing chunk size affects performance, load balancing, and overhead in important ways. If chunks are too small, the overhead of distributing work, communicating results, and managing tasks may dominate the actual work, making a concurrent program slower than a sequential one, despite parallel execution. If chunks are too large, load balancing suffers, because a worker assigned a large chunk may take much longer than others, leaving other workers idle while it finishes, and reducing overall throughput, especially if chunk durations vary unpredictably.
Dynamic chunking, where workers take new chunks as they finish previous ones, from a shared queue, helps with load balancing, because fast workers naturally take more chunks, while slow workers take fewer, and no worker sits idle while work remains, as long as chunks are small enough to be distributed evenly. That dynamic assignment, through a queue, is preferable to static assignment, where chunks are divided in advance and assigned to workers regardless of their speed, because static assignment cannot adapt to varying chunk durations or worker speeds. Queue-based dynamic chunking, discussed earlier for pools, is therefore a standard pattern for data-parallel work with variable chunk sizes.
Chunk size also affects result ordering and memory use. Small chunks produce many results, which must be collected and possibly sorted, adding overhead to result handling. Large chunks produce fewer results, but each result may be large, consuming memory while waiting to be combined. Balancing those factors, by choosing a chunk size that is large enough to amortise overhead but small enough to allow good load balancing and manageable result sizes, is a tuning decision that should be based on measurement with realistic data, rather than on a fixed rule, because optimal chunk size depends on the task, the data, and the system.
For tasks where chunk durations are highly variable, such as processing files of very different sizes, it may help to split large items into smaller chunks, or to sort items so that large and small items are interleaved, preventing a worker from receiving several large items in a row while others receive only small ones. Those strategies, while they add complexity, can improve load balancing significantly for skewed workloads, where a naive division would leave some workers overloaded and others idle. Recognising skew in data, by examining file sizes or record counts before processing, is a practical step in designing chunking for real workloads, not just for uniform test data.
Finally, chunking should be tested with realistic, skewed data, not only with uniform test data, because uniform data hides load balancing problems that skewed data reveals. A program that performs well on uniform test data but poorly on real, skewed data has a chunking problem, not a concurrency problem in general, and fixing chunking, by using dynamic queues and appropriate chunk sizes, may restore its performance without changing its concurrency model. That diagnosis, of chunking versus model, is a useful skill for tuning concurrent programs, and it illustrates why measurement with realistic data matters throughout concurrent design.
Check: Why does dynamic chunking through a queue balance load better than static assignment?
Section 39
The ecosystem of concurrency libraries and tools in Python is broad, and understanding its landscape at a high level helps you choose tools wisely, without needing to master them all in this concept chapter. At one end are low-level tools for threads and processes, which provide direct control over workers, locks, queues, and communication, and which are suitable for programs that need fine-grained control or that have unusual requirements. At the other end are high-level task libraries, which provide pools, futures, and task abstractions, hiding low-level details and allowing programs to be expressed in terms of tasks and results, which is simpler and less error-prone for most applications. Choosing between low-level and high-level tools involves balancing control against simplicity, with high-level tools preferred where they suffice.
Asynchronous libraries form another part of the ecosystem, providing event loops, task management, and asynchronous versions of common operations, such as network requests and file operations, designed to work with the event loop model. Those libraries are necessary for asynchronous programs, because ordinary, blocking libraries cannot be used directly in an asynchronous task without blocking the loop, as discussed earlier. When choosing an asynchronous library, it is important to check that it is designed for asynchronous use, that it handles cancellation and timeouts correctly, and that it is maintained, because an asynchronous library that blocks or that mishandles cancellation can undermine the benefits of asynchronous programming for a whole program.
There are also libraries for specific concurrent patterns, such as data processing pipelines, parallel map operations, or distributed computing across multiple computers, which extend concurrency beyond a single program or a single machine. Those libraries are beyond the scope of this chapter, but they build on the same concepts: tasks, queues, pools, and communication, applied at a larger scale. Recognising that distributed computing is, in many ways, concurrent programming with network communication and failure handling added, helps you see continuity between single-machine concurrency and larger systems, rather than viewing them as unrelated topics.
Tool choice should be guided by the same principles as model choice: simplicity, fit to the task, maintenance, and community support. A well-maintained, widely used library, with clear documentation and examples, is usually a better choice than an obscure library with a novel feature, because its behaviour is better understood, its bugs are more likely to have been found, and help is easier to obtain. A library that is no longer maintained, or that has known concurrency bugs, should be avoided, even if it seems convenient, because its problems will become your program's problems, especially under concurrent load where bugs are harder to diagnose.
Finally, the ecosystem evolves, with new libraries and new versions of existing libraries appearing regularly, and staying informed about major developments, while not chasing every new tool, is a sensible approach. Focus on mastering the concepts in this chapter, which transfer across tools, and learn specific tools as needed for particular projects, starting with high-level, well-supported ones, and moving to low-level tools only when a task requires them. That concept-first, tool-second approach ensures that your concurrency skills remain valuable even as specific libraries change, which they inevitably will over time.
Check: Why are high-level task libraries usually preferred over low-level thread tools for most applications?
Section 40
A concurrency checklist summarises the decisions and checks that should accompany any concurrent design, and using it systematically helps prevent the common mistakes discussed in this chapter. The first checklist item is need: is concurrency actually needed, based on measurement, for waiting time, responsiveness, or throughput, or would a sequential program suffice? If concurrency is not needed, do not add it. If it is needed, the next item is model: which model, threads, processes, or asynchronous tasks, fits the task's nature, its number of tasks, and its communication needs, as discussed in the decision framework earlier? Choosing a model deliberately, rather than by habit, is the foundation of a good concurrent design.
The second checklist area is shared state: what data is shared, how is it protected, and can sharing be minimised through immutability, task-local data, or queues? Listing shared state explicitly, and deciding a protection strategy for each item, turns an implicit risk into an explicit design, which can be reviewed and tested. That list should include external resources, such as files and databases, not just in-memory variables, because external sharing is just as risky as internal sharing, and it is often overlooked in initial designs.
The third checklist area is lifecycle: how are tasks started, how do they communicate results and errors, how are they cancelled, and how does the program shut down cleanly? Answering those questions for each task type, and testing shutdown, cancellation, and failure cases, as well as success cases, ensures that a concurrent program behaves well under adverse conditions, not just in a demo. Lifecycle design, often neglected in favour of happy-path logic, is what distinguishes a robust concurrent program from a fragile one, and it deserves explicit attention in any checklist.
The fourth checklist area is measurement and testing: has the concurrent program been compared with its sequential baseline, for both speed and correctness, under realistic load and with realistic, possibly skewed data? Has it been stress tested, reviewed for races, and tested for shutdown and cancellation? Those checks, performed before a concurrent program is considered complete, provide evidence that it is both faster and correct, rather than merely faster in theory or correct in a single test run. That evidence is what justifies the added complexity of concurrency, and it should be documented, so that future maintainers understand why concurrency was added and how it was verified.
Finally, the checklist should include documentation: does the program's documentation explain its concurrency model, its shared state strategy, its shutdown behaviour, and any limits, such as pool sizes or timeouts, that users or maintainers should know? Documenting concurrency decisions, which are often invisible in code, helps future maintainers avoid breaking them accidentally, such as by changing a pool size without understanding its tuning, or by adding shared state without protection. A concurrent program whose design is documented is a program that can be maintained safely, while one whose concurrency is undocumented is a trap for the next person who touches it.
Check: What four areas does the concurrency checklist in this section cover?
Section 41
Observability for concurrent programs means being able to see what tasks are doing, how long they take, and where they wait, through logs, metrics, and traces that are designed for concurrency. A log message from a concurrent program should include a task identifier, so that messages from different tasks can be distinguished when interleaved in a single log file. Without identifiers, a log becomes a confusing mixture, where it is impossible to tell which task produced which message, or to reconstruct the sequence of events for a particular task. That identifier, whether a task name, a number, or a request identifier, is a small addition that makes concurrent logs usable.
Metrics for concurrent programs include counts of tasks started, completed, failed, and currently running, queue lengths, waiting times, and resource utilisation, such as pool usage and connection counts. Those metrics, collected over time, reveal patterns that single log messages cannot, such as a queue that grows steadily under load, indicating that consumers cannot keep up with producers, or a pool that is frequently exhausted, indicating that its size is too small or that resources are leaking. Monitoring those metrics, with alerts for abnormal values, allows operators to detect concurrency problems before they become outages, such as by noticing a growing queue early and adding capacity or reducing load.
Traces, which record the path of a single request or task through a system, with timings for each step, are particularly valuable for concurrent programs that handle requests across several stages or services. A trace shows where time was spent, which stages waited, and where errors occurred, for a specific request, even when many requests are interleaved. That per-request view complements aggregate metrics, which show overall trends but not individual experiences, and it helps diagnose problems that affect only some requests, such as a slow stage that affects requests with particular data, while most requests are fast.
Logging, metrics, and traces should be designed with performance in mind, because observability itself can affect concurrency. Logging every operation in a high-throughput program can slow it down significantly, and metrics collection that requires locks on every operation can introduce contention that did not exist before. Sampling, where only a fraction of tasks are traced in detail, periodic aggregation, where metrics are collected at intervals rather than on every operation, and asynchronous logging, where log messages are queued and written by a separate task, all reduce observability overhead while preserving most of its value. That balance, between visibility and overhead, is a practical design decision for concurrent programs.
Finally, observability should be tested, by running a concurrent program under load and checking that its logs, metrics, and traces actually reveal what is happening, including during failures. A program whose observability looks good in a quiet test but becomes useless under load, because logs are overwhelmed or metrics are not updated, has an observability problem that will only become apparent when it is most needed, during an incident. Testing observability under load, like testing other aspects of concurrency, ensures that it works when it matters most, and it is a fitting complement to the testing and review practices discussed earlier.
Check: Why should log messages in a concurrent program include a task identifier?
Section 42
Testing concurrent code in isolation means testing its components separately from its concurrency, so that logic errors can be distinguished from concurrency errors. A function that processes an item, for example, can be tested sequentially, with known inputs and expected outputs, without involving threads, processes, or an event loop. That test confirms that the function's logic is correct, regardless of concurrency, and it is deterministic, because it does not depend on timing or interleaving. Once logic is confirmed correct in isolation, concurrency testing can focus on interaction: whether tasks are distributed correctly, whether results are collected correctly, and whether shared state is protected, rather than on whether the underlying logic works.
Isolation also applies to concurrency mechanisms themselves. A queue can be tested by putting and taking items from several tasks in a controlled way, checking that items are not lost, duplicated, or reordered beyond what the queue guarantees. A pool can be tested by submitting tasks with known durations and checking that they complete, that errors are reported, and that shutdown waits for completion. Those mechanism tests, focused on a single component, are easier to reason about and to make deterministic, through controlled scheduling or through small, focused scenarios, than tests of a whole concurrent program, where many mechanisms interact at once.
Mocking, or replacing external resources with controlled substitutes, is another isolation technique for concurrent testing. A task that normally downloads from a network can be tested with a mock that returns predetermined data after a predetermined delay, allowing test timing to be controlled and results to be predicted. That control makes it possible to test scenarios that are hard to reproduce with real networks, such as a slow response, a timeout, or a failure in the middle of a download, and to confirm that the program handles those scenarios correctly, without needing to reproduce them in a real environment. Mocking does not replace testing with real resources, but it complements it by making edge cases testable.
Deterministic scheduling for tests, where task order is controlled rather than left to the system, is a more advanced isolation technique. By running tasks in a fixed order, or by using barriers to force particular interleavings, as discussed for debugging, tests can reproduce specific concurrency scenarios reliably, turning a probabilistic test into a deterministic one for that scenario. That determinism is valuable for regression testing, because a test that fails consistently after a change indicates a real regression, rather than a flaky test caused by timing variation. Building deterministic tests for key concurrency scenarios, alongside stress tests for broader coverage, gives a test suite both reliability and breadth.
Finally, isolated testing should not create a false sense of security. A program whose components pass in isolation may still fail when integrated concurrently, because integration introduces interactions that isolation removes, such as contention for a shared resource, or ordering effects between components. Isolated testing is therefore a foundation, not a complete guarantee, and it should be complemented by integration testing under concurrent load, with realistic data, as discussed earlier. That layered approach, isolation first, integration second, is the most practical way to test concurrent programs, balancing determinism against realism.
Check: Why test a task's logic sequentially before testing it concurrently?
Section 43
Real-time and batch concurrency differ in their goals and constraints, and recognising that difference helps you choose appropriate designs for each. Real-time concurrency, as in interactive programs, servers, and control systems, must respond within deadlines, such as a user interface that must update promptly, or a server that must respond to requests within a reasonable time. For real-time programs, latency matters more than overall throughput: a single slow response is noticeable and harmful, even if overall throughput is high, because users experience delays directly. Designing for real-time concurrency therefore prioritises responsiveness, with timeouts, prioritisation, and graceful degradation under load, so that deadlines are met as often as possible, even when the system is busy.
Batch concurrency, as in data processing jobs, backups, or nightly reports, processes a large amount of work without strict per-item deadlines, and its main goal is throughput: completing the whole job as quickly as possible, or as efficiently as possible, given available resources. For batch programs, latency for a single item matters less, because users do not wait for individual items; they wait for the whole job to finish. Designing for batch concurrency therefore prioritises total time and resource efficiency, with chunking, load balancing, and pool sizing tuned for throughput, rather than for per-item responsiveness. A batch program that is slower per item but faster overall, due to better parallelism, is a success, while a real-time program with the same trade-off might be a failure, because its users would notice the slower per-item responses.
Those different goals affect design choices. A real-time program might limit concurrency to protect latency, by refusing new work when overloaded, rather than accepting it and slowing all responses, as discussed for flow control. A batch program might accept all work, queue it, and process it as fast as resources allow, because there is no user waiting for an immediate response, and queuing does not harm user experience in the same way. A real-time program might prioritise certain tasks, such as user input handling, over background work, to keep the interface responsive, while a batch program might treat all items equally, processing them in any order that maximises throughput.
Scheduling also differs. Real-time programs often need fair scheduling, so that no task starves, as discussed earlier, because starvation in a real-time program means that some users' requests are never handled, which is a serious failure. Batch programs can tolerate some unfairness, if it improves overall throughput, such as by processing small items first to clear a queue quickly, though even batch programs should avoid starving large items indefinitely, because that would mean that some work never completes, which is also a failure, albeit a less visible one. Fairness, like other concurrency properties, should be judged by the program's purpose, not by an absolute rule.
Finally, many programs combine real-time and batch elements, such as a server that handles interactive requests in real time, while also running batch jobs in the background, such as generating reports or cleaning up data. Those combined programs must balance real-time responsiveness against batch throughput, often by limiting batch concurrency when real-time load is high, and allowing it to increase when real-time load is low.
Check: How do real-time and batch programs differ in whether latency or throughput matters more?
Section 44
Concurrency introduces security considerations, because concurrent programs handle many tasks, connections, and resources at once, and an attacker may exploit that concurrency to overwhelm a system or to access data improperly. One classic attack is resource exhaustion, where an attacker opens many connections, submits many tasks, or requests many expensive operations, consuming a server's threads, processes, memory, or connections, until legitimate users cannot be served. That attack, sometimes called denial of service, succeeds not by breaking code, but by exploiting a program's willingness to accept unlimited concurrent work, which is why limits, such as connection limits, task limits, and rate limits, discussed earlier for flow control, are also security measures, not just performance measures.
Another security consideration is race conditions that can be exploited, where an attacker times requests to exploit a window between checking a condition and acting on it, such as checking whether a user is authorised and then performing an action, with the authorisation changing in between. Those time-of-check to time-of-use races, as they are called, are a specific form of race condition with security implications, because they can allow an action that should have been denied, if the check and the action are not atomic. Preventing such races requires making check and action atomic, through locks, transactions, or other synchronisation, so that no other task can change the relevant state between them. That atomicity, for security-critical checks, is more important than for ordinary operations, because its failure can grant unauthorised access, not just an incorrect count.
Shared state security also matters. A concurrent program that caches user data, or that reuses objects between tasks, must ensure that one user's data does not leak to another user's task, through a shared cache, a reused buffer, or a thread-local variable that is not properly cleared. That leakage, known as cross-user data exposure, can occur when a resource is returned to a pool without clearing its previous contents, or when a cache key does not include a user identifier, causing one user's cached data to be served to another. Designing pools, caches, and task-local storage with user isolation in mind, by clearing resources on return and by including user context in cache keys, prevents such leakage.
Rate limiting and authentication, while not strictly concurrency mechanisms, interact with concurrency, because they must themselves be thread-safe and must handle concurrent requests correctly. A rate limiter that counts requests without synchronisation may undercount under concurrency, allowing an attacker to exceed limits by sending many requests at once, each of which reads the same count before any increments it. A rate limiter implemented with atomic operations or with synchronisation, as discussed for counters, prevents that bypass, ensuring that limits hold even under concurrent load.
Finally, security testing for concurrent programs should include adversarial scenarios, such as many simultaneous requests from a single source, requests timed to exploit check-and-act windows, and requests that attempt to access other users' data through shared resources. Those scenarios, tested deliberately, reveal concurrency security issues that ordinary functional testing, with a single user and sequential requests, would miss. Security, like correctness generally, must be designed for concurrency, not assumed to hold because sequential security controls work, because concurrency changes timing, sharing, and resource use in ways that can undermine controls that are correct sequentially but not concurrently.
Check: How can an attacker exploit unlimited concurrency to deny service to legitimate users?
Section 45
Command-line tools benefit from concurrency in specific, practical ways, even though they are often run once and exit, rather than running continuously like servers. A command-line tool that downloads many files, processes many records, or checks many items can use concurrency to finish faster, by interleaving waiting tasks, as discussed throughout this chapter. That speedup is directly visible to users, who wait less for a command to complete, and it is a common reason to add concurrency to a tool that has become slow as its data grows. For a tool used interactively, that speedup improves the user experience, while for a tool used in scripts, it reduces total workflow time, which can be significant when the tool runs frequently.
Concurrency in command-line tools should be exposed through options, not hard-coded, so that users can control it. An option for the number of workers, with a sensible default, allows users to tune concurrency for their system and their data, such as using fewer workers on a small machine, or more workers for a waiting-heavy task on a fast network. Providing a default that works well for typical use, and an option to override it, balances ease of use with flexibility, as with other command-line options discussed in earlier chapters.
Progress reporting and cancellation are particularly important for concurrent command-line tools, because users need to know that a long, concurrent job is progressing, and they need a way to stop it cleanly if it takes too long or if they realise they made a mistake. Progress reporting, as discussed for long-running programs, should show how many items have been completed, how many remain, and whether any failed, without flooding the screen with a message per item. Cancellation, through an interrupt signal, should trigger graceful shutdown, as discussed earlier, finishing or saving completed work and exiting with a clear message, rather than leaving partial output that looks complete.
Error handling in concurrent command-line tools also needs care. If one item fails while others succeed, the tool should report which items failed and why, and it should exit with a non-zero code if the overall job did not fully succeed, even if most items succeeded, so that scripts can detect partial failure. That reporting, with a summary at the end listing successes, failures, and skips, turns a concurrent tool's output from a confusing interleaving of messages into a clear summary that users and scripts can act on. Logging failures to a file, for detailed investigation, while summarising them on screen, is a good balance for tools that process many items.
Finally, concurrent command-line tools should be tested with realistic data and with different worker counts, to confirm that concurrency actually helps for their typical workload, and that results are correct regardless of worker count or task order. A tool whose output depends on worker count, or whose results vary from run to run due to ordering, is a tool whose concurrency has introduced non-determinism that users did not expect, and it should be fixed, by sorting results or by documenting that order may vary, as discussed for ordering. That testing, with worker count as a variable, is a specific form of concurrent testing for command-line tools, and it ensures that concurrency is a benefit, not a source of confusion.
Check: Why should a concurrent command-line tool expose worker count as an option rather than hard-coding it?
Section 46
Reviewing concurrent code requires a checklist that goes beyond ordinary code review, focusing on the specific risks that concurrency introduces. The first review question is about shared state: what data is shared between tasks, is it mutable, and how is it protected? A reviewer should be able to list every piece of shared mutable state, and for each, identify its protection mechanism, whether a lock, a queue, immutability, or confinement to a single task. If shared state cannot be listed, or if its protection is unclear, the code is not ready, because its concurrency design has not been made explicit, and hidden sharing is likely to cause races.
The second review question is about locks and ordering: if locks are used, are they always acquired in the same order, are they always released, even on error, and are they held for the shortest time necessary? A reviewer should check that lock acquisition and release are paired reliably, such as through context managers that release automatically, rather than through manual calls that can be forgotten on an error path. They should also check for nested locks, where one lock is acquired while holding another, because nested locks are a common source of deadlock, and their ordering should be consistent across the codebase, not just within a single function.
The third review question is about lifecycle and shutdown: how are tasks started and stopped, how are errors reported, and what happens on cancellation or shutdown? A reviewer should check that tasks are not left running after the main program exits, that errors in one task do not silently disappear, and that shutdown waits for tasks to finish or cancels them cleanly, with a timeout, as discussed earlier. Those lifecycle checks are easy to overlook when focusing on happy-path logic, but they are where many concurrent bugs hide, because they involve interactions between tasks that only occur at start-up, shutdown, or failure, which are less frequently tested than normal operation.
The fourth review question is about testing: have concurrent scenarios been tested, with stress tests, with deterministic interleaving tests for suspected races, and with tests for shutdown and cancellation, as discussed earlier? A reviewer should not accept a concurrent change based solely on sequential tests, or on a single successful run, because those do not exercise concurrency. They should ask for evidence of concurrent testing, and for reasoning about why the design is correct, not just that it passed a test once. That evidence, combined with the reviewer's own reasoning, provides the best available assurance for concurrent code, short of formal verification, which is beyond most projects.
Finally, review should consider simplicity: is concurrency necessary for this change, is the chosen model the simplest that fits, and could the same benefit be achieved with less concurrency, such as by optimising sequential code or by batching? A reviewer who asks those questions helps prevent unnecessary concurrency from entering a codebase, which is valuable, because every concurrent component adds maintenance burden and risk, even when it is correct. A review that approves concurrency only when it is justified, well-designed, and well-tested, is a review that protects a codebase's long-term health, not just its immediate functionality.
Check: What should a reviewer be able to list first when reviewing concurrent code?
Section 47
Concurrency concepts transfer across programming languages, even though specific tools and syntax differ, because the underlying problems, of shared state, timing, synchronisation, and scheduling, are universal. A programmer who understands race conditions, locks, queues, and event loops in Python can recognise the same concepts in another language, even if that language uses different names, such as mutexes instead of locks, channels instead of queues, or promises instead of futures. That transferability is valuable, because it means that time spent learning concurrency concepts in Python is not wasted when learning another language, and it allows you to read concurrent code in other languages with understanding, even if you cannot write it fluently yet.
Different languages emphasise different concurrency models, based on their design and history. Some languages emphasise threads, with rich threading libraries and synchronisation primitives. Others emphasise message passing, where tasks communicate only through messages, never through shared memory, which eliminates many race conditions by design, at the cost of explicit message handling. Still others emphasise asynchronous programming, with event loops and coroutines as their primary model, similar to Python's asyncio but with different syntax and libraries. Recognising those emphases helps you understand why a language's concurrent code looks the way it does, and why its community recommends certain patterns over others, rather than assuming that Python's patterns are universal.
The GIL, discussed earlier, is specific to Python's standard implementation, and other languages may not have an equivalent limitation, or may have different limitations, such as a different memory model that affects how shared data is visible across threads. That difference means that a concurrency design that is optimal in Python, such as using processes for compute-heavy parallelism, may not be optimal in another language, where threads can parallelise computation effectively. Transferring concepts, therefore, requires also transferring an understanding of each language's execution model, so that design choices are adapted to the new context, rather than copied blindly from Python.
Despite those differences, the design principles from this chapter transfer well: minimise shared mutable state, prefer communication through queues or messages, protect what sharing remains, design for shutdown and cancellation, test under load, and measure before optimising. Those principles are not Python-specific; they are good concurrent design in any language, because they address the universal problems of timing, sharing, and resource management. A programmer who internalises those principles can learn a new language's concurrency tools more quickly, because they will recognise the problems those tools solve, even if the tools themselves are new.
Finally, learning concurrency in multiple languages, over time, deepens understanding of concurrency itself, by revealing which aspects are essential and which are incidental to a particular language's implementation. That deeper understanding, in turn, improves concurrent design in Python, because it allows you to see Python's models more clearly, as choices among alternatives, rather than as the only way to do concurrency. That perspective, gained through transfer, is a long-term benefit of studying concurrency conceptually, as this chapter has done, rather than only learning a single library's syntax.
Check: Why do concurrency concepts transfer across languages despite different tools?
Section 48
Future directions in concurrency include improvements in tools, in language support, and in hardware, but their details are uncertain, and this chapter will discuss them only at a high level, without predicting specific changes. One direction is better tooling for detecting race conditions, deadlocks, and other concurrent bugs automatically, through static analysis, which examines code without running it, or through dynamic analysis, which monitors a running program for problematic patterns. Those tools, as they improve, may make concurrent programming safer, by catching bugs that testing and review miss, though they are unlikely to replace careful design entirely, because concurrent correctness involves reasoning about intent, not just about patterns.
Another direction is language support for safer concurrency, such as type systems that track ownership of data, or that prevent shared mutable state by construction, making certain races impossible to write, rather than merely detectable after the fact. Those language features, where they exist, shift the burden of concurrency safety from the programmer's discipline to the language's guarantees, which can reduce bugs significantly, though they also require learning new concepts and may restrict programming style in ways that feel constraining at first. Python's approach, which relies on discipline and on libraries rather than on language-enforced ownership, is different, but understanding alternative approaches helps you appreciate Python's trade-offs and to use its tools more carefully.
Hardware trends also affect concurrency, as computers gain more cores, faster networks, and new kinds of processors specialised for particular tasks, such as graphics or machine learning. More cores make parallelism more valuable, because there is more parallel capacity to exploit, but they also make concurrency bugs more likely to manifest, because more tasks can run truly in parallel, exposing races that were hidden when tasks mostly interleaved on a single core. That trend means that concurrency skills will become more important, not less, as hardware evolves, because programs that ignore concurrency will leave increasing amounts of hardware capacity unused, while programs that use it incorrectly will fail more visibly.
Distributed concurrency, where tasks run on multiple computers connected by a network, is another growing area, building on single-machine concurrency concepts but adding network failures, latency, and partial failures, where some computers fail while others continue. Those distributed challenges are beyond this chapter, but they follow from the same principles: tasks, communication, shared state, and failure handling, applied at a larger scale with less reliable communication. Understanding single-machine concurrency well is a prerequisite for distributed concurrency, because distributed systems add complexity to concepts that must first be understood in a simpler setting.
Finally, the most reliable future direction is continued emphasis on fundamentals: clear task boundaries, minimal shared state, explicit communication, careful lifecycle design, and measurement-based tuning. Those fundamentals have remained relevant through many changes in tools, languages, and hardware, and they are likely to remain relevant through future changes as well, because they address the essential problems of concurrency, not incidental details of a particular tool. Investing in fundamentals, as this chapter has, is therefore a durable investment, whatever specific concurrency tools become popular in the future.
Check: Why will concurrency skills become more important as computers gain more cores?
Section 49
A download manager is a good case study for concurrent design, because it combines many ideas from this chapter in a realistic, understandable program. Imagine a tool that downloads many files from a network, with a limit on simultaneous downloads, progress reporting, retry on failure, and graceful shutdown. Its design starts with a queue of download tasks, each representing a file to fetch, and a pool of workers, each of which takes a task, downloads its file, and reports success or failure. That queue-and-pool structure, discussed earlier, provides backpressure, load balancing, and a clear place to implement limits, retries, and progress tracking.
Choosing a concurrency model for a download manager depends on its workload. Downloads are waiting-heavy, spending most time waiting for network data, so threads or asynchronous tasks are both suitable, while processes would be wasteful, because they add overhead for a task that does not need parallel computation. For a moderate number of downloads, threads are simple and effective, because each download can be written sequentially, with the thread system handling interleaving. For a very large number of downloads, asynchronous tasks may be more efficient, because they are lightweight, but they require asynchronous network code, which adds complexity. That model choice, based on expected download count and on available libraries, is a practical application of the decision framework from earlier in this chapter.
Progress reporting for a download manager should show overall progress, such as how many files have completed, how many remain, and how many failed, as well as per-file progress for large files, if useful, without flooding the screen. That reporting, through a shared progress structure protected by synchronisation, or through a queue of progress updates processed by a single reporting task, illustrates shared state management in a concrete setting: progress is shared state, updated by many workers, and it must be protected or confined to avoid races that would corrupt counts. Designing progress reporting as a single owner task, receiving updates through a queue, is a clean solution that avoids locks on shared counters.
Retries and failures in a download manager also illustrate concurrent design. A download that fails due to a transient network error should be retried, with a pause and a limit on retries, rather than failing immediately, because transient failures are common in networks. Retries should be counted per file, not globally, so that a persistently failing file does not cause endless retries that block other files. Failed files, after retries are exhausted, should be reported in a summary at the end, with their error messages, so that users can investigate them, rather than being silently skipped or causing the whole job to fail because one file failed. That per-item error handling, with a summary, is a pattern that applies to many concurrent tools, not just download managers.
Finally, graceful shutdown for a download manager means that interrupting it should finish or save completed downloads, report which files were completed, which were in progress, and which were not started, and exit cleanly, so that a rerun can resume or retry only the unfinished files, rather than redownloading everything. A download manager built with those considerations is a good example of concurrent design done well, and it is a useful model for other concurrent tools.
Check: In a download manager, why should retries be counted per file rather than globally?
Section 50
This chapter has introduced concurrency at concept level, covering threads, processes, and asynchronous programming, their strengths and limitations, and the design principles that make concurrent programs correct and maintainable. You have learned about race conditions, locks, queues, worker pools, futures, timeouts, cancellation, deadlock, and other failure modes, as well as about shared state, ordering, flow control, and graceful shutdown. Those concepts, taken together, form a mental model for concurrent programming that will help you recognise concurrent designs, evaluate their trade-offs, and participate in building them, even before you master any single concurrency library in depth.
The central theme of this chapter is that concurrency is a design choice with costs as well as benefits, and that it should be adopted deliberately, based on measurement and on a clear understanding of the task. Concurrency can exploit waiting time, improve responsiveness, and increase throughput, but it also adds complexity, introduces timing-dependent bugs, and requires careful handling of shared state, lifecycle, and failure. A program that uses concurrency well is one where those costs have been considered and managed, through simple designs, explicit shared state strategies, and thorough testing, not one where concurrency was added as an afterthought or as a generic optimisation.
Looking ahead, the next chapters will build on the design thinking developed here, in different domains. Security, covered next, will address how to protect programs and data, including concurrent programs, from misuse and attack. Performance and profiling will address how to measure and improve programs, including concurrent ones, with evidence rather than guesswork. Those chapters, like this one, emphasise principles, measurement, and careful design over specific tricks, and they will deepen your ability to write programs that are not only correct, but also robust, efficient, and trustworthy in real use.
As with previous chapters, the best consolidation is practice with a small, controlled concurrent program, such as a download manager or a file processor, built first sequentially, then with a simple pool and queue, and tested for correctness, ordering, shutdown, and performance, as discussed throughout this chapter. That practice, with a real but manageable program, will confront you with the timing, sharing, and lifecycle issues that only real concurrent code reveals, and it will turn the concepts in this chapter from abstract ideas into practical skills. Start simple, measure, and add complexity only when needed, and your concurrent programs will be both effective and reliable.
Keep this chapter's checklist close when you next face a slow or unresponsive program, and use it to ask whether waiting, computation, or responsiveness is the real problem before choosing a concurrency model.
A program that is correct sequentially, measured honestly, and made concurrent only where it helps will usually outperform a program that was made concurrent first and debugged later.
Check: What is the central theme of this chapter's approach to concurrency?
Practical Task
Build a small concurrent file-processing design on paper, then implement a simple sequential version first. Choose a task such as summarising many small text files, design a queue-and-pool version with a worker limit, progress counts, per-item error reporting, and graceful shutdown, and describe how you would test it for race conditions, ordering, and shutdown. Implement the sequential version, measure it on sample data, and write a short note explaining whether concurrency would help for your data and which model you would choose, with reasons.
Chapter Questions
- Compare threads, processes, and asynchronous tasks: when does each help, and what are its main costs?
- Explain race conditions, deadlock, and starvation, and describe a design strategy that helps prevent each.
- How do queues, worker pools, and futures work together in a task-based concurrent design?
- A program downloads many files sequentially and is slow. Analyse whether concurrency would help, which model fits, and what risks you would manage.
- Design a concurrent tool of your own: state its task, its concurrency model, its shared state strategy, and how you would test shutdown and failure.