Chapter 18 — Standard Library Essentials

Section 1

Python's standard library is the collection of modules that ships with every Python installation. It is often summed up with the phrase "batteries included", because it provides ready-made tools for many common programming tasks without you having to install additional packages from external sources. Those tools cover working with files and directories, manipulating text and data structures, performing mathematics and statistics, handling dates and times, working over a network, and testing programs. Learning the standard library well is one of the most valuable investments a Python programmer can make, because it prevents you from reinventing tools that already exist in tested and documented form. That prevention of reinvention saves time and reduces bugs in real programs that would otherwise need custom implementations for tasks other programmers have already solved.

The standard library is organised into modules that are imported by name, such as import os, import sys, or from pathlib import Path. Each module provides a focused set of functions, classes, and constants for a specific area: file system operations for os, path manipulation for pathlib, and so on. That organisation by area helps you find the right tool for a given task by thinking about which area the task belongs to, rather than searching through one large collection of unrelated functions. The standard library documentation is organised in the same way, by module and by area, so you can look up a module by its name and find all of its tools documented together in one place. Browsing a module's overview before you dive into a single function is usually the fastest way to discover what it can do.

Learning the standard library should be approached by learning a small set of essential modules thoroughly first, such as those covered in this chapter, and then expanding your knowledge gradually as new tasks require new modules. Trying to memorise the entire library at once would be overwhelming and counterproductive, because most of it will not be relevant to the program you are writing today, and unused knowledge fades quickly. A gradual approach works well because the standard library's modules follow consistent conventions for naming and behaviour, such as using functions for simple operations and classes for more complex operations that keep state. Learning one module well therefore makes learning the next one easier, because its conventions will feel familiar.

import sys
print(sys.version.split()[0])

Start with the modules in this chapter, and add new ones only when a real task demands them. When a task feels familiar, pause and ask whether a standard module already solves it, and check the documentation before writing your own version. Over time, that habit builds a working knowledge that grows from use rather than from memorisation, and it leaves you free to spend your effort on the logic that is unique to your program.

Check: Why is learning a small set of essential standard library modules thoroughly a better strategy than trying to memorise the whole library at once?

Section 2

The os and sys modules provide tools for interacting with the operating system and with the Python interpreter itself. The os module covers the environment a program runs in: the current working directory, environment variables, file system paths, and process information. The sys module covers the interpreter: the command-line arguments, the standard streams, the module search path, and the exit status a program returns. Together they form the bridge between a Python program and the world outside it, and almost every program that is meant to run in different places will use at least one of them. Understanding them early makes later topics, such as configuration and command-line tools, much easier to grasp.

Environment variables deserve particular attention, because they are a common way to configure programs without hard-coding values into source code. A program that reads its database location, its output directory, or its operating mode from the environment can run unchanged in development, in testing, and in production, simply by changing the environment around it. The safe way to read such a value is os.environ.get("NAME"), which returns None when the variable is absent, or a default you supply, such as os.environ.get("MODE", "development"). Indexing directly with os.environ["NAME"] raises an error when the variable is missing, which turns a simple configuration mistake into a crash. Hard-coding the value is worse still, because it bakes one environment's details into code that will be shared, copied, or deployed elsewhere.

Command-line arguments are accessed through sys.argv, a list whose first element is the program's name as invoked, and whose later elements are the arguments that followed it. A program run as python tool.py input.txt --verbose would see those strings in order, and it is the program's job to interpret them. For a tiny script, reading sys.argv directly is perfectly reasonable, and you will see that pattern throughout this chapter. For anything larger, with options, flags, and help text, a dedicated argument parser is a better choice, and that topic is covered later in this chapter. The key point here is that arguments arrive as plain strings, so a program that expects a number or a path must convert and validate them itself.

import os
import sys
print("args:", sys.argv)
print("mode:", os.environ.get("MODE", "development"))

Treat configuration as input, not as part of your code. Read it from the environment with a sensible default, fail with a clear message when a required value is genuinely missing, and keep secrets out of source code entirely. Programs written this way are easier to test, because a test can set an environment variable and run the program, and easier to deploy, because nothing in the code needs to change when the surroundings change.

One further habit is worth forming: when a required value is missing, report its name and where it was expected, rather than letting the program fail later with a confusing error far from the cause. A clear message at the point of reading configuration saves debugging time for everyone who runs the program.

Check: Why should programs read configuration from environment variables with os.environ.get(), rather than hard-coding values or indexing directly?

Section 3

Paths are one of the most common sources of small, annoying bugs in programs that work with files. Joining directory names with string concatenation, or with the os.path functions, works, but it leaves the programmer responsible for separators, trailing slashes, and the differences between operating systems. The pathlib module offers a different approach: a path is an object, not a string, and operations on it are methods of that object. That shift in thinking is the reason pathlib is generally preferred for new programs. Instead of remembering which helper function to call, you work with a Path object that knows how to join, inspect, and manipulate itself.

A Path is built by passing its parts to the constructor, or by joining parts with the / operator, which reads naturally from left to right. Once you have a Path, you can ask it for its name, its suffix, its parent directory, or whether it exists, all through readable attributes and methods. You can read text or bytes directly with read_text() and read_bytes(), write with write_text(), iterate over a directory's entries with iterdir(), and create directories with mkdir(). Each of these operations returns or accepts Path objects, so a program tends to stay in the world of paths throughout, converting to a string only at the boundary where some other tool demands one.

The older os.path module is not wrong, and you will meet it in existing code for a long time. Its functions take strings and return strings, which means every step of a path operation is a separate function call on a plain value. That style is harder to read when operations chain together, and it is easier to make a mistake, such as joining a path that already ends in a separator, or forgetting that a function expects a string rather than a bytes object. pathlib removes a whole class of such mistakes by keeping the path's structure inside the object. It also handles the operating system's separator for you, so the same code works on different systems without special cases.

from pathlib import Path
folder = Path("data") / "reports"
print(folder.name)
print(folder.suffix)
print(folder.parent)

Prefer pathlib for new code, and reach for os.path mainly when you are maintaining code that already uses it, or when an interface you must call accepts only strings. When you do convert, str(path) gives you the string form explicitly, which keeps the conversion visible rather than hidden. Thinking of paths as objects, with behaviour of their own, is a small change that makes file-handling code shorter, clearer, and less prone to the separator and joining errors that plague string-based path code.

A final practical point is that pathlib objects work smoothly with most modern tools, and when a tool insists on a string, converting explicitly at that boundary keeps the conversion visible. That explicitness matters, because a hidden conversion can reintroduce exactly the separator confusion that pathlib was meant to remove.

Check: Why is pathlib generally preferred over string-based os.path manipulation for new programs?

Section 4

Counting and grouping are tasks that appear in almost every program that processes data: counting how often each word appears in a text, grouping records by category, or tallying votes, errors, or events. Writing those loops by hand is straightforward, but it is also repetitive, and the repetitive version hides the intent behind bookkeeping. The collections module provides two tools that make the intent visible. Counter counts hashable items, and defaultdict supplies a default value automatically whenever a missing key is accessed. Both replace a small manual loop with a single, readable object, and both are worth reaching for whenever you catch yourself writing counting or grouping code.

A Counter is a dictionary subclass whose values are counts. You can build one directly from an iterable, and it will count each distinct item for you, or you can add items one at a time with update(). Its most_common() method returns the most frequent items in order, which is exactly what many counting tasks need as their final answer. Because it is a dictionary, you can also look up the count of a single item, iterate over its keys, or add two counters together. The important point is that the counting logic, including the handling of first occurrences, lives inside the class, so your code states what it is counting rather than how the counting is performed.

A defaultdict solves the grouping problem in a similar way. An ordinary dictionary raises an error when you try to append to a list stored under a key that does not exist yet, so grouping code usually needs a check: if the key is absent, create an empty list first. A defaultdict(list) creates that empty list automatically on first access, so the grouping loop becomes a single line that appends to the right group. The factory you pass, such as list, int, or set, decides what kind of default value is created, and the result is still an ordinary dictionary in every other respect, so it can be passed to code that expects one.

from collections import Counter, defaultdict
words = ["red", "blue", "red", "green", "blue", "red"]
print(Counter(words).most_common(2))
groups = defaultdict(list)
for word in words:
    groups[word[0]].append(word)
print(dict(groups))

Reach for Counter when the question is how many times each item occurs, and for defaultdict when the question is how to collect items into groups. Keep the manual loop for cases where the logic is genuinely more complex than counting or grouping, such as when an item's group depends on several conditions at once. Choosing the ready-made tool when it fits is not laziness; it is clarity, because the next reader sees the purpose of the code immediately instead of having to reconstruct it from loop bookkeeping.

Check: When should you reach for Counter or defaultdict instead of writing manual counting or grouping loops?

Section 5

The itertools module provides tools for working with iterators, and three of its functions deserve special care because they are easy to misuse. permutations() yields every possible ordering of a collection, and combinations() yields every possible selection of a given size, ignoring order. Both are invaluable when a problem genuinely requires examining arrangements, such as testing every ordering of a small set of tasks, but both grow extremely quickly as the input grows. The number of permutations of ten items is already in the millions, and each additional item multiplies the total again. A program that works instantly on five items can therefore run effectively forever on fifteen, not because the code is wrong, but because the problem itself explodes. The lesson is to check the size of the input before reaching for these functions, and to look for a smarter approach, such as a direct formula or a search that prunes bad options early, whenever the input is large.

accumulate() is a quieter tool, and its value is easier to miss. Where sum() collapses an entire iterable into a single total, accumulate() yields each running total in turn: first the first item, then the first two added together, then the first three, and so on. That sequence of partial results is exactly what you need when a program must track a balance over time, report progress after each step, or find the point at which a cumulative total crosses a threshold. By default it adds, but you can supply another function, such as multiplication, to accumulate in a different way. The result is an iterator, so it produces values lazily and does not build a large list unless you ask it to.

These tools share a common theme: they are lazy. They yield one value at a time, which means a program can stop early, take only the first few results, or process a stream without holding everything in memory. That laziness is what makes them efficient, but it also means you cannot index into them or measure their length directly; convert to a list only when you genuinely need all the values at once. It also means that a careless use of permutations() on a large input will not fail fast with an error, but will simply keep producing values for a very long time, so the responsibility for checking input size rests with you.

from itertools import combinations, accumulate
print(list(combinations(["a", "b", "c"], 2)))
print(list(accumulate([10, 20, 30])))

Use permutations() and combinations() for small, exhaustive searches, and pause to estimate the count before using them on anything larger. Use accumulate() whenever the intermediate totals matter, not just the final sum, and remember that sum() remains the right tool when only the final total is needed. Choosing between them is a matter of asking what answer the program actually requires, and then picking the tool that yields exactly that answer, no more and no less.

Check: Why do permutations() and combinations() need care with large inputs, and what does accumulate() yield that sum() does not?

Section 6

The functools module collects higher-order tools: functions that work with other functions. Two of them, partial() and reduce(), are frequently misunderstood, because each has a simpler alternative that is often better. partial() creates a new function by fixing some arguments of an existing function, leaving the rest to be supplied later. If a program repeatedly calls a function with the same first argument, partial() captures that repetition in one place, so the call sites stay short and the fixed value is visible in exactly one spot. A lambda wrapper can do the same job, but it hides the original function behind an anonymous one, which makes the resulting object harder to inspect, harder to document, and harder to recognise in an error message. When the goal is simply to preset arguments, partial() states that goal directly, and its result carries the original function's name and documentation with it.

reduce() is the opposite case: a powerful tool that is often the wrong choice. It folds an iterable into a single value by repeatedly applying a two-argument function, which can express almost any aggregation. The trouble is that most aggregations a program actually needs already have a built-in form that says what is meant: sum() for a total, max() and min() for extremes, any() and all() for truth tests, and a simple loop or comprehension for anything more specific. Those built-ins are shorter, faster to read, and harder to get wrong, because the reader does not have to simulate the folding process to understand the result. reduce() earns its place only when the operation is genuinely custom and cannot be expressed clearly any other way, such as combining objects with an unusual rule that no built-in captures.

The guiding principle is readability for the next person, who may be you in six months. A lambda wrapper that merely presets arguments adds a layer of indirection without adding meaning, so partial() wins. A reduce() call that merely sums a list adds a layer of mechanism without adding meaning, so sum() wins. In both cases the question to ask is whether the tool makes the intent clearer or merely demonstrates that a cleverer tool exists. Python's standard library rewards the boring choice when the boring choice is also the clear one, and reserves the powerful tool for the problem that actually needs its power.

from functools import partial, reduce
def greet(greeting, name):
    return f"{greeting}, {name}!"
hello = partial(greet, "Hello")
print(hello("Ada"))
print(reduce(lambda a, b: a + b, [1, 2, 3]))

Prefer partial() over a lambda when you are only fixing arguments, and prefer a built-in aggregation over reduce() when one expresses your goal. Keep reduce() for the rare custom fold that no built-in describes, and when you use it, choose a named function over an inline lambda so the operation can be understood at a glance. Small choices like these accumulate into code that is easy to read, easy to test, and easy to change.

Check: When is partial() clearer than a lambda wrapper, and when should a built-in aggregation be preferred over reduce()?

Section 7

Randomness comes in two very different kinds, and confusing them is a genuine security risk. The random module generates pseudo-random numbers from a seed, using an algorithm that is fast and produces values that look random enough for games, simulations, sampling, and shuffling test data. Its output is deterministic: given the same seed, it produces the same sequence, which is a feature for reproducible experiments, because a simulation can be replayed exactly. It is also, however, the reason random must never be used where unpredictability matters. An attacker who observes enough output can reconstruct the internal state and predict future values, which would compromise anything that depends on secrecy, such as a password, a token, or a lottery draw.

For security-sensitive randomness, Python provides the secrets module, which draws from the operating system's source of randomness, designed to be unpredictable even to an observer who has seen previous output. Its interface is deliberately small: secrets.token_hex() and token_urlsafe() produce tokens of a requested length, choice() picks an item from a sequence, and randbelow() produces a number below a bound. Those few functions cover the common security tasks, such as generating a reset token, a session identifier, or a one-time code, without exposing the lower-level details that make misuse easy. If a value must stay secret, or if guessing it would give someone an advantage, it belongs to secrets, not to random.

The distinction is best decided by asking what happens if the value is predicted. If the worst outcome is that a game becomes less fun, a sample becomes biased, or a test becomes flaky, random is appropriate, and its seedability is a genuine benefit. If the worst outcome is that an account is taken over, a session is hijacked, or a supposedly secret value becomes public, only secrets will do, and no amount of clever seeding can make random safe for that purpose. The same rule applies to shuffling: shuffling a playlist with random is fine, but shuffling anything where the order must not be guessable, such as a draw or an assignment, needs the unpredictable source.

import random
import secrets
print(random.choice(["red", "green", "blue"]))
print(secrets.token_hex(16))
print(secrets.choice(["red", "green", "blue"]))

Use random for simulations, games, sampling, and testing, where reproducibility is welcome and prediction is harmless. Use secrets for passwords, tokens, session identifiers, and anything else whose unpredictability protects people or data. Making that choice deliberately, at the moment you write the code, is a small habit that prevents a class of security failure that is difficult to detect and embarrassing to explain afterwards.

A related mistake is to use random for generating something that looks secret but is not, such as a temporary password emailed to a user, on the grounds that it will be changed soon. A temporary secret is still a secret while it is valid, and an attacker who predicts it does not need it to last long. The same reasoning applies to any value that grants access, however briefly, and it is why the choice between random and secrets should be made by asking what prediction would cost, not by asking how long the value will live.

Check: Why is the random module unsuitable for security-sensitive randomness, and which module should be used instead?

Section 8

Two tools help a program keep data in order, and they suit different patterns of use. A heap, provided by the heapq module, is a structure that always gives you its smallest item quickly, while letting you add new items cheaply. It is not a sorted list: its internal order is only partial, and only the smallest element is guaranteed to be in position. That trade-off is exactly what makes it efficient for problems like scheduling the next task, merging sorted streams, or repeatedly taking the smallest available item. Functions such as heappush(), heappop(), and heapify() operate on an ordinary list, turning it into a heap in place, and nlargest() and nsmallest() answer questions about extremes without sorting everything.

A sorted list maintained with bisect serves a different need. The bisect module finds the insertion point for a value in a list that is already sorted, so a program can insert new items in the right place and keep the whole list in order. The cost is that inserting into the middle of a list moves the later elements, which becomes expensive as the list grows large. What you gain in return is a fully ordered collection that you can search, slice, and iterate in order, with every element exactly where a reader would expect it. That makes bisect a good fit when a program needs the whole ordering, not just the smallest item: a list of thresholds, a timeline of events, or a set of breakpoints that must be examined in sequence.

The choice, then, depends on which question the program asks most often. If the question is always "what is the smallest item right now, and what comes next?", a heap answers it with the least work, because it never pays the cost of keeping everything sorted. If the question is "where does this value belong in the full order, and what are its neighbours?", a sorted list maintained with bisect answers it directly, because the full order is always available. Using a heap when you need the full order forces you to sort it anyway, losing the benefit; using a sorted list when you only ever need the smallest item pays for ordering you never use.

import heapq
import bisect
tasks = [5, 1, 4]
heapq.heapify(tasks)
print(heapq.heappop(tasks))
scores = [10, 20, 40]
bisect.insort(scores, 30)
print(scores)

Choose a heap when a program repeatedly needs the smallest item and does not care about the order of the rest. Choose a sorted list with bisect when the program needs the complete order, needs to search within it, or needs to examine neighbours. Naming the question your program asks, before picking the structure, is the habit that keeps this choice correct as the program grows.

Check: When should you use a heap, and when should you use a sorted list maintained with bisect?

Section 9

Every command-line program receives its arguments as strings, and for a tiny script, reading them directly from sys.argv is enough. As soon as a program grows beyond a couple of arguments, however, manual parsing becomes a burden: the program must check how many arguments were given, convert strings to numbers, recognise flags, supply defaults, and print a usage message when the user makes a mistake. Each of those jobs is small, but together they add up to a surprising amount of fiddly code, and the error handling is usually the first part to be skipped. The argparse module exists to take that burden over, and it provides three benefits that manual parsing rarely matches.

The first benefit is automatic, consistent handling of options and arguments. With argparse, you declare what your program accepts: positional arguments, optional flags, values with types, and defaults. The module then parses the command line, converts values to the declared type, applies defaults for missing options, and reports an error when a required argument is absent or a value cannot be converted. That declaration is also documentation: the program's interface is described in one place, in code, rather than scattered through a series of checks. A reader can see at a glance what the program accepts, and the program behaves the same way for every invocation.

The second benefit is help and error messages that users can actually use. argparse generates a help screen from your declarations, listing each option, its meaning, and its default, and it prints a clear error when an argument is wrong, pointing at the problem instead of crashing with a traceback. The third benefit is validation that happens before your program's real work begins. Bad input is rejected at the boundary, so the main logic can assume its inputs are well formed, which makes that logic shorter and easier to test. Manual parsing can provide all of this, but only by reimplementing it, badly, for every program.

import argparse
parser = argparse.ArgumentParser(description="Count lines in a file.")
parser.add_argument("path", help="file to read")
parser.add_argument("--limit", type=int, default=10)
args = parser.parse_args()
print(args.path, args.limit)

Use sys.argv directly only for the smallest scripts, where the interface is one or two plain values and no help is needed. For anything non-trivial, declare the interface with argparse, and let it handle parsing, conversion, defaults, help, and errors. The result is a program that is kinder to its users, because it explains itself when asked and complains helpfully when misused, and kinder to its maintainer, because the interface lives in one readable declaration.

It is also worth noting that argparse handles the small details that manual code usually gets wrong, such as accepting both --flag value and --flag=value, allowing options to appear in any order, and distinguishing a missing value from a value that happens to look like another flag. Those details seem minor until a user encounters them, and handling them consistently is part of what makes a command-line program feel polished rather than improvised.

Check: What three benefits does argparse provide over manual sys.argv parsing for non-trivial command-line programs?

Section 10

Logging is how a program explains itself while it runs, and the logging module is designed around a distinction that is easy to get wrong: libraries log, applications configure. A library is code that other programs import, and it should record events through loggers without deciding where those messages go, how they are formatted, or which levels are shown. An application is the program that is actually run, and it is the application's job to configure logging once, at its entry point, deciding the level, the format, and the destination, such as the console or a file. When that division is respected, a library's messages flow into whatever arrangement the application chose, and several libraries can share one coherent log.

Configuration should happen once, at the entry point, because logging configuration is global state: calling a setup function in several places overwrites earlier settings, changes output mid-run, and makes the program's behaviour depend on import order, which is fragile and confusing to debug. A single setup at the start, typically with basicConfig() or a more detailed configuration, gives the whole program one predictable arrangement. It also gives the user one place to adjust: changing the level to see more detail, or redirecting output to a file, is a single change rather than a hunt through the codebase. Code that runs later should simply log, trusting that the entry point has already decided how those messages are handled.

Libraries must not configure logging themselves for a related reason: a library does not know how its host application wants to present messages, and configuring output inside a library can hijack the application's log, duplicate messages, or write to a file the user never asked for. A library that calls basicConfig() on import is a classic source of such surprises, because the side effect happens before the application has had any chance to choose. The recommended practice is for libraries to obtain a logger with getLogger(__name__) and to add a null handler, so their messages are available but silent until the application enables them. That way, using a library never changes the application's logging against its will.

import logging
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger(__name__)
logger.info("Program started.")

Configure logging once, in the program you run, at the point where it starts. Write libraries that log through named loggers and configure nothing, leaving every decision about output to the application that uses them. That division keeps logs predictable, keeps libraries polite guests in other people's programs, and keeps debugging a matter of adjusting one configuration instead of untangling several competing ones.

Another benefit of a single configuration point is that it makes the program's logging behaviour testable: a test can configure logging for itself, run the program's entry point, and check that messages appear as expected, without fighting settings left behind by imported libraries. When configuration is scattered, that kind of test becomes fragile, because the outcome depends on which modules were imported first.

Check: Why should logging configuration happen once at a program's entry point, and why should libraries not configure logging themselves?

Section 11

The shutil module provides high-level file operations, and one of its functions deserves a warning label: shutil.rmtree() deletes a directory and everything inside it, recursively, without asking for confirmation and without any equivalent of a recycle bin. A single mistake in the path it is given, such as an empty string, a variable that was never set, or a path that points higher in the tree than intended, can destroy far more than the program meant to remove. That danger is not a flaw in the function; recursive deletion is sometimes exactly what is needed, for example when cleaning up a build directory. The danger lies in how silently and completely it acts, and in how little room it leaves for recovery once it has run.

Safe use of rmtree() therefore starts with caution around the path itself. A program should check that the path exists, that it is a directory, and that it lies inside an expected parent directory, before deleting anything. It should never build the path from untrusted input without validation, and it should log or print the path it is about to remove when the deletion is part of an interactive or destructive operation, so a human can spot a mistake before it happens. Some programs add a further safeguard by moving the directory to a temporary location first, or by requiring an explicit confirmation for deletions outside a known workspace. None of these checks makes deletion safe in an absolute sense, but each one turns a silent catastrophe into a mistake that is caught in time.

Tests face a related problem. A test that writes into a fixed-name directory, such as a folder called test-output in the project, can interfere with other tests, with previous runs, and with the developer's own files. A test that fails halfway may leave that directory behind, and the next run then starts from a polluted state, producing failures that are hard to reproduce. Temporary directories solve this: each test gets a fresh, uniquely named directory that is created for that test and removed afterwards, so tests cannot pollute each other and cannot accidentally touch real data. Python's testing tools provide this directly, and the standard library's tempfile module provides it for programs outside a test framework.

import tempfile
from pathlib import Path
with tempfile.TemporaryDirectory() as name:
    folder = Path(name) / "work"
    folder.mkdir()
    print(folder.exists())

Treat recursive deletion as an operation that demands verification of its target, and treat test output as something that belongs in a fresh temporary directory, never in a fixed location. Both habits protect against the same failure: a program acting on the wrong directory, with consequences that cannot be undone. Caution here is not timidity; it is the recognition that deletion is one of the few operations where a single wrong value has no second chance.

Check: Why is shutil.rmtree() dangerous, and why should tests use temporary directories rather than fixed-name directories?

Section 12

Running another program from Python is a common task, and the subprocess module is the standard way to do it. It also contains one of the easiest security mistakes to make: passing a command as a single string with shell=True. When the shell is involved, the string is interpreted as a shell command, which means characters such as semicolons, pipes, and backticks take on their shell meanings. If any part of that string comes from a user, a file name, or any other untrusted source, an attacker can inject additional commands that the program never intended to run. A file name containing a semicolon followed by a destructive command would be executed as two commands, not treated as one file name, and the program would have no way to tell the difference.

Passing the command as a list with shell=False, which is the default, avoids that entire class of problem. Each element of the list is exactly one argument: the program name first, then each argument as a separate string, with no shell interpretation in between. A file name containing a semicolon is then passed to the program as that file name, and nothing more, because there is no shell to reinterpret it. The same applies to spaces, quotes, and other special characters, which are the usual sources of quoting bugs in shell strings. List form is not only safer; it is also more predictable, because the program receives exactly the arguments you listed, in exactly the order you listed them, regardless of what characters they contain.

There are occasions when shell features are genuinely needed, such as a pipeline or a redirection, and in those cases the safe approach is to build the pipeline from separate list-form calls, connecting them in Python, rather than handing a string to the shell. If a shell string is truly unavoidable, it must be built only from trusted, fixed values, never from user input, and even then it deserves a comment explaining why the list form could not be used. The default, however, should always be the list form, and any use of shell=True should be treated as a decision that needs justification, not as a convenience.

import subprocess
result = subprocess.run(
    ["echo", "hello; rm -rf /"],
    capture_output=True, text=True, check=True,
)
print(result.stdout.strip())

Pass command arguments as a list with shell=False, so that each argument means exactly itself and nothing else. Reserve shell=True for the rare case where shell features are truly required, build it only from trusted values, and document why. That single habit removes a whole category of injection vulnerabilities, and it makes the program's behaviour easier to predict, because the command it runs is the command you wrote, not a reinterpretation of it.

Check: Why is passing command arguments as a list with shell=False safer than passing a command string with shell=True?

Section 13

Network requests are unlike local function calls in one important respect: they can hang. A server may be slow, a connection may stall, or a response may never arrive, and a program that waits without a limit will wait forever, holding up everything that depends on it. That is why every network request should specify a timeout: a maximum time to wait for a response before giving up and handling the failure. A timeout turns an unbounded wait into a bounded one, so a program can report an error, retry, or move on to other work instead of freezing. The right value depends on the situation, but the principle does not: an absent timeout is a bug waiting for a slow network to reveal it, and it is one of the most common causes of programs that appear to freeze without explanation.

Timeouts are only part of writing correct network code. The other part is parsing URLs properly, and here the standard library's urllib.parse module does work that manual string splitting does badly. A URL is a structured value, with a scheme, a host, an optional port, a path, and a query string, and those parts can contain characters, such as encoded spaces and separators, that defeat naive splitting on slashes or question marks. Splitting a URL by hand works for the simple examples a programmer tests with, and fails on the real URLs that users paste in, which may include a port number, an empty path, or a query value that itself contains an encoded separator. urllib.parse.urlparse() separates the parts correctly according to the URL's structure, and parse_qs() decodes the query string into its parameters, handling the encoding rules that manual code usually ignores.

Together, these two practices describe a defensive attitude to the network: expect delay, and expect structure. A timeout acknowledges that the network is unreliable, and proper URL parsing acknowledges that input from the outside world is more varied than the examples in a tutorial. Both are cheap to adopt and expensive to retrofit, because a program that freezes in production, or that misroutes a request because it split a URL wrongly, is difficult to debug from its symptoms alone. The failure looks like a hang, or like a wrong page, rather than like the missing timeout or the naive split that caused it.

import urllib.request
from urllib.parse import urlparse
parts = urlparse("https://example.com:8080/docs?page=2")
print(parts.hostname, parts.port, parts.path)

Always give a network request a timeout, and always parse URLs with urllib.parse rather than by splitting strings. Those two habits protect a program against the two most common network surprises: a response that never comes, and an address that is more complicated than it first appears. They are small pieces of discipline, but they are the difference between a program that degrades gracefully and one that simply stops.

Check: Why should every network request specify a timeout, and why should URL parsing use urllib.parse rather than manual string splitting?

Section 14

Dataclasses reduce the boilerplate of writing classes that mainly hold data. Instead of writing an __init__ method that assigns each argument to an attribute, plus methods for comparison and representation, you declare the fields and let the decorator generate those methods for you. That convenience comes with a subtlety around default values. A field's default is evaluated once, when the class is defined, and shared by every instance that does not supply its own value. For an immutable default, such as a number or a string, sharing is harmless, because the value cannot be changed. For a mutable default, such as a list or a dictionary, sharing is a trap: appending to one instance's list would change the list seen by every other instance, because they are all looking at the same object.

The solution is field(default_factory=...), which supplies a callable that is called afresh for each new instance. Writing field(default_factory=list) means that every instance gets its own new, empty list, created at the moment the instance is created, so changes to one instance's list never leak into another's. The same pattern works for dictionaries, sets, and any other mutable type, and it also works for a function that computes a default, such as the current time. The rule is simple to state: if the default could be changed after creation, it must come from a factory, not from a literal value. Python enforces this rule by refusing a mutable literal as a direct default, turning what would be a silent bug into an immediate, clear error.

A dataclass can also be frozen, which makes its instances immutable after creation: assigning to a field raises an error. Freezing is appropriate when the object represents a value rather than a changing entity, such as a point in space, a configuration snapshot, or a record that should not be altered once it has been created. Immutable objects are easier to reason about, because a function that receives one cannot change it by accident, and they can safely be shared between parts of a program. A frozen dataclass is not appropriate when the object's whole purpose is to accumulate changes, such as a running tally, where mutability is the point of the object.

from dataclasses import dataclass, field
@dataclass(frozen=True)
class Point:
    x: float
    y: float
    tags: list = field(default_factory=list)

Use field(default_factory=...) for every mutable default, so each instance owns its own containers. Freeze a dataclass when it models a value that should not change, and leave it mutable when change is its purpose. Those two decisions, made at the point of definition, prevent the shared-state bug that is the classic dataclass pitfall, and they make the object's contract clear to everyone who uses it.

Check: Why do dataclasses require field(default_factory=...) for mutable defaults, and when should a dataclass be frozen?

Section 15

Programs constantly need to represent a fixed set of related values: the days of the week, the states of an order, the levels of a log message, or the directions on a map. The simplest way to do that is with bare strings or integers, such as "pending", "shipped", or 1, 2, 3. That approach works at first, but it is fragile. A misspelling such as "pendng" will not raise an error; it will simply fail to match, and the program will behave wrongly in a way that is hard to trace. An integer offers no clue about its meaning when it appears in a log or a debugger, and nothing prevents a programmer from passing an unrelated number where a state was expected. Both representations leave the set of valid values implicit, known only to whoever remembers them.

Enumerations, provided by the enum module, make that set explicit. An enumeration is a class whose members are the valid values, each with a name and, optionally, an underlying value. Writing OrderStatus.PENDING instead of "pending" means a misspelling becomes an attribute error at the point of use, rather than a silent mismatch at the point of comparison. It also means the valid values can be listed, iterated, and documented in one place, so a new reader can see the complete set without hunting through the code for every string that might be used. The underlying value, if one is needed for storage or display, is kept separate from the name used in code, so the code reads in terms of meaning rather than in terms of stored representation.

Enumerations also improve safety when values cross boundaries. Comparing an enumeration member with a plain string will not accidentally succeed, which prevents the quiet confusion that arises when two different representations of the same idea are mixed. A function that accepts an enumeration member declares, through its type, exactly which values it will accept, and a caller who passes anything else has made a mistake that is visible immediately. That clarity extends to debugging: printing an enumeration member shows its name, which tells a human what state the program was in, rather than showing a bare number that must be decoded from memory.

from enum import Enum, auto
class OrderStatus(Enum):
    PENDING = auto()
    SHIPPED = auto()
    DELIVERED = auto()
print(OrderStatus.PENDING)

Use enumerations whenever a value comes from a fixed, known set, especially when that value controls behaviour, is stored, or is compared in several places. Keep bare strings and integers for values that are genuinely open-ended, such as user input or free-form text, where no fixed set exists. Making the valid set explicit turns a whole class of spelling and mixing errors into immediate, visible failures, and it makes the code read as a statement of intent rather than as a collection of magic values.

Check: Why are enumerations clearer and safer than bare strings or integers for representing a fixed set of related values?

Section 16

Type annotations are notes about what kind of value a variable, argument, or return value is expected to hold, written alongside the code, such as def greet(name: str) -> str. It is natural to assume that such notes change how the program runs, but by themselves they do not affect runtime behaviour at all. Python does not check annotations when a function is called, does not reject a value of the wrong type, and does not optimise code based on them. An annotation is, in effect, a structured comment: it is stored on the function or variable, it can be read by tools, and it is ignored by the interpreter during execution. A program with incorrect annotations will run exactly as it would without them, which is both the strength and the limitation of the feature.

That limitation is addressed by a separate tool: a type checker. A type checker reads the program and its annotations without running it, and reports places where the code contradicts the declared types, such as passing a number where a string was promised, or returning the wrong kind of value. Running a checker is a static analysis step, like linting, performed during development or as part of automated checks, and it catches a class of mistakes that tests may miss, because it examines every path through the code, not only the paths a test happens to exercise. The checker does not change the program either; it only reports, and the programmer decides what to fix. The value of annotations therefore lies entirely in what tools and readers do with them.

Annotations also serve human readers directly, even when no checker is run. A function signature that declares its argument and return types tells a caller what to pass and what to expect, without requiring them to read the body or guess from examples. That documentation stays in step with the code more reliably than a separate document, because it sits beside the code it describes. For that reason, annotations are most valuable at boundaries: public functions, library interfaces, and data structures that pass between parts of a program. Annotating every local variable, by contrast, adds noise without adding information, because the reader can usually see the value's origin nearby.

def add(a: int, b: int) -> int:
    return a + b
print(add(2, 3))

Remember that annotations alone change nothing at runtime, and that a type checker is the tool that makes use of them. Write annotations where they clarify an interface, run a checker as part of your development routine, and do not mistake an annotated program for a checked one. Used that way, annotations become a quiet form of documentation and an early warning system, catching mismatches before the program ever runs.

Check: Why do type annotations not affect runtime behaviour by themselves, and what tool makes use of them?

Section 17

The unittest framework provides assertion methods, such as assertEqual(), assertIn(), and assertRaises(), that do more than a plain assert statement when a test fails. A plain assert reports only that a condition was false, and possibly a message the programmer remembered to include, leaving the reader to work out what the two sides of the comparison actually were. An assertion method, by contrast, knows what kind of comparison it performed, and its failure message shows both values involved, often with a helpful difference: for sequences, it can point to the first element that differs, and for dictionaries, it can show which keys differ. That extra information is what turns a failing test from a puzzle into a diagnosis, because the programmer sees immediately what the program produced and what was expected, without having to rerun the test with added printing.

The framework also structures how tests run, and the setUp() method is central to that structure. setUp() runs before each individual test method, not once for the whole test class, and that distinction matters. Running it before each test gives every test a fresh fixture: a new object, a new temporary directory, or a new connection, untouched by whatever the previous test did. A test that modifies its fixture therefore cannot pollute the next test, and tests can run in any order, or in isolation, with the same result. If setup ran only once, tests would share state, and a failure in one test could cause confusing failures in others, making the suite unreliable and its failures hard to trust.

Together, these features make a test suite a dependable tool rather than a loose collection of checks. Clear failure messages mean a failing test explains itself, and per-test setup means a failing test can be trusted to reflect its own behaviour rather than leftover state from a neighbour. The discipline they encourage is worth adopting even in small projects: write one test method per behaviour, build the fixture in setUp(), and assert with the most specific method available, so that when a test fails, the message tells you what went wrong, and when it passes, you can believe the result.

import unittest
class TestExample(unittest.TestCase):
    def setUp(self):
        self.values = [1, 2, 3]
    def test_length(self):
        self.assertEqual(len(self.values), 3)

Prefer unittest assertion methods over plain assert statements, because their failure messages show the values involved and the nature of the difference. Rely on setUp() to give each test a fresh fixture, so tests stay independent and their results stay trustworthy. A test suite built on those two habits is one you can run with confidence, and one whose failures you can act on without first having to reproduce them by hand.

There is a further advantage to the framework's structure: tests written with unittest can be discovered and run automatically, so a whole suite can be executed with a single command and its results reported together. That automation is what turns testing from an occasional manual habit into a routine check that runs before every change is shared.

Check: Why do unittest assertion methods give clearer failure messages than plain assert statements, and why does setUp() run before each test?

Section 18

Installing packages globally, into the Python installation shared by every project on a machine, seems convenient at first: a package is installed once, and every program can use it. The trouble appears as soon as two projects need different versions of the same package, or a project needs a package that another project must not have. A global installation can hold only one version at a time, so upgrading a package for one project silently changes the environment of every other project, and a program that worked yesterday can break today because an unrelated project asked for an upgrade. Worse, a program developed against global packages may depend on something the programmer forgot to record, and it will then fail on any other machine where that package was never installed.

A virtual environment solves this by giving each project its own isolated set of packages. Creating one produces a directory containing a copy of the Python interpreter and a site-packages folder that belongs to that project alone. Activating the environment makes the project's Python and its installed packages the ones that are used, so installing, upgrading, or removing a package affects only that project. Deactivating returns the shell to the system Python, leaving other projects untouched. The environment directory is usually excluded from version control, and the project's dependencies are recorded in a requirements file, so another developer, or a deployment machine, can recreate the same environment from that list.

That isolation also makes experimentation safe. Trying a new version of a library, or testing whether a package is really needed, becomes a low-risk operation: if it breaks something, only one project's environment is affected, and it can be rebuilt from the requirements file. It also makes the project's dependencies explicit, which is valuable documentation in itself. A requirements file that lists every package, with its version, is a precise statement of what the program needs, and it is far more reliable than a programmer's memory of what happens to be installed on their machine.

import sys
print(sys.prefix)
print(sys.executable)

Give every Python project its own virtual environment, rather than installing packages globally. Record the dependencies in a requirements file, so the environment can be recreated anywhere, and treat the environment directory as disposable: it can always be rebuilt, while the list of dependencies is the part worth keeping. That habit prevents the silent cross-project breakage that global installs invite, and it makes every project self-contained, reproducible, and safe to experiment with.

Virtual environments also help when a project is shared with others, because the requirements file becomes a precise record of what the project needs, and a new contributor can recreate the environment without guessing which packages, or which versions, the original developer happened to have installed. That reproducibility is difficult to achieve with global installs, where the environment is a property of one machine rather than of the project.

Check: Why should every Python project use its own virtual environment, rather than installing packages globally?

Section 19

Environment variables and configuration files solve different parts of the configuration problem, and choosing between them depends on what kind of value is being configured. Environment variables suit values that change between deployments: the operating mode, the host to connect to, or a feature flag that differs between development and production. They are set by the environment that runs the program, they leave no trace in the codebase, and they can be changed without editing any file. A configuration file suits values that are structured, numerous, or shared as a document: a list of options, a set of thresholds, or a mapping that a human needs to read and edit as a whole. A file can express nesting and grouping that a flat set of environment variables expresses only awkwardly, and it can be versioned alongside the code it configures, so changes to configuration are reviewed like any other change.

A configuration file is therefore the better choice when the configuration is complex, when it needs comments explaining its options, or when several related values must be changed together and kept consistent. It is also the better choice when the program is run by people who are not expected to manage environment variables, because editing a clearly named file is a more accessible operation. The trade-off is that a file is part of the project, and anything placed in it may be copied, shared, or committed to version control along with the code. That is precisely why sensitive values must never be committed with it. A password, a token, or a secret key that enters version control is difficult to remove reliably, because history retains it even after the file is edited, and anyone with access to the repository gains the secret.

The safe pattern combines both mechanisms: keep ordinary, non-sensitive configuration in the file, and keep sensitive values in environment variables, or in a secrets store, referenced by the configuration rather than embedded in it. The program then reads the file for its structure and the environment for its secrets, and the file can be committed, shared, and reviewed without risk. A further safeguard is to keep a template or example configuration file, showing every option with placeholder values, so a new deployment can be set up by copying the template and supplying real secrets separately.

import json
import os
from pathlib import Path
config = json.loads(Path("config.json").read_text())
token = os.environ.get("API_TOKEN")
print(config.get("mode"), bool(token))

Choose a configuration file when the settings are structured, numerous, or need to be reviewed as a document, and choose environment variables for values that vary by deployment and for anything sensitive. Never commit a real secret alongside a configuration file, because version history makes that leak hard to undo. Keeping structure in the file and secrets in the environment gives you the readability of a document and the safety of a secret that never enters the codebase.

Check: When is a configuration file a better choice than environment variables, and why must sensitive values never be committed with it?

Section 20

This chapter has introduced many modules, and it is worth stepping back to ask how to choose among them when a new task appears. Three considerations, taken together, form a practical framework: reuse, safety, and clarity. Reuse asks whether a standard tool already solves the problem, because a tested, documented module is almost always preferable to a custom implementation of the same idea. Custom code carries the burden of its own bugs, its own edge cases, and its own maintenance, while a standard tool has already been exercised by many programs. The first question for any familiar task should therefore be whether the standard library already contains the answer, and the documentation's module overview is the place to look.

Safety asks what can go wrong, and how badly. Some choices are dangerous in ways that are not obvious from their convenience: recursive deletion without verifying its target, shell commands built from untrusted strings, randomness that can be predicted, or secrets committed alongside configuration. A tool that is slightly more awkward but removes a class of failure is usually the right choice, because the cost of the failure dwarfs the cost of the extra care. Safety also covers the quieter failures, such as a program that hangs because a network request had no timeout, or a test suite that gives false results because tests share state. Asking "what is the worst outcome if this goes wrong?" before choosing a tool turns safety from an afterthought into a criterion.

Clarity asks whether the next reader will understand what the code does and why. A Counter says counting, a Path says path manipulation, an enumeration says a fixed set of values, and a declared argument parser says what a program accepts. Each of those choices replaces mechanism with meaning, so the code can be read for its purpose rather than decoded from its steps. Clarity is not a luxury; it is what makes code maintainable, because code is read far more often than it is written, and a reader who understands the intent can change it safely.

from collections import Counter
print(Counter(["a", "b", "a"]).most_common(1))

When you face a new task, ask the three questions in order: does a standard tool already solve it, what can go wrong with each candidate, and which choice will be clearest to the next reader? A choice that satisfies all three is rarely wrong, and a choice that fails any one of them deserves a second thought. That framework, applied consistently, is the real lesson of this chapter: not a list of modules to memorise, but a way of choosing tools that keeps programs reliable, safe, and readable as they grow.

It is worth adding that the framework also helps when reviewing someone else's code, because it gives the reviewer concrete questions to ask: is there a standard tool for this, what happens if this fails, and will the next reader understand this at a glance? A review guided by those questions tends to find real problems, rather than merely debating style.

Check: How do reuse, safety, and clarity together form a framework for choosing standard library tools?

Section 21

Text often needs tidying before it is shown to a user, and two small standard library modules help with that job. The textwrap module wraps long lines, indents blocks of text, and shortens text that is too long for a display area. Its wrap() function breaks a paragraph into a list of lines of a given width, while fill() returns the same result as a single string with newlines inserted. That is useful when a program prints help text, reports, or messages in a terminal of limited width, because it keeps output readable without the programmer counting characters by hand.

The string module is a different kind of tool. It holds useful constants, such as the lowercase letters, the digits, and the punctuation characters, and it provides the Template class for simple string substitution. Templates use placeholders like $name and replace them with values from a mapping. They are deliberately simpler than f-strings: they do not evaluate expressions, so they are a safer choice when a template comes from a source you do not fully control, such as a configuration file or a user-supplied message pattern.

Wrapping and templating complement each other in practice. A program might build a message from a template, filling in a user's name and a count, and then wrap the result so it fits neatly in a terminal window. Neither module is large, and neither needs to be learned in full before it is useful. A single function from each — textwrap.fill() and string.Template — covers most everyday cases, and the rest can be discovered when a specific need arises.

A common mistake is to wrap text that already contains its own line breaks, such as a poem or a pre-formatted table, where wrapping would destroy the layout. textwrap is for prose paragraphs, not for fixed-format output. Another mistake is to use templates where f-strings would be clearer, since templates add indirection that only pays off when the template itself is data rather than code. Choose the tool that matches where the text comes from: code you wrote yourself favours f-strings, while text supplied at runtime favours templates.

Together, these modules illustrate a pattern you will see across the standard library: small, focused tools that solve one formatting problem well. Learning to reach for them, rather than writing your own wrapping loop or substitution code, saves time and avoids bugs in edge cases like very long words or missing placeholder values.

import textwrap
from string import Template
msg = Template("Hello $name, you have $n new messages.")
print(textwrap.fill(msg.substitute(name="Ada", n=3), width=20))

When wrapping output, choose a width that matches the display you are targeting, and test with the longest realistic input, since narrow widths can make even short words wrap awkwardly.

Check: When is a string.Template a safer choice than an f-string, and when should textwrap not be used?

Section 22

When a program prints a large or nested data structure with print(), the output often runs across many lines in a way that is hard to scan. The pprint module, short for pretty-printing, formats data structures so they are easier for a human to read. Its pprint() function prints dictionaries, lists, and other structures with one item per line where that helps, indenting nested levels so the shape of the data is visible at a glance. That makes it a good debugging aid: when a value is not what you expected, a pretty-printed view often shows the problem immediately.

The related reprlib module limits how much of a large structure is shown. Its repr() function produces a shortened representation, truncating long lists, dictionaries, and strings to a manageable size. This matters when a structure contains thousands of items and a full printout would flood the terminal. Combined, the two modules give you control over both layout and length: pprint for readable shape, reprlib for bounded size.

Neither module changes the data itself. Both produce representations for display only, and neither should be used to serialise data for storage — that is a job for formats such as JSON, covered in the next chapter, which are designed to be read back reliably. Pretty-printed output is for humans; stored data needs a format with exact round-trip rules.

A useful habit is to reach for pprint in a debugging session before adding more complex logging. When a test fails and the difference between expected and actual values is unclear, printing both values with pprint often makes the mismatch obvious, because the differing item stands out in the aligned layout. It is a small tool, but it earns its place in the standard library by making one frequent task — inspecting data — noticeably easier.

import pprint
data = {"users": [{"name": "Ada", "tags": ["admin", "dev"]}, {"name": "Ben", "tags": []}]}
pprint.pprint(data, width=40)

Pretty-printing also helps when comparing two structures side by side, because aligned indentation makes a single differing value stand out against otherwise identical output. When a structure is very deep, the depth parameter limits how far nesting is expanded, keeping the overview readable while hiding detail that is not currently relevant. A depth-limited view is often enough to locate a problem, and you can then inspect the specific branch in full. Used this way, pprint becomes a quick first step in debugging, before reaching for a debugger or adding temporary logging statements throughout a program. In practice, combine pprint with a short message that says what the printed value is, so the output is self-explanatory when you return to it later. A bare dump without context is easy to misread in a long log.

Check: Why is pprint output suitable for debugging but not for storing data, and what problem does reprlib solve?

Section 23

Copying objects in Python is subtler than it first appears, because assignment never copies an object at all: it only creates another name for the same object. The copy module makes the distinction explicit. Its copy() function makes a shallow copy, which is a new container holding references to the same items as the original, while deepcopy() makes a deep copy, recursively copying the container and everything inside it. Choosing between them depends on whether the items inside the container are shared or independent.

A shallow copy is enough when the items are immutable, such as numbers and strings, because shared immutable items cannot be changed in place. It is also enough when you only intend to change the outer container, for example by appending to a copied list, since the copy has its own list structure even though its items are shared. Problems arise when the items themselves are mutable, such as nested lists: changing a nested list through the copy also changes it through the original, because both copies refer to the same nested object.

A deep copy avoids that sharing entirely, at the cost of copying everything, which can be slow for large structures and can fail for objects that cannot be copied, such as open files. Deep copies also handle shared references and cycles sensibly, preserving the shape of the original structure rather than duplicating shared parts. That makes deepcopy() the right default when you genuinely need an independent copy of a nested structure and are unsure how deep the sharing goes.

A practical rule is to prefer immutable data and explicit construction over copying where possible. Building a new list with a comprehension often expresses intent more clearly than copying and then modifying. When copying is needed, state which kind you mean: use copy() for a new outer container and deepcopy() for full independence, and add a comment when the choice matters for correctness.

import copy
original = [[1, 2], [3, 4]]
shallow = copy.copy(original)
deep = copy.deepcopy(original)
shallow[0].append(99)
print(original, deep)

Another useful distinction is between copying and sharing intentionally. Sometimes two variables should refer to the same object, for example when both represent the same live record that must stay in sync. In that case assignment, not copying, is the correct tool, and copying would introduce a subtle bug where updates in one place no longer appear in the other. The question to ask is always whether the two names should stay connected or become independent. If they should stay connected, use assignment; if they should diverge, copy, and choose shallow or deep copying according to whether the nested items are shared or independent in the new copy. Test your copying choice with a small example before relying on it in a larger program, since sharing bugs are quiet and only show up when a nested value changes unexpectedly.

Check: In what situation does a shallow copy surprise you, and when is a deep copy the right choice despite its cost?

Section 24

The operator module exposes Python's built-in operators as functions, so they can be passed around like any other callable. Instead of writing lambda a, b: a + b, you can use operator.add; instead of lambda item: item["name"], you can use operator.itemgetter("name"). That matters in the places where a function is expected as an argument — sorting keys, mapping functions, and the functools tools from earlier in this chapter — because a named operator function is clearer than an equivalent lambda, and often faster, since it is implemented in C rather than as Python bytecode.

The itemgetter and attrgetter functions deserve special attention. itemgetter("name") builds a function that extracts the "name" item from whatever it is given, which makes it a natural sort key for sorting a list of dictionaries by one field. attrgetter("title") does the same for attributes, making it a natural sort key for sorting objects. Both can take several arguments to extract several fields at once, producing a tuple key that sorts by the first field, then the second, and so on.

The arithmetic and comparison operators are also available, which is useful when an operator itself is data. A small calculator, for example, can map symbols to functions — "+" to operator.add, "*" to operator.mul — and then apply the chosen function, rather than writing a chain of if statements. That table-driven style is easier to extend: adding a new operation means adding one entry to the table, not one more branch to the logic.

One caution is readability. Operator functions shine where the operation is the point, as in a sort key or a dispatch table. Where the logic is more complex than a single operation, a named function with a clear name is better than a clever composition of operator functions. Use operator to remove trivial lambdas, not to replace readable code with puzzles.

import operator
books = [{"title": "Dune", "year": 1965}, {"title": "Emma", "year": 1815}]
print(sorted(books, key=operator.itemgetter("year")))

The operator module also includes functions for getting and setting items and attributes, and for calling functions, which are occasionally useful in generic code that manipulates objects without knowing their structure in advance. These are less common than itemgetter and attrgetter, but they follow the same principle: a named operation is clearer than an anonymous lambda performing the same step. When you find yourself writing the same small lambda in several places, check whether the operator module already names it. Replacing repeated lambdas with operator functions reduces duplication and makes the repeated operation visible as a single, searchable concept in the codebase. Sorting with itemgetter also works with reverse ordering and with several keys, giving compact, readable code for common data-shaping tasks in scripts and reports.

Check: Why is operator.itemgetter("year") a better sort key than an equivalent lambda, and when should you prefer a named function instead?

Section 25

Ordinary floating-point arithmetic is not exact for decimal fractions, because binary floating point cannot represent values like 0.1 precisely. That rarely matters for measurements, but it matters a great deal for money, where a one-cent error is a real error. The decimal module provides the Decimal type, which represents decimal numbers exactly as written, so Decimal("0.1") + Decimal("0.2") equals Decimal("0.3") exactly. That makes it the right tool for financial calculations, invoicing, and anywhere else that decimal rounding rules must match human expectations.

Decimals are constructed from strings, not from floats, and that distinction is essential. Decimal(0.1) converts the already-inexact binary float 0.1 into a decimal, preserving its inexactness, while Decimal("0.1") represents the exact decimal value a human means. A program that builds decimals from floats has gained nothing over using floats directly, so the habit to form is simple: always construct decimals from strings, or from integers, and never from floats.

The module also provides contexts that control precision and rounding. The default context uses 28 significant digits and rounds half to even, but a program can set its own rounding mode, for example rounding half up, which matches the convention many people expect for money. Quantising a result to two decimal places with quantize(Decimal("0.01")) expresses "round to cents" directly in code, making the rounding rule visible rather than hidden in a formatting step.

Decimal arithmetic is slower than float arithmetic, and decimals do not mix freely with floats in expressions, so it is not a drop-in replacement for every numeric task. Use floats for science and simulation, where speed and approximate results are appropriate, and decimals for money and accounting, where exactness and explicit rounding rules are required. Choosing the right numeric type is part of choosing the right tool.

from decimal import Decimal, ROUND_HALF_UP
total = Decimal("19.99") + Decimal("5.00")
print(total.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP))

Rounding also matters when a result is displayed rather than stored. Formatting a float to two decimal places can hide a small error from view, but the stored value remains inexact, and later arithmetic on it can accumulate further error. With decimals, quantising the stored value itself makes the rounding decision permanent and explicit, so every later use of that value works from the rounded figure. That is why accounting systems round at each step where money changes hands, rather than only at display time. The discipline is to decide when rounding happens, record that decision in code, and apply it consistently across the program. When a decimal result must be shown as currency, format the quantised value explicitly, so the display always shows two decimal places even for whole amounts. Keep money as Decimal from input to storage, converting only at the edges where data enters or leaves the system.

Check: Why must decimals be constructed from strings rather than floats, and when is decimal arithmetic the wrong tool?

Section 26

Some quantities are naturally fractions rather than decimals: a recipe uses one third of a cup, a probability is three out of four, and a musical rhythm divides a beat into halves and quarters. The fractions module provides the Fraction type, which represents a rational number exactly as a numerator and a denominator, automatically reduced to lowest terms. Adding Fraction(1, 3) and Fraction(1, 6) gives exactly Fraction(1, 2), with no rounding error at all, because the arithmetic is done on integers.

Fractions are constructed from integers, from strings like "1/3", or from other rationals, and they interoperate cleanly with integers. They do not mix with floats without conversion, which is a deliberate guard: converting a float to a fraction would expose the float's binary inexactness as a strange-looking fraction with a huge denominator, so the module forces you to be explicit about that conversion if you really want it. That explicitness prevents a class of subtle bugs where approximate input silently becomes exact-looking output.

Like decimals, fractions are exact, but they suit different problems. Decimals suit quantities that humans write in decimal notation, especially money, where two decimal places are the norm. Fractions suit quantities that are naturally ratios, where keeping the numerator and denominator visible is clearer than a decimal approximation — for example, when a program must report "one third" rather than "0.333333". Fractions can also grow large denominators through repeated arithmetic, so for long chains of computation a decimal or float may be more practical.

A helpful way to remember the division of labour is that floats approximate, decimals record decimal quantities exactly, and fractions record ratios exactly. Most programs need only one of these for a given quantity, and picking the one that matches how the quantity is understood — as a measurement, as money, or as a ratio — makes the code's intent clear to its readers.

from fractions import Fraction
print(Fraction(1, 3) + Fraction(1, 6))
print(Fraction("2/4"))

Fractions also appear in music, scheduling, and geometry, wherever a whole is divided into equal parts that must be combined exactly. A program that schedules work in fractions of an hour, for example, can add and compare those fractions without worrying that repeated addition of 0.1 has drifted away from the intended total. Converting a fraction to a float is easy when an approximate display value is needed, using float(), and the conversion is explicit, so the loss of exactness is a deliberate step rather than an accident. Keep the exact form for computation and convert only for display, and the program's arithmetic stays trustworthy throughout. For teaching or documentation, fractions make ratios visible in a way decimals cannot, which is why they suit explanations as well as calculations.

Check: When is a fraction a clearer representation than a decimal, and why does the module refuse to mix fractions with floats silently?

Section 27

Every system that stores records eventually needs a way to give each record a unique identifier, and the uuid module generates universally unique identifiers for that purpose. A UUID is a 128-bit value, conventionally written as 32 hexadecimal digits in groups, such as 123e4567-e89b-12d3-a456-426614174000. The most common form, version 4, is generated from random bits, and the space of possible values is so large that two independently generated UUIDs are, for practical purposes, never the same. That makes them suitable as identifiers that can be created on different machines without coordination.

UUIDs are useful precisely where a simple counting sequence is not. A counter needs a single authority to hand out the next number, which is awkward when records are created offline, on several devices, or by several services at once. A UUID can be generated locally, immediately, with no central coordination, and later merged with records from elsewhere without collisions. That is why they appear so often as database keys, filenames for uploaded files, and identifiers in distributed systems.

The module also provides other versions for other needs. Version 5 generates a UUID deterministically from a namespace and a name, so the same name always produces the same UUID — useful when an identifier must be reproducible, for example when deriving a stable key from a known string. Version 1 is based on time and a machine identifier, but it can reveal information about when and where it was created, so version 4 is the usual default when no special property is needed.

UUIDs are not a security feature. A random UUID is hard to guess, but it is not a secret and must not be treated as a password or an access token: anyone who learns a UUID can use it wherever it is accepted as an identifier. Treat UUIDs as unique labels, not as credentials, and keep real access control separate from identification.

import uuid
print(uuid.uuid4())
print(uuid.uuid5(uuid.NAMESPACE_DNS, "example.com"))

UUIDs are also convenient as filenames for stored objects, because a generated name is unlikely to collide with an existing file, even when many processes create files concurrently. That removes the need for a shared counter or a locking scheme just to choose a name. The trade-off is readability: a UUID filename tells a human nothing about the file's contents, so programs that use UUID filenames usually keep a separate record mapping identifiers to human-readable titles. That record is where meaning lives, while the UUID itself carries only uniqueness, which is exactly the job it was designed to do. When identifiers must be short for display, show only the first few characters of a UUID, but keep the full value internally, since truncated identifiers can collide. UUID version 4 is the default choice unless you need reproducibility, in which case version 5 is the better fit.

Check: Why are UUIDs better than counters for records created on several machines, and why must a UUID never be treated as a secret?

Section 28

Network programs often need to work with IP addresses, and parsing them by hand with string splitting is error-prone: addresses have strict formats, ranges, and special cases that are easy to get wrong. The ipaddress module provides types for IPv4 and IPv6 addresses and networks, with validation built in. Creating an address object from a string either succeeds with a valid address or raises an error for an invalid one, so malformed input is caught at the boundary of a program rather than causing strange behaviour later.

Network objects add further value. An IPv4Network represents a range of addresses, and it can tell you whether a given address belongs to that network, how many addresses the network contains, and what its first and last addresses are. That makes it the natural tool for tasks like checking whether a client address falls inside an allowed range, or listing the addresses in a small subnet for a configuration report. Doing the same work with integer arithmetic by hand would be possible but would invite off-by-one errors in the bit manipulation.

The module also handles the distinction between addresses and interfaces. An interface object pairs an address with its network, which is how a host's own configuration is usually expressed: an address plus a prefix length. Keeping that pairing in one object, rather than in two separate strings that might disagree, is another small example of the standard library's habit of making invalid states hard to represent.

This section is only a survey. Real network programming — opening connections, serving requests, handling protocols — belongs to later chapters, and this module is the address-handling companion to that work, not a substitute for it. Its lesson for now is general: when a domain has a strict format, look for a standard type that validates it, rather than writing your own parser and hoping it covers every edge case.

import ipaddress
net = ipaddress.ip_network("192.168.1.0/24")
print("192.168.1.50" in [str(a) for a in net.hosts()][:3], net.num_addresses)
print(ipaddress.ip_address("2001:db8::1").version)

Address objects also compare and sort naturally, so a program can order a list of addresses or test equality without converting them to strings first. String comparison of addresses gives wrong results, because it orders characters rather than numeric values, and it treats equivalent forms of the same address as different strings. Using address objects avoids both problems, since equality and ordering are defined on the numeric value. That is another instance of a general principle: represent a value with a type that understands its rules, and let that type handle comparison, validation, and formatting, rather than reimplementing those rules with strings. When working with both address families, write code that accepts either, since modern networks routinely use IPv4 and IPv6 side by side in the same system.

Check: What does an IPv4Network give you that manual integer arithmetic on address strings does not, and why is validation at the boundary valuable?

Section 29

Data is often encoded as text so it can travel inside formats that only carry text, such as JSON documents, email messages, or URLs. The base64 module performs that encoding: it converts arbitrary bytes into ASCII characters, at the cost of making the output about a third larger than the input. Encoding is not encryption — anyone can decode base64 just as easily as you encoded it — but it solves a real problem, which is that raw bytes may contain values that a text channel would corrupt or reject.

The most common variant is standard base64, produced by b64encode() and reversed by b64decode(). A URL-safe variant replaces two characters that have special meaning in URLs, so encoded values can appear in a web address without additional escaping. Choosing the right variant matters: standard base64 placed in a URL can break parsing, while URL-safe base64 used where standard base64 is expected can fail validation. The module also provides helpers for the hexadecimal encoding used in other contexts, where readability matters more than compactness.

A related habit is to be explicit about the bytes-versus-text boundary. Encoding takes bytes and returns bytes; decoding takes bytes and returns bytes. Converting between bytes and text is a separate step, done with .decode("ascii") or .encode("ascii") as appropriate. Keeping those steps distinct avoids a common confusion where a program treats an encoded string as if it were the original data, or vice versa.

Base64 appears in many everyday places: small images embedded in web pages, tokens carried in headers, and binary attachments in messages. Recognising it on sight — the alphabet of letters, digits, + and /, often ending with = padding — helps you understand data you encounter, and knowing that it is an encoding, not a protection mechanism, keeps you from mistaking obscurity for security.

import base64
encoded = base64.b64encode(b"hello")
print(encoded)
print(base64.b64decode(encoded))

Encoding also matters for data that must survive transport through systems that are not byte-clean, such as older email infrastructure or configuration files that only accept printable characters. In those settings, base64 provides a safe alphabet that survives copying, pasting, and line-wrapping without corruption. The cost is size, and the benefit is robustness: the encoded form can be handled by any tool that handles text. When you encounter a long string of base64 characters in a configuration file or a message, remember that decoding it is trivial, so it should never contain anything you would not also be willing to show in plain text. Always decode with error handling in mind, since malformed base64 input should produce a clear error rather than silently corrupted data in your program. Treat encoded data as opaque when passing it between systems, and decode it only at the point where the original bytes are actually needed.

Check: Why is base64 encoding not a security measure, and why must the URL-safe variant be used inside web addresses?

Section 30

When a program serves, sends, or stores a file, it often needs to know what kind of file it is — not from its name alone, but from a standard mapping between filenames and media types. The mimetypes module provides that mapping. Its guess_type() function takes a filename and returns a media type such as "text/html" for a web page or "image/png" for a picture, based on the file's extension. That is the information a web server needs to set a response's content type, and the information an email program needs to label an attachment.

Guessing from an extension is a heuristic, not a guarantee. A file named report.txt might contain anything, and a file with no extension gives the module nothing to work with. For that reason, mimetypes results should be treated as a best guess for labelling and display, never as a security decision. A program that must trust a file's contents — for example, before processing an uploaded file — should inspect the contents themselves, not rely on the extension or the guessed type. That distinction is a security matter, covered in depth in a later chapter, but the habit starts here: extensions describe intent, contents describe reality.

The module also lets a program register its own types and read the system's own mappings, so a program running on a machine with unusual file types can still guess sensibly. Most programs need only guess_type(), and a fallback for the common case where it returns None, meaning no known type. Handling that None explicitly, rather than assuming every file has a known type, prevents crashes on unfamiliar extensions.

Like several modules in this chapter, mimetypes is small and single-purpose. Its value is that it encodes a large, boring table of filename-to-type mappings that you would otherwise have to maintain yourself, and that table is kept up to date with the platform rather than with your code.

import mimetypes
print(mimetypes.guess_type("photo.png"))
print(mimetypes.guess_type("notes.unknown"))

Media types also matter for downloads, because a browser decides how to handle a downloaded file partly from its declared type. A server that declares the wrong type can cause a browser to display a file it should have downloaded, or download a file it could have displayed, confusing users. Getting the declared type right is therefore part of serving files correctly, even though the type itself is only a label. The mimetypes module helps servers do this consistently, by applying the same table to every file, rather than leaving each handler to guess its own type from the filename in its own way. Register custom types only when your program genuinely serves unusual files, and document any custom mapping so future maintainers understand why it exists.

Check: Why must a guessed media type never be used as a security decision about a file's contents?

Section 31

Archives bundle many files into one, which makes them easier to store, share, and back up. The zipfile module reads and writes ZIP archives, the most widely used archive format, and it does so without needing any external program. A program can create an archive with ZipFile in write mode, add files to it with write(), and later list its contents with namelist() or extract them with extractall(). That covers the common cases: packaging a set of output files for a user, or unpacking an archive a user supplies.

Working with archives through zipfile also means working with paths inside the archive, which are not the same as paths on disk. An archive can contain entries with directory components, and extracting them recreates that directory structure. A careful program checks the names it is about to extract, because a malicious archive can contain entries with .. components that would escape the target directory when extracted naively. Modern Python's extraction guards against the most obvious forms of this, but the habit of validating archive contents before extracting them is still worth forming, especially for archives received from untrusted sources.

Compression is optional in a ZIP archive. Files can be stored uncompressed, which is fast and simple, or compressed with a method such as DEFLATE, which trades CPU time for a smaller archive. The choice depends on the data: text compresses well, while already-compressed formats such as images and videos gain little from further compression. The module lets you choose per archive, and for most purposes the default settings are sensible.

Archives are also useful inside a program's own workflow, not just for distribution. A test suite can store fixture data in a ZIP file to keep a project directory tidy, and a backup tool can snapshot a directory into a dated archive. In both cases, the archive is a container, not a substitute for a proper backup strategy: a single archive file can itself be lost or corrupted, so important data still needs more than one copy in more than one place.

Reading archives lazily is another benefit. A program can open an archive, inspect its table of contents, and read only the entries it needs, without extracting everything to disk first. That saves time and disk space when an archive is large and only a few of its files are relevant. The open() method on a ZipFile returns a file-like object for a single entry, so entry contents can be processed in memory just like an ordinary file.

import zipfile
from pathlib import Path
Path("a.txt").write_text("hello")
with zipfile.ZipFile("a.zip", "w") as z:
    z.write("a.txt")
with zipfile.ZipFile("a.zip") as z:
    print(z.namelist(), z.read("a.txt"))

Test archive creation with small examples first, and verify the result by listing its contents before distributing it.

Check: Why should a program validate archive entry names before extracting an archive from an untrusted source, and when is compression a poor choice?

Section 32

Single files, rather than collections of files, are compressed with a different set of tools. The gzip module reads and writes files in the gzip format, which wraps one stream of data with a header and a checksum, while lzma provides the newer XZ format, which typically compresses text more tightly at the cost of more CPU time. Both modules present a file-like interface, so a program can read a compressed file almost exactly as it would read an ordinary file, with decompression happening transparently as data is read.

The usual pattern is to open a compressed file in text or binary mode and process it line by line or in chunks, exactly as with an uncompressed file. That means a log-processing program can accept both plain and gzipped logs with only a small change to how it opens the file, and a data-export program can offer a compressed download without restructuring its writing code. Compression becomes a property of the file object, not of the logic that uses it.

Choosing between gzip and lzma is a trade-off between speed, compatibility, and size. Gzip is faster and more widely supported by other tools, which makes it the safer default when a file will be exchanged with other systems or opened by command-line tools. Lzma achieves smaller files, which matters when storage or transfer costs dominate, but it is slower and less universally supported. For data that stays inside one program's own storage, either is fine; for data that leaves the program, gzip's compatibility usually wins.

A point worth remembering is that compression does not protect data. A compressed file is not encrypted, and anyone can decompress it just as easily as you compressed it. If data must be kept private, it needs encryption on top of compression, handled by tools designed for that purpose. Compression and encryption solve different problems, and confusing them is a common source of false confidence about data safety.

Another practical note is that compressing already-compressed data, such as JPEG images or ZIP archives, wastes time for almost no reduction in size. Compression works by finding redundancy, and data that is already compressed has little redundancy left to find. A program that compresses everything indiscriminately will be slower without being smaller, so it is worth checking whether a given kind of data actually benefits before adding compression to its pipeline.

import gzip
with gzip.open("note.txt.gz", "wt") as f:
    f.write("hello world\n")
with gzip.open("note.txt.gz", "rt") as f:
    print(f.read())

When reading compressed files, handle the case where a file is not actually compressed, since a program that assumes compression will fail confusingly on plain input. Compressed files are common in data pipelines, where logs and exports are gzipped to save disk space and transfer time, and programs that read them must handle both compressed and plain forms gracefully.

Check: Why is gzip usually preferred over lzma for files exchanged with other systems, and why does compression not protect private data?

Section 33

Where ZIP archives are the common choice for general use, TAR archives are the traditional choice on Unix-like systems, and the tarfile module reads and writes them. A TAR archive bundles files and directories into a single stream, preserving names, permissions, and directory structure, and it is commonly combined with gzip compression, producing the familiar .tar.gz file. The module handles that combination directly: opening a file in "r:gz" mode reads a gzipped TAR archive, and "w:gz" writes one.

The same caution about extraction applies here as with ZIP archives, and more strongly, because TAR archives can also contain links and device entries in addition to ordinary files. A program that extracts an archive from an untrusted source should inspect its members first, checking names and types, and should extract only the entries it actually needs, into a directory it controls. Modern Python adds filters to the extraction methods that reject dangerous members, and using those filters is now the recommended practice rather than extracting blindly.

TAR archives preserve more metadata than ZIP archives, including file permissions and ownership, which is why they are favoured for backups and for distributing source trees where executable permissions matter. A script stored in a TAR archive keeps its executable bit when extracted, while the same script in a ZIP archive may not, depending on how the archive was created. That difference is a practical reason to choose TAR for backups of a working directory, even when ZIP would be simpler.

Creating archives programmatically follows the same pattern as reading them: open the archive, add files or directories with add(), and close it. Adding a whole directory recursively is a single call, which makes a small backup script surprisingly short. Listing an archive's members before extracting, with getnames() or getmembers(), lets a program report what an archive contains or verify that it contains the expected files before committing to extraction.

As with other archive formats, a TAR file is a container, not a backup strategy by itself. Keep more than one copy, in more than one place, and test restoring from an archive occasionally, because an archive that has never been restored from is a hope, not a backup.

import tarfile
from pathlib import Path
Path("b.txt").write_text("hello")
with tarfile.open("b.tar.gz", "w:gz") as t:
    t.add("b.txt")
with tarfile.open("b.tar.gz") as t:
    print(t.getnames())

For archives you create yourself, record the archive format and compression in the filename, so future readers know how to open it without guessing. TAR archives also store directory entries themselves, not just files, so an extracted directory tree recreates empty directories as well as files, which matters for projects where an empty directory has meaning, such as a placeholder for generated output. Test restores regularly, not just archive creation.

Check: Why is TAR favoured for backups where permissions matter, and what should a program check before extracting an untrusted TAR archive?

Section 34

Two small modules help programs match and split text in the ways shells do. The fnmatch module matches filenames against shell-style patterns, where * matches any sequence of characters, ? matches a single character, and [abc] matches one character from a set. Its fnmatch() function tests one name against one pattern, while filter() selects the matching names from a list. That is useful for a program that lets a user specify which files to process with a pattern like *.txt, without implementing pattern matching itself.

The shlex module splits a string the way a shell would, respecting quotes and escapes. Given a command line such as copy "my file.txt" out/, shlex.split() returns the three arguments copy, my file.txt, and out/, keeping the quoted filename together. That is the correct way to turn a command-line string into an argument list, for example before passing it to subprocess.run() with shell=False, as discussed earlier in this chapter. Splitting naively on spaces would break quoted arguments and produce wrong results.

These two modules complement each other: fnmatch answers "does this name match this pattern?", while shlex answers "how does this string break into arguments?". A small file-processing tool might use both — shlex to parse a user-supplied command line, and fnmatch to select files matching a pattern from that command line — and each module keeps its own job small and well defined.

Both modules have limits worth knowing. fnmatch patterns are not regular expressions, and they do not understand directory separators specially on every platform, so a pattern like *.txt matches names in one directory, not a recursive search through subdirectories — that is a job for pathlib's globbing or for os.walk. shlex follows POSIX shell quoting rules by default, which is what most users expect, but it can also operate in a non-POSIX mode for other quoting conventions. Knowing which mode you need avoids surprises when input contains unusual quoting.

The general lesson is to reuse the standard library's text tools rather than writing your own pattern matcher or argument splitter. Hand-written versions almost always miss an edge case — a quoted space, a bracket expression, an escaped character — that the standard modules handle correctly.

import fnmatch, shlex
print(fnmatch.fnmatch("report.txt", "*.txt"))
print(shlex.split('copy "my file.txt" out/'))

For recursive file searches, prefer pathlib globbing over fnmatch alone, since fnmatch does not traverse directories by itself and needs a separate walk to visit subdirectories. Shell-style patterns are also used in configuration files and command-line tools, where a user expects familiar wildcard behaviour, and fnmatch gives a program that behaviour without depending on an actual shell. Combine fnmatch with os.walk for recursive matching across a tree. Test pattern matching with edge cases like hidden files and names containing brackets.

Check: Why is shlex.split() the correct way to turn a command-line string into arguments, and what can fnmatch patterns not do?

Section 35

Comparing files and reading many files are two more tasks the standard library handles directly. The filecmp module compares files and directories, with cmp() testing whether two files have the same contents and dircmp() comparing the contents of two directories, reporting files that appear in only one directory and files that differ. That is useful for verifying a backup, checking whether a copy operation succeeded, or writing a test that compares a program's output with an expected file.

By default, filecmp.cmp() uses a shallow comparison: it first checks file metadata such as size and modification time, and only reads the contents if the metadata suggests the files might differ. Passing shallow=False forces a full content comparison, which is slower but definitive. The distinction matters when metadata is unreliable — for example, when a file has been copied in a way that preserves its timestamp, or when two different files happen to share a size. For verification purposes, a full comparison is the safer choice, and the cost is usually acceptable for files of ordinary size.

The fileinput module provides a different convenience: it lets a program read lines from several files in sequence, as if they were one continuous input stream, and it also handles the convention where a filename of - means standard input. That makes it easy to write small filter-style programs — the kind that read input, transform each line, and write output — that accept any number of input files on the command line. It also supports in-place editing, where a program rewrites its input files with transformed contents, keeping a backup of the original if requested.

In-place editing deserves a warning. It modifies files directly, so a bug in the transformation can destroy the original data. Using the backup option, or testing the transformation on copies first, is prudent. In-place editing is convenient for quick scripts, but for anything important, writing to a new file and replacing the original only after success — the pattern from the files chapter — is safer.

Both modules are small, and both reward the same habit: before writing file-comparison or multi-file-reading code by hand, check whether the standard library already provides it, because its edge cases have already been considered.

import filecmp
from pathlib import Path
Path("x.txt").write_text("same")
Path("y.txt").write_text("same")
print(filecmp.cmp("x.txt", "y.txt", shallow=False))

When comparing directories, remember that dircmp reports only names and shallow differences by default, so follow up with full content comparisons for files that matter. File comparison is also useful in tests that check a program's output against a golden file, where a byte-for-byte comparison is stricter and more informative than comparing printed summaries. Golden-file tests catch regressions that unit assertions might miss. Compare files after a copy operation to confirm the copy is complete.

Check: When is a shallow file comparison misleading, and why is in-place editing with fileinput riskier than writing to a new file?

Section 36

Finding the differences between two pieces of text is a frequent task, from showing what changed in a file to helping a user understand why a test's output differs from the expected output. The difflib module provides tools for that job. Its SequenceMatcher compares two sequences and finds their matching and differing blocks, while unified_diff() produces a diff in the unified format familiar from version control, with lines marked as added, removed, or unchanged, plus a few lines of context around each change.

A unified diff is designed for human readers. Context lines show where a change sits, and the + and - markers make it obvious what was added and what was removed. That makes it a good format for reports: a program that compares an expected output with an actual output can present the difference as a diff, and the reader immediately sees the mismatch without scanning two full texts side by side. The Differ class provides a more detailed, line-by-line comparison for cases where the unified format hides too much.

difflib also offers get_close_matches(), which finds the closest matches to a word from a list of possibilities. That is useful for "did you mean?" suggestions: when a user mistypes a command name or a configuration key, a program can suggest the nearest valid option instead of failing with a bare error. The suggestions are based on a similarity ratio, and the cutoff parameter controls how similar a candidate must be to count as a suggestion, so tuning it matters — too low a cutoff produces irrelevant suggestions, too high a cutoff misses helpful ones.

One limitation is scale. SequenceMatcher is not designed for very large texts, and its algorithm can be slow on long inputs with many differences. For comparing large files, dedicated diff tools are faster. For the everyday cases in a program — comparing outputs, suggesting corrections, showing small changes — difflib is more than adequate, and its output formats are ones that users and developers already know how to read.

import difflib
a = ["alpha\n", "beta\n"]
b = ["alpha\n", "gamma\n"]
print("".join(difflib.unified_diff(a, b, lineterm="")))

When suggesting corrections, show only the closest few matches, since a long list of weak suggestions is less helpful than one or two strong ones. Diff output can also be stored as a patch, which records changes in a form that other tools can apply, making difflib output useful beyond human reading, for example in review tools and migration scripts. For large files, use a dedicated diff tool instead of difflib, since speed matters more than convenience at that scale. Use difflib to show test failures clearly, so a failing test reports exactly what changed. A short diff is often the best test failure message.

Check: Why is a unified diff easier to read than two full texts side by side, and what does the cutoff parameter control in get_close_matches()?

Section 37

Not every problem in a program deserves to stop execution. Some situations are worth warning about — a deprecated feature is being used, a configuration value is unusual, a fallback has been applied — while the program continues with a sensible behaviour. The warnings module provides that middle ground. Calling warnings.warn() issues a warning of a chosen category, such as DeprecationWarning for features that will be removed, or UserWarning for general user-facing concerns, and the program carries on.

Warnings are configurable, which is what makes them useful rather than noisy. A program can filter warnings by category, by module, or by message, showing them once, always, or never, and it can turn warnings into errors during testing, so that a deprecated call fails a test suite rather than passing silently. That last option is a good practice: turning deprecation warnings into errors in tests surfaces future breakage early, when it is cheap to fix, rather than at the moment a dependency is upgraded.

Library authors use warnings differently from application authors. A library should warn, not print, when it encounters a questionable situation, because printing to standard output pollutes a user's program output, while warnings can be filtered or redirected by the caller. An application, by contrast, usually configures warning filters at its entry point, deciding which warnings its users should see. That division of labour — libraries warn, applications configure — keeps warnings useful across the ecosystem.

Warnings also have a relationship with the logging module from earlier in this chapter, and the two should not be confused. Logging records events for later inspection, at levels from debug to critical, and it is the right tool for operational information. Warnings are for conditions that a developer should know about, typically because code needs to change: a deprecated function, an invalid input that was tolerated this time, a behaviour that will change in a future version. Use logging for what happened, and warnings for what should change.

import warnings
warnings.warn("old_function is deprecated, use new_function", DeprecationWarning)
print("continuing")

Document which warnings your program turns into errors, so users running tests understand why a warning fails the suite rather than merely appearing in output. Warning categories also help users filter noise: a program can show DeprecationWarnings to developers while hiding them from end users, who cannot act on them, by configuring filters appropriately. Keep warnings visible in development and quiet in production, adjusting filters per environment rather than disabling them entirely. Warnings are not errors, but they signal work that should be scheduled before a future upgrade breaks the code. Use warnings for deprecations and logging for operational events, keeping the two channels distinct. Review warnings regularly, since ignored warnings accumulate into future breakage.

Check: Why should a library warn rather than print when it encounters a questionable situation, and why turn warnings into errors during testing?

Section 38

Context managers were covered in the previous chapter, and the contextlib module extends them with utilities that solve recurring problems. The most useful is ExitStack, which manages a collection of context managers whose number is not known in advance. A program that must open a variable number of files, for example one per input file named on the command line, can enter each one into an ExitStack and be sure that all of them are closed when the stack exits, even if an error occurs partway through. Without ExitStack, that pattern requires manual bookkeeping that is easy to get wrong.

The suppress() context manager is a smaller convenience: it suppresses chosen exception types within its block, so code that attempts an operation which may legitimately fail — deleting a file that may not exist, for example — can be written without a try/except block that does nothing in its handler. Suppressing only specific exceptions matters here: suppressing all exceptions would hide genuine bugs, while suppressing exactly the expected failure keeps the code honest about what it tolerates.

contextlib also provides closing(), which ensures that an object with a close() method is closed on exit, even if that object is not itself a context manager. That is useful for objects from libraries that predate context manager support, or for resources whose cleanup method is named close() rather than following the context manager protocol. It is a small adapter, but it lets older code participate in the with statement's guarantees.

These utilities share a theme with the rest of this chapter: they remove boilerplate that is easy to write incorrectly. Manual resource bookkeeping, empty exception handlers, and forgotten close() calls are all common sources of bugs, and contextlib replaces each with a named, tested pattern. When you find yourself writing such boilerplate, check whether contextlib already names it.

from contextlib import suppress, ExitStack
from pathlib import Path
with suppress(FileNotFoundError):
    Path("missing.txt").unlink()
print("suppressed")

Use ExitStack when the number of resources depends on input, such as a list of files supplied at runtime, rather than being fixed in the source code. ExitStack also supports entering context managers conditionally, so a program can acquire a resource only when a condition holds, and still guarantee its release, without nested if statements around with blocks. An ExitStack can also manage non-file resources, such as locks or temporary directories, in a single place with one exit point. ExitStack also helps when resources must be released in reverse order of acquisition, which it handles automatically on exit. When the set of resources is known in advance, a plain with statement with several managers is clearer than an ExitStack. Test ExitStack cleanup with an exception inside the block, to confirm every resource is released.

Check: When is ExitStack needed instead of a plain with statement, and why must suppress() name specific exception types?

Section 39

The abc module provides abstract base classes, which are classes that cannot be instantiated directly and that define methods subclasses must implement. A class declares an abstract method with the @abstractmethod decorator, and any subclass that does not implement all abstract methods also cannot be instantiated: attempting to create an instance raises a TypeError. That turns a documentation convention — "subclasses should implement this method" — into an enforced rule that fails at instantiation time, when the mistake is easy to find, rather than later, when a missing method is called.

Abstract base classes are a design tool, not a requirement for every hierarchy. They suit situations where a family of classes shares an interface that must be complete: a plugin system where every plugin must provide certain operations, a storage backend where every backend must support reading and writing, or a test double where a fake must honour the same contract as the real object. In those cases, the abstract class documents the contract and enforces it in one place, so a new subclass that forgets a method is caught immediately.

They also interact with the protocol ideas from the dunder methods chapter. A protocol describes behaviour informally — any object with certain methods qualifies — while an abstract base class describes it formally, through inheritance. Formality has a cost: a class must inherit from the abstract base to be recognised, which couples it to that hierarchy. Informal protocols are more flexible, since any object with the right methods qualifies without inheritance. The choice between them depends on whether enforcement or flexibility matters more for a given design.

A common mistake is to make every base class abstract, which adds ceremony without benefit when a hierarchy has only one or two subclasses that are developed together. Abstract base classes pay off when the subclasses are written by different people, at different times, or as plugins, where the contract needs enforcement. For a small, private hierarchy, a plain base class with documented methods is simpler and equally clear.

from abc import ABC, abstractmethod
class Shape(ABC):
    @abstractmethod
    def area(self):
        pass
class Square(Shape):
    def __init__(self, side):
        self.side = side
    def area(self):
        return self.side ** 2
print(Square(3).area())

When designing a plugin interface, write the abstract base class first, then a simple example plugin, to check that the required methods are complete and sensible. Abstract methods can also have a default implementation that subclasses may call with super(), which is useful when most subclasses share a common step but some need to extend it. Document the contract of each abstract method clearly, including its parameters, return value, and any exceptions it may raise. Keep abstract hierarchies shallow, since deep abstract layers are hard to follow and rarely add enforcement value.

Check: What does an abstract base class enforce that a documented convention does not, and when is that enforcement not worth its cost?

Section 40

Binary data appears whenever a program reads a file format, a network packet, or a hardware device's output, and the struct module packs and unpacks such data according to a format string. A format string describes the layout of a sequence of values — for example, ">Ih" means a big-endian unsigned integer followed by a signed short — and pack() converts Python values into bytes in that layout, while unpack() converts bytes back into values. That makes it possible to read and write binary formats precisely, without manual bit manipulation.

Endianness is the first concept to grasp. Multi-byte numbers can be stored most-significant-byte first, called big-endian, or least-significant-byte first, called little-endian, and a format string's first character selects which. Getting this wrong produces values that look plausible but are wrong — a classic binary-format bug — so the byte order should always be stated explicitly in the format string, never left to a platform default, when data will be exchanged between systems or stored in a file.

Alignment and padding are the second concept. Some formats insert padding bytes between fields to align them to word boundaries, and struct handles standard sizes and alignment when asked, but the safest approach for file formats is to use standard size and no alignment, indicated by = or by an explicit endianness character, so the layout is exactly the sum of its fields. That avoids surprises when the same code runs on platforms with different alignment rules.

The calcsize() function reports how many bytes a format occupies, which is useful for reading fixed-size records from a file: read that many bytes, unpack them, and repeat. Combined with the file reading techniques from the files chapter, that gives a straightforward pattern for parsing binary files record by record, checking at each step that enough bytes remain for a complete record rather than unpacking a truncated buffer and getting a confusing error.

Binary formats are less forgiving than text formats, because a single misplaced byte shifts every following field. Working with struct therefore rewards care: document the format, test with known data, and validate lengths before unpacking. The module itself is small; the discipline around it is what makes binary parsing reliable.

import struct
data = struct.pack(">Ih", 300, -5)
print(data, struct.unpack(">Ih", data))

For binary formats you define yourself, prefer text formats unless binary size or speed genuinely matters, since text is easier to inspect and debug. When parsing untrusted binary data, always validate lengths and magic values before unpacking, since malformed input can otherwise cause crashes or misleading results in later processing. Document every binary format you parse, including field order, sizes, and byte order, so future readers can verify the format string.

Check: Why must byte order be stated explicitly in a format string for exchanged data, and what does calcsize() let a program check before unpacking?

Section 41

Most Python programs use lists for sequences of numbers, and lists are the right default: they are flexible, well supported, and easy to read. The array module offers a more specialised alternative for the case where a sequence contains many values of a single numeric type. An array stores its items in a compact, uniform representation, using the same amount of memory per item, rather than storing a reference to a separate Python object for each item as a list does. For large numeric datasets, that compactness can reduce memory use substantially.

Arrays are created with a type code that states the item type, such as "i" for signed integers or "d" for doubles, and they support most list operations: indexing, slicing, appending, and iteration. They also convert to and from bytes with tobytes() and frombytes(), which makes them convenient for reading numeric data from a binary file or preparing it for writing. That byte conversion is their main advantage over a list comprehension that converts values one at a time.

The trade-off is type uniformity. An array of integers cannot hold a string, and inserting a value of the wrong type raises an error. That strictness is a feature when the data genuinely is uniform, because it catches mistakes early, but it is a limitation when the data is mixed. For mixed data, a list, a tuple, or a dataclass remains the better choice, and forcing it into an array would obscure its structure.

Related to arrays is the numbers module, which defines the abstract numeric tower: Number, Complex, Real, and Integral, among others. These abstract classes let a function declare that it accepts any real number, for example, rather than only int or only float, and they let a custom numeric type register itself as a member of the appropriate category. Most programs never need to use numbers directly, but it is the reason that generic numeric code can treat integers and floats uniformly.

The practical guidance is simple: use lists by default, reach for array when a large, uniform numeric sequence's memory use or byte conversion matters, and remember that specialised tools exist for specialised needs rather than replacing the general tool everywhere.

from array import array
a = array("i", [1, 2, 3])
a.append(4)
print(a, a.tobytes())

Arrays also support the buffer protocol, so they can be passed directly to functions that accept binary data, without an intermediate conversion step that copies the data. For most programs, the memory saving of an array only matters at large scale, so measure before switching from a list, and keep the simpler representation when the saving is small. An array's type code is part of its contract, so document it where the array is created.

Check: When does an array beat a list for numeric data, and what does its type uniformity cost?

Section 42

References normally keep objects alive: as long as a variable, a list, or a dictionary refers to an object, Python's garbage collector will not reclaim it. That is usually what you want, but it causes a problem for caches and registries, where keeping a reference to every object ever registered would prevent any of them from ever being freed, even after the rest of the program has finished with them. The weakref module solves this by providing weak references, which refer to an object without keeping it alive.

A weak reference lets a program observe an object while it exists, and notice when it disappears. Calling a weak reference returns the object if it is still alive, or None if it has been collected. A WeakValueDictionary applies the same idea to a mapping: its values are held weakly, so entries vanish automatically when their values are collected elsewhere. That makes it a natural building block for caches, where an entry should survive only as long as something else still uses the cached object.

Weak references also support callbacks, which run when the referenced object is collected. A callback can clean up associated state, such as removing a record from a secondary index, at exactly the moment the object disappears. That keeps bookkeeping consistent without requiring the program to remember to clean up manually at every place where an object might be dropped.

Not every object supports weak references. Most user-defined classes do, but some built-in types, such as plain lists and dictionaries, do not, unless they are subclassed. That limitation is worth knowing before designing around weak references: if the objects to be tracked are built-in containers, a wrapper class or a different design may be needed. Testing with a small example reveals the limitation quickly, before it shapes a larger design.

Weak references are an advanced tool, and most programs do not need them. Their lesson is broader, though: Python's memory management is automatic, but it follows rules about references, and understanding those rules — strong references keep objects alive, weak references do not — explains behaviour that otherwise looks mysterious, such as a cache that grows without bound.

import weakref
class Item:
    pass
item = Item()
ref = weakref.ref(item)
print(ref() is item)
del item
print(ref())

A weak reference callback receives the weak reference itself as its argument, so it can identify which object disappeared when several objects share one callback function. Weak dictionaries are also used in frameworks to associate metadata with objects without preventing their collection, a pattern that keeps framework bookkeeping from leaking memory over time. Test weak reference behaviour with a small script before relying on it in a design. Weak references are rarely needed in small programs.

Check: Why would a cache use weak references, and what does calling a weak reference return after its object is collected?

Section 43

Configuration files need a format that is easy for humans to read and write, and TOML has become a popular choice for that role, notably for project metadata. The tomllib module reads TOML files, parsing them into ordinary Python dictionaries, lists, strings, numbers, and dates. A program that reads its settings from a TOML file can use tomllib.load() with a file opened in binary mode, and then access settings through the resulting dictionary, just as with any other mapping.

TOML's structure is straightforward: key-value pairs, sections marked with [section] headers, nested tables with dotted headers, and arrays of tables for repeated entries. That structure maps naturally onto Python dictionaries, which is why the parsed result feels familiar to work with. Dates and times in TOML are parsed into date and time objects, which is convenient, though the details of date handling belong to the next chapter and need not be explored here.

One important asymmetry is that tomllib only reads TOML; it does not write it. Writing TOML requires either constructing the text by hand, which is error-prone for complex structures, or using an external library. For programs that only need to read configuration supplied by a user or by another tool, that limitation does not matter. For programs that must write configuration back, a different format — or hand-written output for a simple, fixed structure — may be more practical.

Choosing a configuration format is a small design decision with long consequences. TOML suits human-edited configuration with sections and nested structure; the INI-style format from configparser, covered earlier, suits simpler configuration; JSON, covered in the next chapter, suits configuration produced by programs rather than edited by hand. Each format's strengths follow from who writes it and how often it changes.

As with other configuration formats, sensitive values do not belong in a TOML file that might be shared or committed to version control. Secrets belong in environment variables or a dedicated secrets store, and configuration files should hold only the non-sensitive settings that describe how a program behaves.

import tomllib
data = tomllib.loads('[server]\nhost = "localhost"\nport = 8080\n')
print(data["server"]["host"], data["server"]["port"])

When a TOML file grows large, split it into sections that mirror your program's structure, so each part of the program reads only the section it needs. Validate configuration after loading it, checking that required keys exist and values are of the expected type, so a typo in a configuration file produces a clear error at startup rather than a failure later. Keep configuration files under version control when they contain no secrets, so changes are reviewed and reversible. TOML suits configuration edited by humans, not data generated by programs. Choose the format that fits who writes the configuration.

Check: Why does it matter that tomllib reads but does not write TOML, and when is TOML a better configuration choice than JSON?

Section 44

Programs usually import modules with the import statement, which is clear and sufficient for almost every case. The importlib module provides the machinery behind that statement for the rarer cases where importing must happen dynamically — when the module name is only known at runtime, for example from a configuration file, a plugin directory, or a command-line argument. Its import_module() function imports a module by name given as a string, returning the module object just as an import statement would.

Dynamic importing enables plugin systems. A program can read a list of plugin names from configuration, import each one, and call a known function from it, without hard-coding the list of plugins in its source. That makes the program extensible: adding a plugin means adding a module and a configuration entry, not modifying the program's code. The same pattern supports optional features, where a program tries to import an optional dependency and falls back gracefully if it is absent.

That flexibility carries risks that static imports do not. A module name from an untrusted source is effectively code execution: importing a module runs its top-level code, so a program must never import a name supplied by an untrusted user without validation. Even with trusted sources, dynamic imports make code harder to follow, because a reader cannot see from the source alone which modules a program uses. Static analysis tools and packaging tools also struggle with dynamic imports, sometimes missing a dependency that is only named in a string.

The guidance, then, is to prefer static imports wherever possible, and to use dynamic imports only where the module genuinely varies at runtime, such as a plugin system or an optional dependency. When you do use them, validate names against an allow-list where the names come from outside the program, document the expected interface of dynamically loaded modules, and handle import failure explicitly rather than letting it crash the program with a confusing error.

importlib also provides tools for reloading modules and for inspecting import machinery, but those are advanced uses. Reloading, in particular, rarely does what a developer hopes, because existing objects keep referring to the old module's classes, and it is almost always better to restart a program than to reload a module inside it.

import importlib
math_mod = importlib.import_module("math")
print(math_mod.sqrt(16))

For optional dependencies, catch ImportError explicitly and provide a clear fallback message, rather than letting the import failure surface as an unrelated crash later. Document dynamically loaded modules in your project readme, listing their expected names and interfaces, so a maintainer can find them without tracing runtime import logic. Prefer a plugin registry over dynamic imports when the set of plugins is known at development time. Dynamic imports hide dependencies from readers and packaging tools alike.

Check: Why is importing a module name from an untrusted source dangerous, and when is dynamic importing justified despite that risk?

Section 45

A running program can inspect its own objects — their types, their attributes, their signatures — and the inspect module provides the tools for that introspection. Its getsource() function returns the source code of a function or class, signature() describes a callable's parameters, and getmembers() lists an object's attributes. That is useful for debugging tools, documentation generators, and frameworks that adapt their behaviour to the shape of the objects they are given.

Introspection also powers the interactive help system. When you call help() on an object in a REPL, or use dir() to list its attributes, you are using the same kind of introspection that inspect formalises. Building the habit of inspecting unfamiliar objects interactively — checking a signature before calling a function, listing an object's methods before using it — is one of the fastest ways to learn a new library, because it turns guesswork into observation.

Frameworks use introspection more heavily than application code. A test framework, for example, discovers test functions by inspecting modules for names matching a pattern; a serialisation library inspects an object's attributes to decide what to store. Application code rarely needs that level of introspection, and when it does, it is often a sign that a simpler design — an explicit registration, a dictionary of handlers — would be clearer. Introspection is powerful, but it makes code harder to follow, because behaviour depends on the shape of objects rather than on explicit statements.

There is also a performance consideration. Introspection is slower than direct attribute access, and it should not sit in a hot loop. Use it at setup time, to discover or configure something once, and then use direct calls for the repeated work. That pattern — inspect once, act many times — keeps the flexibility of introspection without paying its cost repeatedly.

For everyday learning, the most valuable inspect functions are the simplest: signature() to understand a function you are about to call, and getsource() to read how a small utility is implemented. Both reward curiosity, and both make the standard library itself a readable textbook.

import inspect
def greet(name, greeting="hello"):
    return f"{greeting} {name}"
print(inspect.signature(greet))

Inspecting a function signature also reveals default values, which tell you which arguments are optional and what behaviour you get when you omit them. Use inspect sparingly in application code, reserving it for tools and frameworks, and prefer explicit code paths where the behaviour must be obvious to a reader. A signature printed by inspect is often enough to use a function correctly without reading its full documentation. Introspection is a tool for frameworks and debugging, not for everyday logic. Check a signature before calling an unfamiliar function. Inspect once at setup, then use direct calls afterwards.

Check: Why should introspection be used at setup time rather than in a hot loop, and when is explicit registration clearer than introspection?

Section 46

Programs that behave differently on different platforms need to know which platform they are running on, and the platform module provides that information. Its system() function returns the operating system name, machine() returns the hardware type, and python_version() returns the Python version in use. That lets a program choose a platform-specific path, select an appropriate external command, or report its environment in a diagnostic message, without parsing system details by hand.

The sysconfig module complements it with information about how Python itself was installed: where its standard library lives, where third-party packages are installed, and which compiler flags were used. That is mainly useful for tools that build or install packages, rather than for everyday application code, but it is the reason that packaging tools can locate the right directories on any system without hard-coding paths. Knowing that such a module exists is enough for most developers; using it directly is a specialist task.

Platform-dependent code should be kept small and isolated. A common pattern is to detect the platform once, at a program's entry point or in a dedicated module, and then use plain variables or small helper functions for the rest of the program, rather than scattering platform checks through the logic. That keeps the main logic readable and makes the platform-specific parts easy to find and test.

A related caution is to avoid platform checks when a portable abstraction already exists. The pathlib module, covered earlier, handles path separators across platforms, so a program should not check the platform to decide which separator to use. Platform detection is for genuine differences — a command that exists only on one system, a directory convention that differs — not for differences the standard library already hides. Reaching for platform first, before checking whether a portable tool exists, is a common beginner mistake that produces unnecessarily complex code.

Diagnostic output is another good use. When a user reports a problem, asking for the output of a small environment report — operating system, Python version, and relevant paths — often narrows the diagnosis immediately, and platform makes generating that report trivial.

import platform
print(platform.system(), platform.python_version())

Environment reports are also useful in bug reports you file yourself, since maintainers can reproduce a problem faster when they know the platform and Python version. When a program must support several platforms, test on each one regularly, since platform-specific code that is only exercised on one system tends to rot unnoticed. Report platform and Python version together, since behaviour can depend on both, not on either alone. Isolate platform differences in one module, and keep the rest of the program portable. Test platform-specific code on every platform it claims to support. Prefer portable abstractions over platform checks wherever one exists.

Check: Why should platform checks be isolated in one place, and when is a platform check unnecessary because a portable abstraction exists?

Section 47

Randomness for security is different from randomness for games or simulations, and the secrets module exists for the security case. Its functions generate random values suitable for passwords, tokens, and other secrets, using the operating system's best source of randomness. secrets.token_hex() produces a random token as hexadecimal text, token_urlsafe() produces one safe for use in URLs, and choice() picks randomly from a sequence using the same secure source. These should be used wherever a value must be unpredictable to an attacker.

The distinction matters because the random module, covered earlier, is designed for modelling and simulation, not for security. Its generator is deterministic given a seed, which is a feature for reproducible tests but a weakness for secrets: anyone who learns or guesses the seed can predict every value it produces. Using random to generate a password or a session token is therefore a real vulnerability, even though the values look random to a casual observer. The rule is simple and absolute: random for simulation, secrets for secrets, with no exceptions.

The hashlib module complements secrets for a different purpose: computing digests, or fingerprints, of data. A digest is a fixed-size value derived from input data, such that changing the input changes the digest, which makes digests useful for verifying that a file has not been altered — comparing a downloaded file's digest with a published one, for example. The module provides several digest algorithms, and the choice among them matters: older algorithms with known weaknesses should not be chosen for new work, while current standard algorithms are appropriate for integrity checking.

Digests are not encryption, and a plain digest is not a safe way to store a password. Password storage needs specialised, deliberately slow hashing designed for that purpose, which is a security topic for a later chapter. The survey point here is narrower: hashlib verifies integrity, secrets generates unpredictability, and neither should be confused with the other, or with encryption, which keeps data private rather than merely verifying or generating it.

import secrets, hashlib
print(secrets.token_hex(8))
print(hashlib.sha256(b"hello").hexdigest())

Tokens generated for sessions should be long enough that guessing is infeasible, and they should be stored hashed where possible, so a leaked database does not expose live tokens. Never log a token or secret, even in debug output, since logs are often stored longer and shared more widely than the program's author expects. Rotate tokens periodically, and invalidate old tokens when they are replaced, so a leaked token has a limited useful lifetime. Treat every generated token as sensitive from the moment it is created. Use secrets for tokens, and random only for simulations and games. A digest verifies integrity; it does not keep data private. Use the secure module for every secret.

Check: Why is random unsuitable for generating secrets even though its output looks random, and what is a digest used for?

Section 48

Some programs need to ask a user for a password or another sensitive value, and reading it with ordinary input() would echo it on the screen, where anyone nearby could read it. The getpass module solves that small but important problem: its getpass() function prompts the user and reads their input without echoing it, so the value does not appear on the terminal. That is the correct way to read a password interactively, and it is a one-line change from input() that meaningfully improves a program's behaviour.

getpass also provides getuser(), which returns the current user's login name, useful for defaulting a username prompt or for labelling output. Together, the two functions cover the common interactive case: ask for a username with a sensible default, and ask for a password without echoing it. Neither function stores or manages the password; they only read it, and what the program does with the value afterwards — how it is used, and whether it is kept in memory longer than necessary — is the program's own responsibility.

Reading a password without echo is a user-interface courtesy and a modest security improvement, not a complete security solution. A password typed on a shared screen can still be observed by other means, and a program that stores a password in a variable, writes it to a log, or sends it over an insecure channel has undone the benefit. The habit to form is to treat sensitive input as sensitive throughout its lifetime: read it without echo, use it promptly, avoid logging it, and clear it from memory when it is no longer needed where that is practical.

Where a program runs non-interactively, such as a scheduled job, prompting is impossible, and secrets must come from a secure source such as an environment variable or a secrets store, as discussed for configuration earlier in this chapter. getpass is for interactive use; automated use needs a different mechanism, and hard-coding a password in source code is never an acceptable substitute for either.

import getpass
user = getpass.getuser()
print("current user:", user)

When a program must run both interactively and non-interactively, support both input methods, prompting with getpass when a terminal is present and reading from a secure source otherwise. After reading a password, avoid passing it through string formatting or logging calls, since either could accidentally record it in output or in an error message. Clear sensitive variables as soon as they are no longer needed, where the language and libraries make that practical. Never hard-code a password in source code, even temporarily, since temporary code has a habit of becoming permanent. Read passwords without echo, and handle them carefully thereafter. Interactive prompts need a terminal; scheduled jobs need a secrets store.

Check: What does getpass() improve over input() for passwords, and why is that improvement not a complete security solution?

Section 49

The standard library is large, and no one memorises it all. The skill that matters most is learning how to learn a module quickly, and Python provides built-in tools for exactly that. The help() function, used interactively, displays a module's, class's, or function's documentation, including its signature and a description of its purpose. The dir() function lists an object's attributes, giving a quick map of what an object can do. Together, they turn any unfamiliar object into something you can explore without leaving the interpreter.

Reading documentation effectively is a skill in itself. Start with a module's overview, which states what it is for and when to use it, then look at its main functions or classes, and only then read the details of the function you need. Skipping straight to a single function's details often leaves you without the context that explains why the function exists or what its arguments mean. Examples in the documentation are especially valuable: running a small example, modifying it, and observing the result teaches faster than reading a description alone.

Another habit is to search before building. When you face a task — parsing a configuration file, comparing two texts, generating a unique identifier — ask first whether the standard library already provides a tool for it. The chapters of this book, and this chapter in particular, are a map of the most useful tools, but the full library is larger, and a quick search of its documentation often reveals a module you had not known existed. Reusing a tested tool beats writing your own version, which will have edge cases you have not yet discovered.

Finally, learn from the library's own source. Much of the standard library is written in Python, and reading a small module's implementation shows idiomatic code, clear naming, and careful handling of edge cases. It is one of the best ways to improve as a Python programmer, because it shows how experienced developers solve the same kinds of problems you face, using the same language features you are learning.

import math
print([name for name in dir(math) if not name.startswith("_")][:5])

Keep a personal notebook of modules you discover, with a one-line note of what each does, so you can find the right tool again months later without searching from scratch. Reading documentation is faster than trial and error: a few minutes with the overview of a module often saves an hour of guessing at its behaviour. Try examples from documentation in a REPL, modifying them slightly, to confirm your understanding before using a module in real code. A short experiment in a REPL often answers a documentation question faster than re-reading the text. Search the standard library before building a common tool yourself.

Check: Why should you read a module's overview before a single function's details, and what should you do before writing your own version of a common tool?

Section 50

This chapter has surveyed a large part of the standard library, from file and path tools through collections, mathematics, text handling, archives, and configuration, to the smaller utilities that solve one problem well. The survey was deliberately broad rather than deep: each module was introduced with its purpose, its main functions, and the situations where it is the right choice, so that you can recognise the tool when you need it and know where to look for its details. Depth comes from using a module in a real program, not from reading about it a second time.

The principles that recur across the survey are worth carrying forward. Prefer a standard tool over a custom implementation for a common task, because the standard tool has been tested against edge cases you have not yet imagined. Understand the security implications of the tools you use, whether that is shell injection through subprocess, predictable randomness through random, or extraction risks from untrusted archives. Choose tools whose names and behaviour make your program's intent clear, because code is read more often than it is written, and a reader who recognises a standard tool understands your code faster.

A practical way to consolidate this chapter is to revisit a program you have already written — perhaps the file-processing program from an earlier chapter — and look for places where a standard library module could replace hand-written code. A manual file search might become a pathlib glob, a hand-rolled counter might become a Counter, a custom argument parser might become argparse. Each replacement shortens the program and removes a potential bug, and the exercise of finding replacements trains the habit of searching before building.

The next chapters build on this foundation with focused topics: dates and JSON in the next chapter, regular expressions after that, and then testing, debugging, and the deeper subjects that follow. Each of those chapters will use modules from this survey without reintroducing them, so the time spent here pays off directly. Keep this chapter as a reference: when a later chapter mentions a module in passing, return here for its introduction, and then to the module's own documentation for its full details.

from collections import Counter
print(Counter(["a", "b", "a"]))

The standard library rewards curiosity: the more modules you recognise, the fewer problems you solve from scratch, and the more time you spend on the logic that is unique to your program. Revisit this chapter whenever a new task feels familiar, since recognising a problem as one the standard library already solves is the skill this chapter teaches. Use this chapter as a map, not a memorisation exercise, returning to it whenever a task needs a tool you half-remember.

Check: What habit does this chapter most want you to form, and why does replacing hand-written code with a standard tool remove bugs?

Practical Task

Build a small command-line tool called stdtool.py that uses at least six standard library modules from this chapter. It should accept a directory path as a positional argument (use argparse, with a --help message), list the files in that directory matching a pattern given with an optional --pattern argument (default *.txt, matched with pathlib globbing), and print a summary: the number of matching files, the total size in bytes, and the most common file extension among them (use Counter). It should log its progress at INFO level with the logging module, handle a missing directory with a clear error message and a non-zero exit (not a bare traceback), and include a --dry-run flag that prints what it would do without reading file contents. Write the summary as a dataclass with fields for the count, total size, and most common extension, and add a unittest test file with at least five tests: missing directory exits non-zero, empty directory reports zero files, a directory with known files reports the correct count and total size, the pattern filter excludes non-matching files, and the most common extension is correct for a mixed set. Run the tests and fix any failures before considering the task done.

Chapter Questions

1. When should you use pathlib instead of os.path, and when might a program still need os or sys directly?
2. Why is shell=True with untrusted input dangerous in subprocess, and what is the safe alternative?
3. How do you decide between a list, a Counter, a defaultdict, and a deque for a given task, and what goes wrong if you choose the wrong one?
4. When is decimal arithmetic required instead of floating point, and why must decimals be built from strings rather than floats?
5. A colleague proposes writing a custom configuration parser, a custom argument parser, and a custom pattern matcher for a new tool. How would you argue for standard library modules instead, and in what situation would a custom implementation actually be justified?